Download opendata graph Free Java Code
Description
Code to crawl Common Crawl corpus in order to create a graph of french opendata websites.
Source Files
The download file opendata-graph-master.zip has the following entries.
.gitignore/*from w w w . j a v a 2s . c o m*/
README.md
commonCrawl/README.md
commonCrawl/pom.xml
commonCrawl/src/main/java/com/datapublica/commoncrawl/aggregation/Aggregation.java
commonCrawl/src/main/java/com/datapublica/commoncrawl/aggregation/NumericAggregationMapper.java
commonCrawl/src/main/java/com/datapublica/commoncrawl/aggregation/NumericAggregationWithFilterMapper.java
commonCrawl/src/main/java/com/datapublica/commoncrawl/aggregation/TextualAggregationMapper.java
commonCrawl/src/main/java/com/datapublica/commoncrawl/indexing/FrenchWebIndexMapper.java
commonCrawl/src/main/java/com/datapublica/commoncrawl/indexing/OpenDataIndexMapper.java
commonCrawl/src/main/java/com/datapublica/commoncrawl/indexing/RunFrenchWebIndexing.java
commonCrawl/src/main/java/com/datapublica/commoncrawl/linking/OpenDataLinkingMapper.java
commonCrawl/src/main/java/com/datapublica/commoncrawl/linking/RunOpenDataLinking.java
commonCrawl/src/main/java/com/datapublica/commoncrawl/stats/opendata/OpenDataStatsMapper.java
commonCrawl/src/main/java/com/datapublica/commoncrawl/stats/opendata/RunOpenDataStats.java
commonCrawl/src/main/java/com/datapublica/commoncrawl/stats/opendata/SitesPageRatesReducer.java
commonCrawl/src/main/java/com/datapublica/commoncrawl/utils/JobHelper.java
commonCrawl/src/main/java/com/datapublica/commoncrawl/utils/LanguageDetector.java
commonCrawl/src/main/java/com/datapublica/commoncrawl/utils/Loggers.java
commonCrawl/src/main/java/com/datapublica/commoncrawl/utils/MapHelper.java
commonCrawl/src/main/java/com/datapublica/commoncrawl/utils/ProjectDeployer.java
commonCrawl/src/main/resources/aws.properties
commonCrawl/src/main/resources/com/datapublica/commoncrawl/utils/opendata-paths.txt
commonCrawl/src/main/resources/com/datapublica/commoncrawl/utils/opendata-sites.txt
commonCrawl/src/main/resources/com/datapublica/commoncrawl/utils/profiles-list.txt
commonCrawl/src/test/java/com/datapublica/commoncrawl/utils/mandatoryElementsTests.java
graph/README.md
graph/categories_graph.pdf
graph/graph.gephi
graph/print/print_categories.pdf
graph/print/print_roles.pdf
graph/roles_graph.pdf
preview/cluster-meteo.png
preview/cluster-pyrenees.png
preview/core.png
preview/overview.png
Download
Click the following link to download opendata-graph-master.zip.
opendata-graph-master.zip