我正在使用Crawler4j示例代码，但是我发现我遇到了异常。
这是我的例外：
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/http/conn/scheme/SchemeSocketFactory
at LocalDataCollectorController.main(LocalDataCollectorController.java:24)
Caused by: java.lang.ClassNotFoundException: org.apache.http.conn.scheme.SchemeSocketFactory
这是我的代码：
public static void main(String[] args) throws Exception {
String root Folder = "D:\\";
int numberOfCrawlers = 5;
System.out.println("numberOfCrawlers"+numberOfCrawlers);
System.out.println(rootFolder);
CrawlConfig config = new CrawlConfig();
config.setCrawlStorageFolder(rootFolder);
config.setMaxPagesToFetch(10);
config.setPolitenessDelay(1000);
PageFetcher pageFetcher = new PageFetcher(config);
RobotstxtConfig robotstxtConfig = new RobotstxtConfig();
RobotstxtServer robotstxtServer = new RobotstxtServer(robotstxtConfig, pageFetcher);
CrawlController controller = new CrawlController(config, pageFetcher, robotstxtServer);
controller.addSeed("http://www.ohloh.net/p/crawler4j");
controller.start(LocalDataCollectorCrawler.class, numberOfCrawlers);
List<Object> crawlersLocalData = controller.getCrawlersLocalData();
long totalLinks = 0;
long totalTextSize = 0;
int totalProcessedPages = 0;
for (Object localData : crawlersLocalData) {
CrawlStat stat = (CrawlStat) localData;
totalLinks += stat.getTotalLinks();
totalTextSize += stat.getTotalTextSize();
totalProcessedPages += stat.getTotalProcessedPages();
}
System.out.println("Aggregated Statistics:");
System.out.println(" Processed Pages: " + totalProcessedPages);
System.out.println(" Total Links found: " + totalLinks);
System.out.println(" Total Text Size: " + totalTextSize);
}
}
最佳答案
下载HttpClient并将其添加到您的构建路径。
还有一个包，其中包含download section中的所有crawler4j依赖项。您应该使用它来避免进一步的问题。
关于java - 使用Crawler4j时，线程“main”中的异常java.lang.NoClassDefFoundError:org/apache/http/conn/scheme/SchemeSocketFactory，我们在Stack Overflow上找到一个类似的问题：https://stackoverflow.com/questions/10430846/