我有以下工作代码(在此处和此处更改了,因此在复制和粘贴时会用到大脑)。我想对其进行改进,以便它检测到所有无效页面,包括待出售的域名。它的效率约为89%。如果您看到任何东西,我可以通过使用其他现有的库或进行一些微妙的调整来改善。
List all = linkService.getAllLinks();
notValidLinks = new LinkedList();
final ArrayBlockingQueue<Runnable> queue = new ArrayBlockingQueue<Runnable>(39867);
int poolSize = 90;
int maxPoolSize = 100;
long keepAliveTime = 40;
ThreadPoolExecutor tpe = new ThreadPoolExecutor(poolSize, maxPoolSize,
keepAliveTime, TimeUnit.SECONDS, queue);
for (link : all) {
Thread task = new CheckSite(link);
tpe.execute(task);
System.out.println("Task count:" + queue.size());
}
class CheckSite extends Thread {
Link link;
CheckSite(Link link) {
this.link = link;
}
public void run() {
boolean notValid = false;
try {
log.info(link.getLink() + " " + link.getId());
URL u = new URL(link.getLink());
HttpURLConnection huc = (HttpURLConnection) u.openConnection();
HttpURLConnection.setFollowRedirects(false);
huc.setConnectTimeout(40000);
huc.setRequestMethod("GET");
huc.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729)");
huc.connect();
int code = huc.getResponseCode();
if (code != HttpURLConnection.HTTP_OK
&& code != HttpURLConnection.HTTP_MOVED_PERM
&& code != HttpURLConnection.HTTP_MOVED_TEMP ){
notValid = true;
log.info("Invalid code: " + code + " - " + link.getLink());
}
if (code == HttpURLConnection.HTTP_MOVED_PERM) {
log.info(link.getLink() + " Perm move");
}
if (code == HttpURLConnection.HTTP_MOVED_TEMP) {
log.info(link.getLink() + " Temp move");
}
try {
if (!notValid) {
BufferedReader reader = new BufferedReader(new InputStreamReader(huc.getInputStream()));
StringBuilder stringBuilder = new StringBuilder();
String line;
while ((line = reader.readLine()) != null) {
stringBuilder.append(line);
}
notValid = StringUtils.containsIgnoreCase(Jsoup.parse(stringBuilder.toString()).text(), "Related Searches");
}
} catch (Exception e) {
log.error(e.getMessage());
}
huc.disconnect();
} catch (MalformedURLException me) {
log.info("Malformed URL:" + link.getLink());
notValid = true;
} catch (IOException e) {
log.info("Refused connection | Does not exist:" + link.getLink());
notValid = true;
}
if (notValid) {
link.setApproved(false);
link.setDateApproved(null);
notValidLinks.add(linkService.save(link));
}
log.debug("URL Finieshed!");
}
}
最佳答案
请查看Bloom Filter [wiki]。这将帮助您快速且高效地进行内存查找。Bloom过滤器的问题在于它会误报。对于不存在的内容,它会说是对的。但是,如果Bloom过滤器说为false,那肯定是错误的。