通过VPN /代理进行JSoup

问题描述

我正在尝试使用JSoup来抓取登台服务器上的某些页面。要使用浏览器查看登台服务器上的页面，我需要连接到VPN。

I'm trying to use JSoup to scrape some pages that are on a staging server. To view the pages on the staging server with a browser I need to be connected to a VPN.

我连接到VPN但是当我使用JSoup尝试刮它保持超时的页面。如何让我的程序使用VPN连接。或者在这里我还有其他的东西我没想到？

I am connected to the VPN but when I use JSoup to try to scrape the page it keeps timing out. How can I make my program use the VPN connection. Or is there something else here I'm not thinking of?

注意：我也在程序的另一部分使用HttpClient。有没有办法在程序初始化后设置我的程序连接到VPN / Proxy，这样JSoup和HttpClient都使用VPN /代理。

Note: I also make use of HttpClient in another part of my program. Is there a way I can set my program to connect to the VPN/Proxy once the program initialises so both JSoup and HttpClient use the VPN/Proxy.

谢谢

推荐答案

您可以为代理设置java属性：

You can set java properties for the proxy:

// if you use https, set it here too
System.setProperty("http.proxyHost", "<proxyip>"); // set proxy server
System.setProperty("http.proxyPort", "<proxyport>"); // set proxy port

Document doc = Jsoup.connect("http://your.url.here").get(); // Jsoup now connects via proxy

或者将网站下载到一个字符串中并解析它：

or download the website into a string and parse it then:

final URL website = new URL("http://your.url.here"); // The website you want to connect

// -- Setup connection through proxy
Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("<proxyserver>", 1234)); // set proxy server and port
HttpURLConnection httpUrlConnetion = (HttpURLConnection) website.openConnection(proxy);
httpUrlConnetion.connect();

// -- Download the website into a buffer
BufferedReader br = new BufferedReader(new InputStreamReader(httpUrlConnetion.getInputStream()));
StringBuilder buffer = new StringBuilder();
String str;

while( (str = br.readLine()) != null )
{
    buffer.append(str);
}

// -- Parse the buffer with Jsoup
Document doc = Jsoup.parse(buffer.toString());

您可以使用 HttpClient 作为此解决方案好吧。

You can use HttpClient for this solution as well.

这篇关于通过VPN /代理进行JSoup的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！