问题描述
我正尝试使用操作>下载> CSV下的下载CSV链接从此网站提取数据
I'm trying to pull data from this website using the download CSV link found under Actions > Download > CSV
我在UrlFetchApp行上始终收到错误302,但我不明白为什么。我使用httpexception运行记录器,并收到错误消息,提示它无法连接到服务器。该网站是公开的,可以通过任何网络访问,因此我很难理解正在发生的事情。手动输入时,URL似乎下载了CSV,但是我无法让UrlFetchApp抓取它。
I get an error 302 on the UrlFetchApp line consistently and I'm not understanding why. I ran the logger with the httpexception and got the error that it could not connect to the server. The website is public and accessible regardless of network so I'm having a hard time understanding what's going on. The URL seems to download the CSV when input manually but I cannot get the UrlFetchApp to grab it.
var EquipUrl = "https://app.udot.utah.gov/apex/eom/f?p=288:300:::CSV";
var EquipContent = UrlFetchApp.fetch(EquipUrl).getContentText();
var EquipData = Utilities.parseCsv(EquipContent);
任何帮助将不胜感激!
我只是在注释中尝试了每个建议的以下内容,并且得到了相同的错误302结果。我还尝试将重定向设置为false,并且此操作以空的EquipData数组结尾
I just tried the following per suggestion in comments and had the same Error 302 result. I also tried setting redirects to false and this proceeds and ends with an empty EquipData array
var options = {
'followRedirects' :true
];
var EquipUrl = "https://app.udot.utah.gov/apex/eom/f?p=288:300:::CSV::::";
var EquipContent = UrlFetchApp.fetch(EquipUrl, options).getContentText();
var EquipData = Utilities.parseCsv(EquipContent);
感谢您对TheMaster的帮助!我在评论中说,该代码在var cookie行上吐出了一个错误,但该错误已自行纠正,因此我认为这可能对您而言是代码的错误副本。现在的问题是,最后的Logger行吐出无法解析文本错误。
Thanks for the help so far TheMaster! I said in my comment that the code was spitting out an error on the var cookies line but that error rectified itself so I think it may have been a bad copy of your code on my part. The problem is now that the final Logger line spits out a "Could not parse text" error.
function fetchUrlWithCookie() {
var url = 'https://app.udot.utah.gov/apex/eom/f?p=288:300:::CSV';
var response = UrlFetchApp.fetch(url, {
muteHttpExceptions: true,
followRedirects: false,
});
var cookie = response.getAllHeaders()['Set-Cookie'];
response = UrlFetchApp.fetch(url, {
muteHttpExceptions: true,
headers: {
Cookie: cookie, //send the cookie we got as header
},
});
Logger.log(Utilities.parseCsv(response.getContentText()));//parseCSV
}
我尝试了几种其他选择,例如尝试先登录以查看是否能找到任何东西。我可以从Stackdriver日志中提取任何信息来帮助发现问题吗?
I tried a few different options like trying to log before to see if I could find anything. Is there any information I could pull from Stackdriver logging that could help identify issues?
非常感谢TheMaster!有效!如果您能发表评论并提出解决问题的过程,我将不胜感激,这样我可以帮助您了解下次发生这种情况时要注意的事情,但我知道这是很多问题。
Thank you so much TheMaster! It worked! I would love if you could comment and address your problem solving process so I can help learn what to be on the lookout for next time this happens but I know that's asking a lot.
var url = 'https://app.udot.utah.gov/apex/eom/f?p=288:300::CSV::::';
var response = UrlFetchApp.fetch(url, {
muteHttpExceptions: true,
followRedirects: true,
});
var cookie = response.getAllHeaders()['Set-Cookie'];
response = UrlFetchApp.fetch(url, {
muteHttpExceptions: true,
headers: {
Cookie: cookie, //send the cookie we got as headerw
},
});
Logger.log(Utilities.parseCsv(response.getContentText()));//parseCSV
这是功能齐全的代码的副本。再次感谢您在这几天内为解决此问题所提供的帮助。
That's a copy of the code that fully functions. Thank you again for your help over this several day period to address this issue.
推荐答案
要点:
某些网站使用 cookies
来维护状态,会话并完成请求。由于 urlFetch
在服务器端而不是浏览器中运行,因此不会在请求之间维护cookie。但是,我们可以在第一个请求中手动获取cookie,然后在后续请求中发送它。
Key Points:
Some websites use cookies
for maintaining state, session and completing the request. Since urlFetch
runs on server side and not in a browser, cookies are not maintained across requests. However, We can manually get the cookie in the first request and send it in subsequent requests.
function fetchUrlWithCookie() {
var url = 'xxxx'; //Your csv url
var response = UrlFetchApp.fetch(url, {
muteHttpExceptions: true,
followRedirects: false,
});
var cookie = response.getAllHeaders()['Set-Cookie']; //Get cookie from header
response = UrlFetchApp.fetch(url, {
muteHttpExceptions: true,
headers: {
Cookie: cookie, //send the cookie we got as header
},
});
Logger.log(Utilities.parseCsv(response.getContentText()));//parseCSV
}
参考文献:
- RFC6265
- UrlFetchApp
- Response headers
References:
这篇关于UrlFetchApp无法连接到公共URL的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!