无法让我的脚本执行下一页按钮上的所有点击 | 无法让我的脚本执行下一页按钮上的所有点击

本文介绍了无法让我的脚本执行下一页按钮上的所有点击的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

我使用 pyppeteer 在 python 中创建了一个脚本em> 收集从一个网站遍历多个页面的不同机构的名称.我希望做的是让我的脚本在解析每个页面的名称时通过单击下一页按钮来移动不同的页面.

I've created a script in python using pyppeteer to collect the names of different institutions traversing multiple pages from a website. What I wish to do is let my script rove different pages by clicking on the next page button while parsing the names from each page.

网站地址

我尝试过的:

import asyncio
from pyppeteer import launch

url = "https://www.incometaxindia.gov.in/Pages/utilities/exempted-institutions.aspx"

async def fetch_table(link):
    browser = await launch(headless=False)
    [page] = await browser.pages()
    await page.goto(link)
    while True:
        await page.waitForSelector("h1.faqsno-heading", {'visible':True})
        for item in await page.querySelectorAll("h1.faqsno-heading"):
            name = await item.querySelectorEval("div[id^='arrowex']",'e => e.innerText')
            print(name)

        try:
            elem =  await page.querySelector("[title='Next Page']")
            await elem.click()
        except Exception: break

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(fetch_table(url))

上述脚本运行良好，直到遇到 5 到 10 页之间的错误.不过，页面可能会有所不同.

The above script is doing it's job just fine until it encounters an error somewhere between 5 to 10 pages. Pages may vary, though.

Traceback (most recent call last):
  File "C:\Users\WCS\AppData\Local\Programs\Python\Python37-32\demo.py", line 23, in <module>
    loop.run_until_complete(fetch_table(url))
  File "C:\Users\WCS\AppData\Local\Programs\Python\Python37-32\lib\asyncio\base_events.py", line 568, in run_until_complete
    return future.result()
  File "C:\Users\WCS\AppData\Local\Programs\Python\Python37-32\demo.py", line 11, in fetch_table
    await page.waitForSelector("h1.faqsno-heading", {'visible':True})
  File "C:\Users\WCS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pyppeteer\frame_manager.py", line 834, in __await__
    raise result
pyppeteer.errors.TimeoutError: Waiting for selector "h1.faqsno-heading" failed: timeout 30000ms exceeds.

然而，当我带来一个小的改变并像这样尝试时，我可以看到脚本也在完成它的工作，直到它遇到以下错误:

However, when I bring about a minor change and try like this, I can see that the script also does it's job until it encounters the following error:

try:
    await page.click("[title='Next Page']")
except Exception: break

我收到以下错误:

Traceback (most recent call last):
  File "C:\Users\WCS\AppData\Local\Programs\Python\Python37-32\demo.py", line 48, in <module>
    loop.run_until_complete(fetch_table(url))
  File "C:\Users\WCS\AppData\Local\Programs\Python\Python37-32\lib\asyncio\base_events.py", line 568, in run_until_complete
    return future.result()
  File "C:\Users\WCS\AppData\Local\Programs\Python\Python37-32\demo.py", line 37, in fetch_table
    await page.waitForSelector("h1.faqsno-heading", {'visible':True})
  File "C:\Users\WCS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pyppeteer\frame_manager.py", line 832, in __await__
    result = yield from self.promise
  File "C:\Users\WCS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pyppeteer\frame_manager.py", line 859, in rerun
    *self._args,
  File "C:\Users\WCS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pyppeteer\execution_context.py", line 109, in evaluateHandle
    _rewriteError(e)
  File "C:\Users\WCS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pyppeteer\execution_context.py", line 239, in _rewriteError
    raise error
  File "C:\Users\WCS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pyppeteer\execution_context.py", line 106, in evaluateHandle
    'userGesture': True,
pyppeteer.errors.NetworkError: Protocol error Runtime.callFunctionOn: Target closed.

如何让我的脚本一直运行直到所有点击都执行完毕?

推荐答案

请注意，您尝试抓取的网站有数百页！我不想让我的系统卡住太久运行过程.相反，我尝试了 slot=20 个页面，它似乎有效.您可以更改插槽数量以进行实验.我正在使用 python 3.6，websockets 6.0.我在 Windows 8.1 上.我添加了几行代码来限制页数.除此之外，我还添加了await page.waitForSelector("[title='Next Page']", {'visible':True}) 在几个地方.

Please note that the website you trying to scrape has hundreds of pages! I didn't want to my system get stuck for longrunning process. Instead I tried slots=20 pages, it appears to be working. You may change the number of slots to experiment yourself.I am using python 3.6, websockets 6.0. I am on windows 8.1.I have added few lines of code to limit number of pages. In addition to that I have addedawait page.waitForSelector("[title='Next Page']", {'visible':True}) in couple of places.

这是代码

import asyncio
from pyppeteer import launch

url = "https://www.incometaxindia.gov.in/Pages/utilities/exempted-institutions.aspx"

async def fetch_table(link):
    browser = await launch(headless=False)
    [page] = await browser.pages()
    await page.goto(link)
    slots=20 # change here for number of pages you want to scrape
    i=0
    while True:
        i=i+1
        if(i>slots):
           await page.waitForSelector("[title='Next Page']", {'visible':True})
           break
        await page.waitForSelector("h1.faqsno-heading", {'visible':True})
        for item in await page.querySelectorAll("h1.faqsno-heading"):
            name = await item.querySelectorEval("div[id^='arrowex']",'e => e.innerText')
            print(name)

        try:
            await page.waitForSelector("[title='Next Page']", {'visible':True})
            elem =  await page.querySelector("[title='Next Page']")
            await elem.click()
        except Exception: break


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(fetch_table(url))

-输出大约第 20 页

-Output around 20th page

(testenv) C:\Py\pypuppeteer1>python stack5.py
....
....
SHREE SUBRAHMANYA VANGMAYEE PARISHAD, GOAAAPTS2410M
SHREE SUBRAHMANYA VANGMAYEE PARISHAD, GOAAAPTS2410M
WORD FOR THE WORLD FELLOWSHIPAAAAW6295Q
JANA SEVA TRUSTAACTJ0594Q
VAGDEVI VILAS EDUCATIONAL AND CHARITABLE TRUSTAABTV8264G
NCORE IMPACT FOUNDATIONAAFCN9985K
M V M EDUCATIONAL TRUSTAACTM5633K
SOCIETY FOR BETTERMENT OF EDUCATIONAAHAS9354D
SWASTIKAM CHARITABLE TRUSTAAJTS9298K
M/S SANKALP YUVA PRERIT SANVARDHAN BAHUUDDESHIYA SANSTHAAAITS8452J
TRAILOKYA BOUDHA MAHASANGHA SAHAYYAK GAN NAGPURAAABT2581K
MISSIONAL YATRA INDIA (MY INDIA) CHARITABLE TRUSTAAOTM9109M
VRUNDAVAN SHIKSHAN VA BAHUUDDESHIYA SANSTHAAABAV6403C
SHRI JAGDAMBA GOVIGYAN ANUSANDHAN KENDRAAAQTS8474C
SUSHILABAI DEUSKAR PRATISHTHANAALTS8647L
AMRAVATI DISTRICT OPTHALMIC SOCIETYAAETA8499F
ALUMNI ASSOCIATION OF INDIRA GANDHI GOVERNMENT MEDICAL COLLEGE NAGPURAAGTA1367C
VIDYA NIDHI NAGPURAABTN4351L
LATE RAJSINGH DUNGAPUR MEMORIAL FOUNDATIONAABTL5457B
ARTHIK DRUSTYA MAGASVARGIYA SAMAJ SHIKSHAN SANSTHAAACTA6288L
SPARSHAADAS4064Q
LATE PADMADEVI R. MALOO FOUNDATIONAAATL4181B
VISHWARACHNA GRAMINS VIKAS SANSTHAAAATV5359D

这篇关于无法让我的脚本执行下一页按钮上的所有点击的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！