python - 用 BeautifulSoup 刮多页

我正在尝试抓取网址的多个页面。

但是能够仅刮取第一页，因此可以获取所有页面。

这是我的代码。

from bs4 import BeautifulSoup as Soup
import urllib, requests, re, pandas as pd

pd.set_option('max_colwidth',500)    # to remove column limit (Otherwise, we'll lose some info)
df = pd.DataFrame()

Comp_urls = ['https://www.indeed.com/jobs?q=Dell&rbc=DELL&jcid=0918a251e6902f97', 'https://www.indeed.com/jobs?q=Harman&rbc=Harman&jcid=4faf342d2307e9ed','https://www.indeed.com/jobs?q=johnson+%26+johnson&rbc=Johnson+%26+Johnson+Family+of+Companies&jcid=08849387e791ebc6','https://www.indeed.com/jobs?q=nova&rbc=Nova+Biomedical&jcid=051380d3bdd5b915']

for url in Comp_urls:
    target = Soup(urllib.request.urlopen(url), "lxml")
    targetElements = target.findAll('div', class_ =' row result')

    for elem in targetElements:
        comp_name = elem.find('span', attrs={'class':'company'}).getText().strip()
        job_title = elem.find('a', attrs={'class':'turnstileLink'}).attrs['title']
        home_url = "http://www.indeed.com"
        job_link = "%s%s" % (home_url,elem.find('a').get('href'))
        job_addr = elem.find('span', attrs={'class':'location'}).getText()
        date_posted = elem.find('span', attrs={'class': 'date'}).getText()
        description = elem.find('span', attrs={'class': 'summary'}).getText().strip()


        comp_link_overall = elem.find('span', attrs={'class':'company'}).find('a')
        if comp_link_overall != None:
        comp_link_overall = "%s%s" % (home_url, comp_link_overall.attrs['href'])
        else: comp_link_overall = None

        df = df.append({'comp_name': comp_name, 'job_title': job_title,
                'job_link': job_link, 'date_posted': date_posted,
                'overall_link': comp_link_overall, 'job_location': job_addr, 'description': description
                }, ignore_index=True)


df

df.to_csv('path\\web_scrape_Indeed.csv', sep=',', encoding='utf-8')

无论如何，请提出建议。

最佳答案

情况1：此处提供的代码正是您所拥有的

Comp_urls = ['https://www.indeed.com/jobs?q=Dell&rbc=DELL&jcid=0918a251e6902f97', 'https://www.indeed.com/jobs?q=Harman&rbc=Harman&jcid=4faf342d2307e9ed','https://www.indeed.com/jobs?q=johnson+%26+johnson&rbc=Johnson+%26+Johnson+Family+of+Companies&jcid=08849387e791ebc6','https://www.indeed.com/jobs?q=nova&rbc=Nova+Biomedical&jcid=051380d3bdd5b915']

for url in Comp_urls:
    target = Soup(urllib.request.urlopen(url), "lxml")
    targetElements = target.findAll('div', class_ =' row result')

for elem in targetElements:

这里的问题是，在第一个for循环中，每次迭代都会更改targetElements。

为避免这种情况，请在第一个循环中缩进第二个for循环，如下所示：

for url in Comp_urls:
    target = Soup(urllib.request.urlopen(url), "lxml")
    targetElements = target.findAll('div', class_ =' row result')

    for elem in targetElements:

情况2：您的错误不是由于缩进不当造成的（即与您原始帖子中的内容不同）
如果是正确识别您的代码的情况，则targetElements是空列表的情况也可能。这意味着target.findAll('div', class_ =' row result')不返回任何内容。在这种情况下，请访问站点，签出dom，然后修改您的抓取程序。