我最近在抓取HackerNews网站时遇到了这个SyntaxError:“标题”是参数，并且是非本地的，这是相同代码。
import requests
from bs4 import BeautifulSoup
import time
# Url Generator
def urls_gen(base_url, pages=1):
try:
for page in range(1, int(pages) + 1):
yield f'{base_url}news?p={str(page)}'
except ValueError as e:
print(e)
# Url Crawler
# max pages = 15
def urls_crawler(urls):
for url in urls:
try:
res = requests.get(url)
soup = BeautifulSoup(res.text, 'html.parser')
soup.prettify()
titles = soup.select('.storylink')
subtext = soup.select('.subtext')
except IndexError:
return('Max Pages Limit Reached!')
def custom_hn(titles, subtext):
data = []
for i, item in enumerate(titles):
nonlocal titles
nonlocal subtext
title = titles[i].getText()
link = titles[i].get('href', None)
print(title, link)
time.sleep(1)
return custom_hn(titles, subtext)
urls = urls_gen('https://news.ycombinator.com/', 'a')
links = urls_crawler(urls)
任何帮助将不胜感激!
最佳答案
我对您的脚本进行了一些修改，并删除了try..except语句。另外，您可以测试是否在最后一页上，如果没有任何链接需要刮一下。
例如:
import requests
from bs4 import BeautifulSoup
def urls_gen(base_url, pages=1):
for page in range(1, pages + 1): # <-- remove int()
yield f'{base_url}news?p={page}' # <-- remove str()
def urls_crawler(urls):
for url in urls:
res = requests.get(url)
soup = BeautifulSoup(res.text, 'html.parser')
all_titles = soup.select('.storylink')
if not all_titles: # <-- if there arent't any link, break the loop
break
for title in all_titles:
yield title.get_text(), title['href']
urls = urls_gen('https://news.ycombinator.com/', 2) # <-- put number here instead of 'a'
for text, link in urls_crawler(urls):
print('{:<80} {}'.format(text, link))
Show HN: A weirdly detailed graphical analysis of women's tops sold by Goodwill https://goodwill.awardwinninghuman.com/
EasyOCR: Ready-to-use OCR with 40 languages https://github.com/JaidedAI/EasyOCR
Tauri – toolchain for building highly secure native apps that have tiny binaries https://github.com/tauri-apps/tauri
Foreign data wrappers: PostgreSQL's secret weapon? https://www.splitgraph.com/blog/foreign-data-wrappers
Why Developers Stop Learning: Rise of the Expert Beginner (2012) https://daedtech.com/how-developers-stop-learning-rise-of-the-expert-beginner/
The EU General Data Protection Regulation Explained by Americans https://hroy.eu/posts/gdprExplainedByUS/
Git commit accepts several message flags (-m) to allow multiline commits https://www.stefanjudis.com/today-i-learned/git-commit-accepts-several-message-flags-m-to-allow-multiline-commits/
Ariane RISC-V CPU – An open source CPU capable of booting Linux https://github.com/openhwgroup/cva6
SUSE to Acquire Rancher Labs https://rancher.com/press/suse-to-acquire-rancher/
Yoloface-500k: ultra-light real-time face detection model, 500kb https://github.com/dog-qiuqiu/MobileNetv2-YOLOV3#500kb%E7%9A%84yolo-face-detection
Google drops blogspot.in breaking hundred thousands of permalinks item?id=23767781
Symbolic execution with SymCC: Don't interpret, compile http://www.s3.eurecom.fr/tools/symbolic_execution/symcc.html
The More Senior Your Job Title, the More You Need to Keep a Journal https://hbr.org/2017/07/the-more-senior-your-job-title-the-more-you-need-to-keep-a-journal
SUSE to Acquire Rancher Labs https://www.suse.com/c/news/suse-acquires-rancher/
Mcfly – neural-network powered directory and context-aware shell history search https://github.com/cantino/mcfly
Ron Graham has died https://www.ams.org/news?news_id=6244
SUSE acquires Kubernetes management platform Rancher Labs https://techcrunch.com/2020/07/08/suse-acquires-kubernetes-management-platform-rancher-labs/
KeePassXC 2.6.0 Released https://keepassxc.org/blog/2020-07-07-2.6.0-released/
SymPy - a Python library for symbolic mathematics https://www.sympy.org/en/index.html
Curry Before Columbus https://contingentmagazine.org/2020/06/25/curry-before-columbus/
...and so on.
关于python-3.x - SyntaxError;名称 'x'是参数且非本地，我们在Stack Overflow上找到一个类似的问题：https://stackoverflow.com/questions/62795616/