python - Python从Tripadvisor抓取``要做的事情''

在this页面上，我要抓取“在迈阿密可以做的事情类型”列表（您可以在页面末尾找到它）。这是我到目前为止的内容：

import requests
from bs4 import BeautifulSoup

# Define header to prevent errors
user_agent = "Mozilla/44.0.2 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.109 Safari/9.0.2"

headers = {'User-Agent': user_agent}

new_url = "https://www.tripadvisor.com/Attractions-g34438-Activities-Miami_Florida.html"
# Get response from url
response = requests.get(new_url, headers = headers)
# Encode response for parsing
html = response.text.encode('utf-8')
# Soupify response
soup = BeautifulSoup(html, "lxml")

tag_elements = soup.findAll("a", {"class":"attractions-attraction-overview-main-Pill__pill--23S2Q"})

# Iterate over tag_elements and exctract strings
tags_list = []
for i in tag_elements:
    tags_list.append(i.string)

问题是，我从页面的“事物类型...”部分下面的页面的“在迈阿密常见搜索”区域中获得了类似'Good for Couples (201)', 'Good for Big Groups (130)', 'Good for Kids (100)'的值。我也没有得到一些我需要的值，例如"Traveler Resources (7)", "Day Trips (7)"等。这两个列表的类名称“事情要做...”和“常用搜索...”是相同的，我正在使用类在soup.findAll()中，这可能是导致此问题的原因。正确的方法是什么？我还应该采取其他方法吗？

最佳答案

仅获取Types of Things to Do in Miami标头中的内容有些棘手。为此，您需要像下面所做的那样以有组织的方式定义选择器。以下脚本应在上述标题下单击See all按钮。启动点击后，脚本将解析您要查找的相关内容：

from selenium import webdriver
from selenium.webdriver.support import ui
from bs4 import BeautifulSoup

driver = webdriver.Chrome()
wait = ui.WebDriverWait(driver, 10)
driver.get("https://www.tripadvisor.com/Attractions-g34438-Activities-Miami_Florida.html")

show_more = wait.until(lambda driver: driver.find_element_by_css_selector("[class='ui_container'] div:nth-of-type(1) .caret-down"))
driver.execute_script("arguments[0].click();",show_more)
soup = BeautifulSoup(driver.page_source,"lxml")
items = [item.text for item in soup.select("[class='ui_container'] div:nth-of-type(1) a[href^='/Attractions-']")]
print(items)
driver.quit()

输出产生：

['Tours (277)', 'Outdoor Activities (255)', 'Boat Tours & Water Sports (184)', 'Shopping (126)', 'Nightlife (126)', 'Spas & Wellness (109)', 'Fun & Games (67)', 'Transportation (66)', 'Museums (61)', 'Sights & Landmarks (54)', 'Nature & Parks (54)', 'Food & Drink (27)', 'Concerts & Shows (25)', 'Classes & Workshops (22)', 'Zoos & Aquariums (7)', 'Traveler Resources (7)', 'Day Trips (7)', 'Water & Amusement Parks (5)', 'Casinos & Gambling (3)', 'Events (2)']

关于python - Python从Tripadvisor抓取``要做的事情''，我们在Stack Overflow上找到一个类似的问题：https://stackoverflow.com/questions/53452863/