python - Web刮HTML加油站

Closed. This question needs to be more focused。它当前不接受答案。
                            
                        
                    
                
                            
                                
                
                        
                            
                        
                    
                        
                            想改善这个问题吗？更新问题，使其仅通过editing this post专注于一个问题。
                        
                        去年关闭。
                                                                                            
                
        
给定此链接http://www.nj.gov/treasury/administration/statewide-support/motor-fuel-locations.shtml，我想抓取每个加油站及其信息

import requests
from bs4 import BeautifulSoup
page=requests.get("http://www.nj.gov/treasury/administration/statewide-support/motor-fuel-locations.shtml")
soup=BeautifulSoup(page.content,'html.parser')
for x in soup.find_all('p'):
    print x

提取正确数据的下一步是什么？

最佳答案

更新
这是@ Dan-Dev附带的最终代码。有点荒谬……对不起，我没时间写短代码。

import re
import requests
from bs4 import BeautifulSoup
from pprint import pprint

def is_phone_number(txt):
    r = re.compile(r"(\d{3}[-\.\s]??\d{3}[-\.\s]??\d{4}|\(\d{3}\)\s*\d{3}[-\.\s]??\d{4}|\d{3}[-\.\s]??\d{4})")
    return r.match(txt)

def is_gas_type(txt):
    return any(term in txt.lower() for term in ['lead', 'diesel'])

def is_lat_lon(txt):
    return any(term in txt.lower() for term in ['lat', 'lon'])

def is_hour_of_operation(txt):
    return any(term in txt.lower() for term in ['24 hrs', ' am ', ' pm ', 'm-f'])

def str_colon_list_to_str_float_dict(rlist):
    """["a:1.0", "b:2.0"] => {"a":1.0, "b":2.0}"""
    intermediate_dict = dict(map(lambda s: s.split(':'), rlist))
    return dict((k, float(v)) for k, v in intermediate_dict.iteritems())

page = requests.get("http://www.nj.gov/treasury/administration/statewide-support/motor-fuel-locations.shtml")
soup = BeautifulSoup(page.content, 'html.parser')
table = soup.find_all('table')[5]

gas_stations = []
for x in table.find_all('td', {'valign': 'top', 'colspan': None, 'width': None}):
    gas_station = []
    # split text on line breaks and then remove whitespace
    for y in x.text.splitlines():
        line = ' '.join(y.split())
        gas_station.append(line)
    # skip lines consisting of only empty strings
    if not ('' in set(gas_station) and len(set(gas_station)) == 1):
        gas_stations.append(gas_station)

gas_stations_dict = {}
for gas_station in gas_stations:
    gas_station_dict = {}
    address_list = []
    lat_long_list = []
    for i, g in enumerate(gas_station):
        g = g.encode("utf-8")
        if i == 0:
            gas_station_dict['Name'] = g
        elif is_phone_number(g):
            gas_station_dict['Phone Number'] = g
        elif is_lat_lon(g):
            lat_long_list.append(g)
        elif is_gas_type(g):
            gas_station_dict['Gas Type'] = g
        elif is_hour_of_operation(g):
            gas_station_dict['Hours of Operation'] = g
        else:
            address_list.append(g)
    gas_station_dict['Coordinates'] = str_colon_list_to_str_float_dict(lat_long_list)
    gas_station_dict['Address'] = ' '.join(address_list)

    gas_stations_dict[gas_station_dict['Name']] = gas_station_dict

pprint(gas_stations_dict)

结果：

    {'Bayside Facility': {'Address': '4294 Rt. 47 Leesburg',
                           'Coordinates': {'Latitude': 39.23339997,
                                           'Longitude': -74.96568202},
                           'Gas Type': 'Unleaded / Diesel',
                           'Hours of Operation': 'Open 24 Hrs',
                           'Name': 'Bayside Facility',
                           'Phone Number': '856-785-0040 X-5429'},
     'Bedminster DOT': {'Address': '455 Rt. 202/206 South Pluckemin',
                         'Coordinates': {'Latitude': 40.65123677,
                                         'Longitude': -74.64499021},
                         'Gas Type': 'Unleaded / Diesel',
                         'Hours of Operation': 'Open 24 Hrs',
                         'Name': 'Bedminster DOT',
                         'Phone Number': '908-234-2130'},
        ...
        }

我的老答案：

就像我在评论中提到的那样，我尝试使用Selector Gadget，但是在html中没有找到任何能抓住所有电台名称的一致模式。我抓了很多东西，在很多政府网站上也遇到了同样的麻烦。我不知道这是否是不称职的，还是故意做的以防止刮擦...无论如何，这是一些输出一些信息的代码：

import requests
from bs4 import BeautifulSoup, NavigableString
page=requests.get("http://www.nj.gov/treasury/administration/statewide-support/motor-fuel-locations.shtml")
soup=BeautifulSoup(page.content,'html.parser')
for x in soup.find_all('p'):
    for y in x:
        if isinstance(y, NavigableString):
            print y.encode("utf-8")
        else:
            for z in y:
                if isinstance(z, NavigableString):
                    print z.encode("utf-8")

至此，您可以根据所需信息对其进行修改。只是盯着它，看起来每个站组的最后一行是“经度”。

最后，完成后，我会仔细检查以确保您拥有所需的所有信息。例如，当您输入find_all p标签时，Folsom DOT不会被拉出。

关于python - Web刮HTML加油站，我们在Stack Overflow上找到一个类似的问题：https://stackoverflow.com/questions/49562654/