我试图在Wiki页面“ https://en.wikipedia.org/wiki/Glossary_of_nautical_terms”上抓取一个列表，以每个航海术语的标题/描述为准，我的第一个问题是正确处理如下描述中的列表：
import requests
from bs4 import BeautifulSoup
url = 'https://en.wikipedia.org/wiki/Glossary_of_nautical_terms'
page = requests.get(url)
get_title = []
get_desc = []
corrected_desc = []
output = ''
if page.status_code == 200:
soup = BeautifulSoup(page.text, 'html.parser')
get_title = soup.find_all('dt', class_='glossary')
get_desc = soup.find_all('dd', class_='glossary')
for i in get_desc:
first_char = i.get_text()[:1]
second_char = i.get_text()[1:2]
if (first_char.isnumeric() and second_char == '.'):
if(first_char == '1' and output):
corrected_desc.append(output)
output = ''
output += '{} '.format(i.get_text())
continue
else:
output += '{} '.format(i.get_text())
continue
if output:
corrected_desc.append(output)
output = ''
corrected_desc.append(i.get_text())
else:
corrected_desc.append(i.get_text())
else:
print('failed to get the page!')
print(str(len(get_title)) + ' - ' + str(len(corrected_desc)))
zipped = zip(get_title, corrected_desc)
for j in zipped:
output = '{}, {}\n'.format(j[0].get_text(), j[1].strip())
with open('test.txt', "a", encoding='utf-8') as myfile:
myfile.write(output)
但是我似乎无法弄清楚如何使用列表和句子来处理描述。
编辑：
我正在寻找的输出是：
"Title", "Description"
"Title", "Description"
"Title", "Description"
"Title", "Description"
但是我不确定如何修改我的代码以处理描述是列表+句子的情况。
最佳答案
所有标题都位于<dt>标记内，描述位于<dd>标记内。因此，第一步是找到所有这些标签。可以使用soup.find_all(['dt', 'dd'])完成。然后，循环这些标签，并使用dt检查标签是dd还是if tag.name == 'dt'。如果标签是dd，则将其内容附加到description变量，否则打印变量的当前内容。
完整的代码：
import requests
from bs4 import BeautifulSoup
r = requests.get('https://en.wikipedia.org/wiki/Glossary_of_nautical_terms')
soup = BeautifulSoup(r.text, 'lxml')
curr_title, curr_description = '', ''
for tag in soup.find_all(['dt', 'dd']):
if tag.name == 'dt':
if curr_title:
print('{}: {}'.format(curr_title, curr_description))
curr_description = ''
curr_title = tag.text.strip()
else:
curr_description = ' '.join((curr_description, tag.text.strip()))
部分输出：
A-back: A foresail when against the wind, used when tacking to help the vessel turn.[1]
Abaft: Toward the stern, relative to some object ("abaft the fore hatch").
Abaft the beam: Further aft than the beam: a relative bearing of greater than 90 degrees from the bow: "two points abaft the beam, starboard side". That would describe "an object lying 22.5 degrees toward the rear of the ship, as measured clockwise from a perpendicular line from the right side, center, of the ship, toward the horizon."[2]
Abandon ship!: An imperative to leave the vessel immediately, usually in the face of some imminent overwhelming danger.[3] It is an order issued by the Master or a delegated person in command. (It must be a verbal order). It is usually the last resort after all other mitigating actions have failed or become impossible, and destruction or loss of the ship is imminent; and customarily followed by a command to "man the lifeboats" or life rafts.[3][4]
Abeam: On the beam, a relative bearing at right angles to the ship's keel.[5]
Able seaman: Also able-bodied seaman. A merchant seaman qualified to perform all routine duties, or a junior rank in some navies.
Aboard: On or in a vessel. Synonymous with "on board." (See also close aboard.)
About: "To go about is to change the course of a ship by tacking. Ready about, or boutship, is the order to prepare for tacking."[6]
Above board: On or above the deck, in plain view, not hiding anything. Pirates would hide their crews below decks, thereby creating the false impression that an encounter with another ship was a casual matter of chance.[7]
Above-water hull: The hull section of a vessel above the waterline, the visible part of a ship. Also, topsides.
Absentee pennant: Special pennant flown to indicate absence of commanding officer, admiral, his chief of staff, or officer whose flag is flying (division, squadron, or flotilla commander).
Absolute bearing: The bearing of an object in relation to north. Either true bearing, using the geographical or true north, or magnetic bearing, using magnetic north. See also bearing and relative bearing.
Accommodation ladder: A portable flight of steps down a ship's side.
Accommodation ship (or accommodation hulk): A ship or hulk used as housing, generally when there is a lack of quarters available ashore. An operational ship can be used, but more commonly a hulk modified for accommodation is used.
Act of Pardon or Act of Grace: A letter from a state or power authorising action by a privateer. See also Letter of marque.
Action Stations: See Battle stations.
Admiral: Senior naval officer of Flag rank. In ascending order of seniority, Rear Admiral, Vice Admiral, Admiral and (until about 2001 when all UK five-star ranks were discontinued) Admiral of the Fleet (Royal Navy). Derivation Arabic, from Amir al-Bahr ("Ruler of the sea").
Admiralty: 1. A high naval authority in charge of a state's Navy or a major territorial component. In the Royal Navy (UK) the Board of Admiralty, executing the office of the Lord High Admiral, promulgates Naval law in the form of Queen's (or King's) Regulations and Admiralty Instructions. 2. Admiralty law
Admiralty law: Body of law that deals with maritime cases. In the UK administered by the Probate, Divorce and Admiralty Division of the High Court of Justice or supreme court.
Adrift: 1. Afloat and unattached in any way to the shore or seabed, but not under way. When referring to a vessel, it implies that the vessel is not under control and therefore goes where the wind and current take her (loose from moorings or out of place). 2. Any gear not fastened down or put away properly. 3. Any person or thing that is misplaced or missing. When applied to a member of the navy or marine corps, such a person is "absent without leave" (AWOL) or, in US Navy and US Marine Corps terminology, is guilty of an "unauthorized absence" (UA).[8]
关于python - BeautifulSoup -抓取Wiki页面，我们在Stack Overflow上找到一个类似的问题：https://stackoverflow.com/questions/49833207/