我正在使用Beutifulsoup 4和Python 3.5+提取Web数据。我从中提取以下html:
<div class="the-one-i-want">
<p>
content
</p>
<p>
content
</p>
<p>
content
</p>
<p>
content
</p>
<ol>
<li>
list item
</li>
<li>
list item
</li>
</ol>
<div class='something-i-don't-want>
content
</div>
<script class="something-else-i-dont-want'>
script
</script>
<p>
content
</p>
</div>
我要提取的所有内容都在
<div class="the-one-i-want">
元素中找到。现在,我正在使用以下大多数时间都有效的方法:soup = Beautifulsoup(html.text, 'lxml')
content = soup.find('div', class_='the-one-i-want').findAll('p')
这不包括脚本,奇怪的插入
div
以及其他不可预测的内容,例如广告或“推荐内容”类型的内容。现在,在某些实例中,除了
<p>
标记之外,还有其他元素,这些元素具有对主要内容在上下文上很重要的内容,例如列表。有没有一种方法可以从
<div class="the-one-i-want">
中获取内容,例如:soup = Beautifulsoup(html.text, 'lxml')
content = soup.find('div', class_='the-one-i-want').findAll(desired-content-elements)
desired-content-elements
应包含我认为适合该特定内容的每个元素吗?例如,所有<p>
标签,所有<ol>
和<li>
标签,但没有<div>
或<script>
标签。也许值得注意的是我保存内容的方法:
content_string = ''
for p in content:
content_string += str(p)
这种方法按出现的顺序收集数据,如果我只是通过不同的迭代过程简单地找到了不同的元素类型,那么将证明难以管理。我希望不必管理拆分列表的重新构建,以尽可能地重新组合每个元素最初出现在内容中的顺序。
最佳答案
您可以传递所需的标签列表:
content = soup.find('div', class_='the-one-i-want').find_all(["p", "ol", "whatever"])
如果我们在您的问题网址上运行类似的内容以查找p和pre标记,则可以看到我们同时获得了两者:
...: for ele in soup.select_one("td.postcell").find_all(["pre","p"]):
...: print(ele)
...:
<p>I'm using Beutifulsoup 4 and Python 3.5+ to extract webdata. I have the following html, from which I am extracting:</p>
<pre><code><div class="the-one-i-want">
<p>
content
</p>
<p>
content
</p>
<p>
content
</p>
<p>
content
</p>
<ol>
<li>
list item
</li>
<li>
list item
</li>
</ol>
<div class='something-i-don't-want>
content
</div>
<script class="something-else-i-dont-want'>
script
</script>
<p>
content
</p>
</div>
</code></pre>
<p>All of the content that I want to extract is found within the <code><div class="the-one-i-want"></code> element. Right now, I'm using the following methods, which work most of the time:</p>
<pre><code>soup = Beautifulsoup(html.text, 'lxml')
content = soup.find('div', class_='the-one-i-want').findAll('p')
</code></pre>
<p>This excludes scripts, weird insert <code>div</code>'s and otherwise un-predictable content such as ads or 'recommended content' type stuff.</p>
<p>Now, there are some instances in which there are elements other than just the <code><p></code> tags, which has content that is contextually important to the main content, such as lists.</p>
<p>Is there a way to get the content from the <code><div class="the-one-i-want"></code> in a manner as such:</p>
<pre><code>soup = Beautifulsoup(html.text, 'lxml')
content = soup.find('div', class_='the-one-i-want').findAll(desired-content-elements)
</code></pre>
<p>Where <code>desired-content-elements</code>would be inclusive of every element that I deemed fit for that particular content? Such as, all <code><p></code> tags, all <code><ol></code> and <code><li></code> tags, but no <code><div></code> or <code><script></code> tags.</p>
<p>Perhaps noteworthy, is my method of saving the content:</p>
<pre><code>content_string = ''
for p in content:
content_string += str(p)
</code></pre>
<p>This approach collects the data, in order of occurrence, which would prove difficult to manage if I simply found different element types through different iteration processes. I'm looking to NOT have to manage re-construction of split lists to re-assemble the order in which each element originally occurred in the content, if possible.</p>
关于python - 使用Beautifulsoup,提取指定元素以外的元素标签,我们在Stack Overflow上找到一个类似的问题:https://stackoverflow.com/questions/38507514/