python - 用Python爬行WoS

我正在尝试从WoS（Web of Science）数据库下载信息。我需要信息，例如文章名称，作者，引用时间，数量等


这是我的代码：

import sys
from BeautifulSoup import BeautifulSoup
import urllib
import re
    var = raw_input("Link WoS: ")
    conn = urllib.urlopen(var)
    html = conn.read()
    soup = BeautifulSoup(html)
    titles = re.findall('<value lang_id="">(.+?)</value>',str(soup))
    volume = re.findall('Volume: </span><span class="data_bold"><value>(.+?)</value>', str(soup))
    print(volume)

它非常适合获得标题。但是，我在获取以下信息时遇到问题：数量，问题，页面，日期（已发布）和引用时间。这是网页的来源：

</span><span name="source_title_1" id="source_title_1">
<value>
<span class="hitHilite">EDUCATIONAL RESEARCH</span>
</value>
</span>&nbsp;&nbsp;<span class="label">Volume: </span><span     class="data_bold">
<value>35</value>
</span> &nbsp;&nbsp;<span class="label">Issue: </span><span  class="data_bold">
<value>1</value>
</span> &nbsp;&nbsp;<span class="label">Pages: </span><span class="data_bold">
<value>3-25</value>
</span> &nbsp;&nbsp;<span class="label">Published: </span><span class="data_bold">
<value>SPR 1993</value>
</span>
</div>
<div style="display: inline-block" id="links_1">
<nobr><span id="links_openurl_1"></span> <span id="links_full_text_1">     </span> <span id="links_doc_del_1"></span> <span id="links_patent_1">    </span> </nobr>
</div>
<div class="search-action-item">
<span id="solo_full_text_1" class="solo_full_text"></span><a      name="full_text_1" id="full_text_1" title="Full Text" class="button2link     button-ft" href="javascript:;"><span id="full_text_1" name="full_text_1" title="Full Text" class="button2 button-ft">Full Text</span></a>
<div class="popup-full-text" id="full_text_1_menu">
<span id="full_text_1_links"></span>
</div>
</div>
<script type="text/javascript">$("#full_text_1").hide();</script><span style="display: inline-block" class="button-abstract" id="ViewAbstract1_text"><a title="View Abstract" alt="View Abstract" onclick="return hide_show_abstract('1', 'http://images.webofknowledge.com/WOKRS523R4/images/spacer.gif', 'http://images.webofknowledge.com/WOKRS523R4/images/spacer.gif', 'View Abstract', 'Close Abstract');" href="javascript:;" class="button9"><img align="absmiddle" title="View Abstract" alt="View Abstract" src="http://images.webofknowledge.com/WOKRS523R4/images/spacer.gif" id="ViewAbstract1_img">View Abstract<nobr></nobr></a></span><span style="display: none" class="button-abstract" id="HideAbstract1_text"><a title="Close Abstract" alt="Close Abstract" onclick="return hide_show_abstract('1',  'http://images.webofknowledge.com/WOKRS523R4/images/spacer.gif', 'http://images.webofknowledge.com/WOKRS523R4/images/spacer.gif', 'View Abstract', 'Close Abstract');" href="javascript:;" class="button9"><img align="absmiddle" title="Close Abstract" alt="Close Abstract" src="http://images.webofknowledge.com/WOKRS523R4/images/spacer.gif" id="HideAbstract1_img">Close Abstract<nobr></nobr></a></span><span style="display: none" url="http://apps.webofknowledge.com/ViewAbstract.do?product=WOS&amp;search_mode=GeneralSearch&amp;viewType=ViewAbstract&amp;qid=5&amp;SID=W1tvVEGCvoimqQujw4V&amp;page=1&amp;doc=1" id="ViewAbstract_Span1">
<!----></span></div><div class="search-results-data">
<div class="search-results-data-cite">Times Cited: <a title="View all of the articles that cite this one" href="/CitingArticles.do?product=WOS&amp;SID=W1tvVEGCvoimqQujw4V&amp;search_mode=CitingArticles&amp;parentProduct=WOS&amp;parentQid=5&amp;parentDoc=1&amp;REFID=448550&amp;excludeEventConfig=ExcludeIfFromNonInterProduct">487</a>
<br>

我认为我有问题，因为数据是数字的...请您能帮我吗？

最佳答案

BeautifulSoup将为您做很多繁重的工作。正则表达式通常是涉及HTML的不得已的方法。如下面的代码所示，最好也使用此产品的最新版本。

HTML = '''\
<value>
<span class="htmllite">EDUCATIONAL RESEARCH</span>
</value>
</span>&nbsp;&nbsp;<span class="label">Volume: </span><span class="data_bold">
<value>29</value>
</span>&nbsp;&nbsp;<span class="label">Issue: </span><span class="data_bold">
<value>2</value>
</span>&nbsp;&nbsp;<span class="label">Pages: </span><span class="data_bold">
<value>26-152</value>
</span>&nbsp;&nbsp;<span class="label">Published: </span><span class="data_bold">
<value>JUN 1987</value>
'''

from bs4 import BeautifulSoup

soup = BeautifulSoup(HTML, "html.parser")

items = soup.findAll('span', attrs = {'class': 'label' })
for item in items:
    label = item.contents[0]
    sibling = item.find_next_sibling('span')
    value = sibling.select('value')[0].text
    print (label, value )

结果：

Volume:  29
Issue:  2
Pages:  26-152
Published:  JUN 1987

我还没有足够的聪明才能够编写此书而不尝试一些失败的可能性。您是否正在使用诸如IDLE之类的方法来建议替代方案，并尝试一些代码片段以查看其产生的结果？

PS：当您再次回到SO时，请以文本形式发布HTML和其他文本（而不是图像文件），以便答复者可以使用剪切和粘贴功能。

关于python - 用Python爬行WoS，我们在Stack Overflow上找到一个类似的问题：https://stackoverflow.com/questions/41065038/