我正在从此页面抓取“普通财富游戏”奖牌计数：https://en.wikipedia.org/wiki/1930_British_Empire_Games
抓取数据后，我想移至下一页。为此，我想选择一个具有<table>的attribute ID '#collapsibleTable1'标记。
现在是有趣的部分。在chrome控制台上执行$('#collapsibleTable1')时，得到所需的输出。
但是，当我尝试在scrapy shell中执行response.css('#collapsibleTable1')时，它给出了一个空列表。
如果有人可以解释为什么这样做会很有帮助。
最佳答案
似乎发生了一些JavaScript操作，因为该ID并未包含在实际的HTML源代码中（您可以通过print(response.text)查看）
Chrome的开发人员工具会在所有JavaScript执行完毕后显示DOM的当前状态，这并不是Scrapy所能看到的。
查看源，所需的数据显示为：
<table class="nowraplinks collapsible autocollapse navbox-inner" style="border-spacing:0;background:transparent;color:inherit">
<tr>
<th scope="col" class="navbox-title" colspan="2">
<div class="plainlinks hlist navbar mini">
<ul>
<li class="nv-view"><a href="/wiki/Template:Commonwealth_Games_Medal_Counts" title="Template:Commonwealth Games Medal Counts"><abbr title="View this template" style=";;background:none transparent;border:none;-moz-box-shadow:none;-webkit-box-shadow:none;box-shadow:none;">v</abbr></a></li>
<li class="nv-talk"><a href="/wiki/Template_talk:Commonwealth_Games_Medal_Counts" title="Template talk:Commonwealth Games Medal Counts"><abbr title="Discuss this template" style=";;background:none transparent;border:none;-moz-box-shadow:none;-webkit-box-shadow:none;box-shadow:none;">t</abbr></a></li>
<li class="nv-edit"><a class="external text" href="//en.wikipedia.org/w/index.php?title=Template:Commonwealth_Games_Medal_Counts&action=edit"><abbr title="Edit this template" style=";;background:none transparent;border:none;-moz-box-shadow:none;-webkit-box-shadow:none;box-shadow:none;">e</abbr></a></li>
</ul>
</div>
<div id="Commonwealth_Games_medal_tables" style="font-size:114%;margin:0 4em"><a href="/wiki/All-time_Commonwealth_Games_medal_table" title="All-time Commonwealth Games medal table">Commonwealth Games medal tables</a></div>
</th>
</tr>
<tr>
<td colspan="2" class="navbox-list navbox-odd hlist" style="width:100%;padding:0px">
<div style="padding:0em 0.25em">
<ul>
<li><a href="/wiki/1930_British_Empire_Games#Medal_table" title="1930 British Empire Games">1930</a></li>
<li><a href="/wiki/1934_British_Empire_Games#Medals_by_country" title="1934 British Empire Games">1934</a></li>
<li><a href="/wiki/1938_British_Empire_Games#Medals_by_country" title="1938 British Empire Games">1938</a></li>
<li><a href="/wiki/1950_British_Empire_Games#Medals_by_country" title="1950 British Empire Games">1950</a></li>
<li><a href="/wiki/1954_British_Empire_and_Commonwealth_Games#Medal_table" title="1954 British Empire and Commonwealth Games">1954</a></li>
<li><a href="/wiki/1958_British_Empire_and_Commonwealth_Games#Medals_by_country" title="1958 British Empire and Commonwealth Games">1958</a></li>
<li><a href="/wiki/1962_British_Empire_and_Commonwealth_Games#Medals_by_country" title="1962 British Empire and Commonwealth Games">1962</a></li>
<li><a href="/wiki/1966_British_Empire_and_Commonwealth_Games#Medals_by_country" title="1966 British Empire and Commonwealth Games">1966</a></li>
<li><a href="/wiki/1970_British_Commonwealth_Games#Medals_by_country" title="1970 British Commonwealth Games">1970</a></li>
<li><a href="/wiki/1974_British_Commonwealth_Games#Medals_by_country" title="1974 British Commonwealth Games">1974</a></li>
<li><a href="/wiki/1978_Commonwealth_Games#Medals_by_country" title="1978 Commonwealth Games">1978</a></li>
<li><a href="/wiki/1982_Commonwealth_Games#Medals_by_country" title="1982 Commonwealth Games">1982</a></li>
<li><a href="/wiki/1986_Commonwealth_Games#Medals_by_country" title="1986 Commonwealth Games">1986</a></li>
<li><a href="/wiki/1990_Commonwealth_Games#Medals_by_country" title="1990 Commonwealth Games">1990</a></li>
<li><a href="/wiki/1994_Commonwealth_Games#Medal_table" title="1994 Commonwealth Games">1994</a></li>
<li><a href="/wiki/1998_Commonwealth_Games#Medal_table" title="1998 Commonwealth Games">1998</a></li>
<li><a href="/wiki/2002_Commonwealth_Games#Final_medal_table" title="2002 Commonwealth Games">2002</a></li>
<li><a href="/wiki/2006_Commonwealth_Games_medal_table" title="2006 Commonwealth Games medal table">2006</a></li>
<li><a href="/wiki/2010_Commonwealth_Games_medal_table" title="2010 Commonwealth Games medal table">2010</a></li>
<li><a href="/wiki/2014_Commonwealth_Games_medal_table" title="2014 Commonwealth Games medal table">2014</a></li>
<li><a href="/wiki/2018_Commonwealth_Games_medal_table" title="2018 Commonwealth Games medal table">2018</a></li>
</ul>
</div>
</td>
</tr>
</table>