如何使用scrapy的XmlFeedSpider解析sitemap.xml文件?

问题描述

我正在尝试使用scrapy解析sitemap.xml文件，站点地图文件就像下面的文件，只是具有更多的url节点.

I am trying to parse sitemap.xml files using scrapy, the sitemap files are like the following one with just much more url nodes.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
        xmlns:video="http://www.sitemaps.org/schemas/sitemap-video/1.1">
    <url>
        <loc>
            http://www.site.com/page.html
        </loc>
        <video:video>
            <video:thumbnail_loc>
                http://www.site.com/thumb.jpg
            </video:thumbnail_loc>
            <video:content_loc>http://www.example.com/video123.flv</video:content_loc>
            <video:player_loc allow_embed="yes" autoplay="ap=1">
                http://www.example.com/videoplayer.swf?video=123
            </video:player_loc>
            <video:title>here is the page title</video:title>
            <video:description>and an awesome description</video:description>
            <video:duration>302</video:duration>
            <video:publication_date>2011-02-24T02:03:43+02:00</video:publication_date>
            <video:tag>w00t</video:tag>
            <video:tag>awesome</video:tag>
            <video:tag>omgwtfbbq</video:tag>
            <video:tag>kthxby</video:tag>
        </video:video>
    </url>
</urlset>

我查看了相关的 scrapy的文档，并编写了以下代码段看看我是否做对了(看来我不^^):

I looked at the related scrapy's documentation, and i wrote the following snippet to see if i was doing the right way (and it seems i don't ^^):

class SitemapSpider(XMLFeedSpider):
    name = "sitemap"
    namespaces = [
        ('', 'http://www.sitemaps.org/schemas/sitemap/0.9'),
        ('video', 'http://www.sitemaps.org/schemas/sitemap-video/1.1'),
    ]
    start_urls = ["http://example.com/sitemap.xml"]
    itertag = 'url'

    def parse_node(self, response, node):
        print "Parsing: %s" % str(node)

但是当我运行蜘蛛时，出现此错误:

But when i run the spider, i get this error:

File "/.../python2.7/site-packages/scrapy/utils/iterators.py", line 32, in xmliter
    yield XmlXPathSelector(text=nodetext).select('//' + nodename)[0]
    exceptions.IndexError: list index out of range

我认为我没有定义默认"名称空间( http://www.sitemaps .org/schemas/sitemap/0.9 )，但我找不到执行该操作的方法.

I think i'm not defining the "default" namespace (http://www.sitemaps.org/schemas/sitemap/0.9) properly, but i can't find how to do this.

迭代url节点然后能够从其子节点提取所需信息的正确方法是什么?

What's the correct way to iterate over the url nodes and then be able to extract the needed infos from its childs?

答案:

不幸的是，我无法使用XMLFeedSpider(这应该是使用scrapy解析XML的方法)，但是由于simplebias的回答，我得以找到一种实现方法这种老派方式".我想出了以下代码(这次有效！):

Unfortunately, i wasn't able to use the XMLFeedSpider (which is supposed to be the way to parse XML with scrapy), but thanks to simplebias' answer, i have been able to figure a way to achieve this "the old-school way". I came up with the following code (which works, this time!):

class SitemapSpider(BaseSpider):
    name = 'sitemap'
    namespaces = {
        'sitemap': 'http://www.sitemaps.org/schemas/sitemap/0.9',
        'video': 'http://www.sitemaps.org/schemas/sitemap-video/1.1',
    }

    def parse(self, response):
        xxs = XmlXPathSelector(response)
        for namespace, schema in self.namespaces.iteritems():
            xxs.register_namespace(namespace, schema)
        for urlnode in xxs.select('//sitemap:url'):
            extract_datas_here()

推荐答案

Scrapy在后台使用了 lxml/libxml2 ，最终调用了node.xpath()方法来执行选择. xpath表达式中任何带有名称空间的元素都必须加上前缀，并且必须传递一个映射来告诉选择器每个前缀都解析为哪个名称空间.

Scrapy uses lxml / libxml2 under the hood, eventually invoking the node.xpath() method to perform the selection. Any elements in your xpath expression which are namespaced must be prefixed, and you must pass a mapping to tell the selector which namespace each prefix resolves to.

下面是一个示例，说明使用node.xpath()方法时如何将前缀映射到名称空间:

Here is an example to illustrate how to map prefixes to namespaces when using the node.xpath() method:

doc = '<root xmlns="chaos"><bar /></root>'
tree = lxml.etree.fromstring(doc)
tree.xpath('//bar')
[]
tree.xpath('//x:bar', namespaces={'x': 'chaos'})
[<Element {chaos}bar at 7fa40f9c50a8>]

如果没有使用这个令人毛骨悚然的XMLFeedSpider类，我猜测您的名称空间映射和itertag需要遵循相同的方案:

Without having used this scrapy XMLFeedSpider class, I'm guessing your namespace map and itertag need to follow the same scheme:

class SitemapSpider(XMLFeedSpider):
    namespaces = [
        ('sm', 'http://www.sitemaps.org/schemas/sitemap/0.9'),
        ]
     itertag = 'sm:url'

这篇关于如何使用scrapy的XmlFeedSpider解析sitemap.xml文件?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！