python - 从多个网页收集格式化的内容

我正在做一个研究项目，需要节目成绩单的内容作为数据。问题是，成绩单是为特定的wiki（Arrested Development wiki）格式化的，而我需要它们是机器可读的。

下载所有这些成绩单并将其重新格式化的最佳方法是什么？ Python's HTMLParser是我最好的选择吗？

最佳答案

我用python编写了一个脚本，该脚本将Wiki脚本的链接作为输入，然后在文本文件中为您提供脚本的纯文本版本作为输出。希望对您的项目有所帮助。

from pycurl import *
import cStringIO
import re

link = raw_input("Link to transcript: ")
filename = link.split("/")[-1]+".txt"

buf = cStringIO.StringIO()

c = Curl()
c.setopt(c.URL, link)
c.setopt(c.WRITEFUNCTION, buf.write)
c.perform()
html = buf.getvalue()
buf.close()

Speaker = ""
SpeakerPositions = [m.start() for m in re.finditer(':</b>', html)]

file = open(filename, 'w')

for x in range(0, len(SpeakerPositions)):
    if html[SpeakerPositions[x] + 5] != "<":

        searchpos = SpeakerPositions[x] - 1
        char = ""
        while char != ">":
            char = html[searchpos]
            searchpos = searchpos - 1
            if char != ">":
                Speaker += char

        Speaker = Speaker[::-1]
        Speaker += ": "

        searchpos = SpeakerPositions[x] + 5
        char = ""
        while char != "<":
            char = html[searchpos]
            searchpos = searchpos + 1
            if char != "<":
                Speaker += char

        Speaker = Speaker.replace("&#160;", "")
        file.write(Speaker + "\n")
        Speaker = ""

file.close()