python - 带有objectify的lxml iterparse

如何解析大XML文件并将其元素作为ObjectifiedElement处理（使用objectify解析器）。

我没有找到比以下更好的解决方案：

from lxml import etree, objectify
for event, elt in etree.iterparse('onebigfile.xml', tag='MyTag'):
    oelt = objectify.fromstring(etree.tostring(elt))
    my_process(oelt)

如何避免这种中间字符串表示形式？

最佳答案

我认为使用iterparse来构建自定义数据提取器真的很容易，它完全消除了使用objectify的需要。

为了这个示例，我使用了一个看起来像这样的.NET参考XML文件：

<doc>
  <assembly>
    <name>System.IO</name>
  </assembly>
  <members>
    <member name="T:System.IO.BinaryReader">
      <summary>Reads primitive data types as binary values in a specific encoding.</summary>
      <filterpriority>2</filterpriority>
    </member>
    <member name="M:System.IO.BinaryReader.#ctor(System.IO.Stream)">
      <summary>Initializes a new instance of the <see cref="T:System.IO.BinaryReader" /> class based on the specified stream and using UTF-8 encoding.</summary>
      <param name="input">The input stream. </param>
      <exception cref="T:System.ArgumentException">The stream does not support reading, is null, or is already closed. </exception>
    </member>
    <member name="M:System.IO.BinaryReader.#ctor(System.IO.Stream,System.Text.Encoding)">
      <summary>Initializes a new instance of the <see cref="T:System.IO.BinaryReader" /> class based on the specified stream and character encoding.</summary>
      <param name="input">The input stream. </param>
      <param name="encoding">The character encoding to use. </param>
      <exception cref="T:System.ArgumentException">The stream does not support reading, is null, or is already closed. </exception>
      <exception cref="T:System.ArgumentNullException">
        <paramref name="encoding" /> is null. </exception>
    </member>
    <!-- ... many more members like this -->
  </members>
</doc>

假设您要提取所有成员及其名称，摘要和属性，作为像这样的字典列表：

{
  'summary': 'Reads primitive data types as binary values in a specific encoding.',
  'name': 'T:System.IO.BinaryReader'
}
{
  'summary': 'Initializes a new instance of the ',
  '@input': 'The input stream. ',
  'name': 'M:System.IO.BinaryReader.#ctor(System.IO.Stream)'
}
{
  'summary': 'Initializes a new instance of the class based on the specified stream and using UTF-8 encoding.',
  '@input': 'The input stream. ',
  '@encoding': 'The character encoding to use. ',
  'name': 'M:System.IO.BinaryReader.#ctor(System.IO.Stream,System.Text.Encoding)'
}

您可以这样做：

将lxml.iterparse与start和end事件一起使用
<member>元素启动时，准备一个新字典（item）
当我们在<member>元素中时，将我们感兴趣的所有内容添加到字典中
当<member>元素结束时，完成字典并产生它
将item设置为None用作“ <member>的内部/外部”标志

在代码中：

import lxml
from lxml import etree

def text_content(elt):
    return ' '.join([t.strip() for t in elt.itertext()])

def extract_data(xmlfile):
    item = None

    for event, elt in etree.iterparse(xmlfile, events=['start', 'end']):
        if elt.tag == 'member':
            if event == 'start':
                item = {}
            else:
                item['name'] = elt.attrib['name']
                yield item
                item = None

        if item == None:
            continue

        if event == 'end':
            if elt.tag in ('summary', 'returns'):
                item[elt.tag] = text_content(elt)
                continue

            if elt.tag == 'param':
                item['@' + elt.attrib['name']] = text_content(elt)
                continue


testfile = r'C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETCore\v4.5.1\System.IO.xml'

for item in extract_data(testfile):
    print(item)

这样，您可以获得最快，最节省内存的解析，并且可以很好地控制要查看的数据。即使没有中间objectify / tostring()，使用fromstring()也会比这更浪费。

关于python - 带有objectify的lxml iterparse，我们在Stack Overflow上找到一个类似的问题：https://stackoverflow.com/questions/49880545/

Objectify

python - 带有objectify的lxml iterparse