使用python ijson读取具有多个json对象的大型json文件

问题描述

我正在尝试使用ijson包解析一个大(〜100MB)的json文件，该包允许我以一种有效的方式与该文件进行交互.但是，在编写了这样的代码之后，

I'm trying to parse a large (~100MB) json file using ijson package which allows me to interact with the file in an efficient way. However, after writing some code like this,

with open(filename, 'r') as f:
    parser = ijson.parse(f)
    for prefix, event, value in parser:
        if prefix == "name":
            print(value)

我发现代码仅解析文件的第一行，而不解析其余行！

I found that the code parses only the first line and not the rest of the lines from the file!!

这是我的json文件的一部分的样子:

Here is how a portion of my json file looks like:

{"name":"accelerator_pedal_position","value":0,"timestamp":1364323939.012000}
{"name":"engine_speed","value":772,"timestamp":1364323939.027000}
{"name":"vehicle_speed","value":0,"timestamp":1364323939.029000}
{"name":"accelerator_pedal_position","value":0,"timestamp":1364323939.035000}

我认为ijson仅解析一个json对象.

In my opinion, I think ijson parses only one json object.

有人可以建议如何解决此问题吗?

Can someone please suggest how to work around this?

推荐答案

由于所提供的块看起来更像是一组行，每行组成一个独立的JSON，因此应相应地对其进行解析:

Since the provided chunk looks more like a set of lines each composing an independent JSON, it should be parsed accordingly:

# each JSON is small, there's no need in iterative processing
import json
with open(filename, 'r') as f:
    for line in f:
        data = json.loads(line)
        # data[u'name'], data[u'engine_speed'], data[u'timestamp'] now
        # contain correspoding values

这篇关于使用python ijson读取具有多个json对象的大型json文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！