python - 从文本格式读取数据到Python Pandas数据框

我在Windows上运行Python 2.7。
我有一个大的文本文件（2 GB），涉及500多封电子邮件。文件没有显式文件类型，格式为：

email_message#: 1
email_message_sent: 10/10/1991 02:31:01
From: [e-mail]| Tom Foo |abc company|
To: [e-mail]| Alex Dee |abc company|
To: [e-mail]| Ben For |xyz company|
email_message#: 2
email_message_sent: 10/12/1991 01:28:12
From: [e-mail]| Tim Tee |abc company|
To: [e-mail]| Tom Foo |abc company|
To: [e-mail]| Alex Dee |abc company|
To: [e-mail]| Ben For|xyz company|
email_message#: 3
email_message_sent: 10/13/1991 12:01:16
From: [e-mail]| Ben For |xyz company|
To: [e-mail]| Tom Foo |abc company|
To: [e-mail]| Tatiana Xocarsky |numbers firm |
...

如您所见，每个电子邮件都有以下相关数据：
1）发送时间
2）发送邮件的电子邮件地址
3）发送人姓名
4）为其工作的公司
5）收到电子邮件的每个电子邮件地址
6）收到电子邮件的每个人的姓名
7）收到电子邮件的每个人的公司
在文本文件中有50万封以上的电子邮件，电子邮件可以有多达16千个收件人。在电子邮件中，没有关于他们如何称呼工作人员或公司的模式。
我想取这个大文件并在python中操作它，使它最终成为PandasDataframe。我希望pandasdataframe的格式与下面excel的屏幕截图类似：

编辑
我的解决方案是编写一个“解析器”，它接受这个文本文件并读取每一行，将每一行中的文本分配给pandasdataframe的特定列。
我打算写一些像下面这样的东西。有人能确认这是正确的执行方式吗？我想确保我没有丢失内置的pandas函数或其他module函数。

#connect to object
data = open('.../Emails', 'r')

#build empty dataframe
import pandas as pd
df = pd.DataFrame()

#function to read lines of the object and put pieces of text into the
# correct column of the dataframe
for line in data:
     n = data.readline()
    if n.startswith("email_message#:"):
        #put a slice of the text into a dataframe
    elif n.startswith("email_message_sent:"):
        #put a slice of the text into a dataframe
    elif n.startswith("From:"):
        #put slices of the text into a dataframe
    elif n.startswith("To:"):
        #put slices of the text into a dataframe

最佳答案

我忍不住发痒，所以这是我的方法。

from __future__ import unicode_literals

import io

import pandas as pd
from pandas.compat import string_types


def iter_fields(buf):
    for l in buf:
        yield l.rstrip('\n\r').split(':', 1)


def iter_messages(buf):
    it = iter_fields(buf)
    k, v = next(it)
    while True:
        n = int(v)
        _, v = next(it)
        date = pd.Timestamp(v)
        _, v = next(it)
        from_add, from_name, from_comp = v.split('|')[:-1]
        k, v = next(it)
        to = []
        while k == 'To':
            to_add, to_name, to_comp = v.split('|')[:-1]
            yield (n, date, from_add[1:], from_name[1:-1], from_comp,
                   to_add[1:], to_name[1:-1], to_comp)
            k, v = next(it)

    if not hasattr(filepath_or_buffer, read):
        filepath_or_buffer


def _read_email_headers(buf):
    columns=['email_message#', 'email_message_sent',
             'from_address', 'from_name', 'from_company',
             'to_address', 'to_name', 'to_company']
    return pd.DataFrame(iter_messages(buf), columns=columns)


def read_email_headers(path_or_buf):
    close_buf = False
    if isinstance(path_or_buf, string_types):
        path_or_buf = io.open(path_or_buf)
        close_buf = True
    try:
        return _read_email_headers(path_or_buf)
    finally:
        if close_buf:
            path_or_buf.close

以下是您将如何使用它：

df = read_email_headers('.../data_file')

只要用文件的路径调用它，就有了数据帧。
现在，以下内容仅用于测试目的。你这样做并不是为了在现实生活中处理你的实际数据。
由于我（或随机StackOverflow读取器）没有您的文件副本，我必须使用字符串来伪造它：

text = '''email_message#: 1
email_message_sent: 10/10/1991 02:31:01
From: [e-mail]| Tom Foo |abc company|
To: [e-mail]| Alex Dee |abc company|
To: [e-mail]| Ben For |xyz company|
email_message#: 2
email_message_sent: 10/12/1991 01:28:12
From: [e-mail]| Tim Tee |abc company|
To: [e-mail]| Tom Foo |abc company|
To: [e-mail]| Alex Dee |abc company|
To: [e-mail]| Ben For|xyz company|'''

然后我可以创建一个类似文件的对象并将其传递给函数：

df = read_email_headers(io.StringIO(text))
print(df.to_string())

   email_message#  email_message_sent  from_address from_name from_company        to_address   to_name   to_company
0               1 1991-10-10 02:31:01  [e-mail]   Tom Foo  abc company      [e-mail]  Alex Dee  abc company
1               1 1991-10-10 02:31:01  [e-mail]   Tom Foo  abc company  [e-mail]   Ben For  xyz company
2               2 1991-10-12 01:28:12  [e-mail]   Tim Tee  abc company      [e-mail]   Tom Foo  abc company
3               2 1991-10-12 01:28:12  [e-mail]   Tim Tee  abc company      [e-mail]  Alex Dee  abc company
4               2 1991-10-12 01:28:12  [e-mail]   Tim Tee  abc company  [e-mail]    Ben Fo  xyz company

或者，如果我想使用一个实际的文件：

with io.open('test_file.txt', 'w') as f:
    f.write(text)

df = read_email_headers('test_file.txt')
print(df.to_string())  # Same output as before.

但是，同样，您不必这样做就可以对数据使用函数。只要用文件路径调用它。

关于python - 从文本格式读取数据到Python Pandas数据框，我们在Stack Overflow上找到一个类似的问题：https://stackoverflow.com/questions/35158954/