本文介绍了从一个大的 CSV 文件中读取一个小的随机样本到一个 Python 数据框中的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我想读取的 CSV 文件不适合主内存.如何读取其中的几行(~10K)随机行并对选定的数据框进行一些简单的统计?
The CSV file that I want to read does not fit into main memory. How can I read a few (~10K) random lines of it and do some simple statistics on the selected data frame?
推荐答案
假设 CSV 文件中没有标题:
Assuming no header in the CSV file:
import pandas
import random
n = 1000000 #number of records in file
s = 10000 #desired sample size
filename = "data.txt"
skip = sorted(random.sample(range(n),n-s))
df = pandas.read_csv(filename, skiprows=skip)
如果 read_csv 有一个 keeprows 会更好,或者如果 skiprows 采用回调函数而不是列表.
would be better if read_csv had a keeprows, or if skiprows took a callback func instead of a list.
带有标题和未知文件长度:
With header and unknown file length:
import pandas
import random
filename = "data.txt"
n = sum(1 for line in open(filename)) - 1 #number of records in file (excludes header)
s = 10000 #desired sample size
skip = sorted(random.sample(range(1,n+1),n-s)) #the 0-indexed header will not be included in the skip list
df = pandas.read_csv(filename, skiprows=skip)
这篇关于从一个大的 CSV 文件中读取一个小的随机样本到一个 Python 数据框中的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!