在python中将多个numpy文件追加到一个大numpy文件

问题描述

我正在尝试放置许多numpy文件以获得一个大的numpy文件，我尝试遵循此链接，这就是我所做的:

I am trying to put many numpy files to get one big numpy file, I tried to follow this link Python append multiple files in given order to one big file and this is what I did:

import matplotlib.pyplot as plt
import numpy as np
import os, sys

#Read in list of files. You might want to look into os.listdir()

path= "/home/user/Desktop/ALLMyTraces.npy/test"
#Test folder contains all my numpy file traces
traces= os.listdir(path)

# Create new File
f = open("/home/user/Desktop/ALLMyTraces.npy", "w")

for j,trace in enumerate(traces):

    # Find the path of the file
    filepath = os.path.join(path, trace)

    # Load file
    dataArray= np.load(filepath)
    f.write(dataArray)

文件已创建，并使用以下代码验证我的内容是否正确:

File is created, and to verify that I have the good contents, I used this code:

import numpy as np
dataArray= np.load(r'/home/user/Desktop/ALLMyTraces.npy')
print(dataArray)

由此产生此错误:

 dataArray= np.load(r'/home/user/Desktop/ALLMyTraces.npy')
  File "/usr/lib/python2.7/dist-packages/numpy/lib/npyio.py", line 401, in load
    "Failed to interpret file %s as a pickle" % repr(file))
IOError: Failed to interpret file '/home/user/Desktop/ALLMyTraces.npy' as a pickle

我真的不知道问题所在.任何帮助将不胜感激.

I don't know really the problem. Any help would be appreciated.

推荐答案

您应使用 numpy.save 或 numpy.savez 创建腌制的.npy或.npz二进制文件. numpy.load()只能读取这些文件.一个>.由于您是使用f.write(dataArray)创建文本文件的，因此np.load()失败并出现上述错误

You should use numpy.save or numpy.savez to create pickled .npy or .npz binary files. Only those file can be read by numpy.load(). Since you are creating a text file using f.write(dataArray), np.load() is failing with the above mentioned error

这是一个样本

fpath ="path to big file"
npyfilespath ='path to nympy files to be merged '
os.chdir(npyfilespath)

with open(fpath, 'wb') as f_handle:
    for npfile in glob.glob("*.npy"):

        # Find the path of the file
        filepath = os.path.join(path, npfile)
        print filepath
        # Load file
        dataArray= np.load(filepath)
        print dataArray
        np.save(f_handle,dataArray)
dataArray= np.load(fpath)
print dataArray

刚刚发现numpy加载中确实有一些有趣的东西.它不会一次加载所有附加数组:).阅读此帖子以获取更多信息.

Just found that there is something really interesting in numpy load. It wont load all append arrays at once :). Read this post for more info.

这意味着，如果要读取所有附加数组，则需要多次加载它们.

This means , if you want to read all appended arrays , you need to load them multiple times.

f = open(fpath, 'rb')
dataArray= np.load(f) #loads first array
print dataArray
dataArray= np.load(f)  #loads Second array
print dataArray
dataArray= np.load(f) #loads Third array
print dataArray

这篇关于在python中将多个numpy文件追加到一个大numpy文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！