我试图打开一个大的IDL生成的拟合数据立方体(159,2,4096,4096):
In [37]: hdulist = fits.open('/randpath/randname1.fits')
In [38]: hdulist.info()
Filename: /randpath/randname1.fits
No. Name Type Cards Dimensions Format
0 PRIMARY PrimaryHDU 11 (159, 2, 4096, 4096) float32
In [39]: scidata = hdulist[0].data
发生以下错误:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-39-d492d4e07eb1> in <module>()
----> 1 scidata = hdulist[0].data
/opt/local/anaconda/anaconda-2.2.0/lib/python2.7/site-packages/astropy/utils/decorators.py in __get__(self, obj, owner)
513 return obj.__dict__[self._key]
514 except KeyError:
--> 515 val = self.fget(obj)
516 obj.__dict__[self._key] = val
517 return val
/opt/local/anaconda/anaconda-2.2.0/lib/python2.7/site-packages/astropy/io/fits/hdu/image.py in data(self)
206 return
207
--> 208 data = self._get_scaled_image_data(self._data_offset, self.shape)
209 self._update_header_scale_info(data.dtype)
210
/opt/local/anaconda/anaconda-2.2.0/lib/python2.7/site-packages/astropy/io/fits/hdu/image.py in _get_scaled_image_data(self, offset, shape)
619 code = BITPIX2DTYPE[self._orig_bitpix]
620
--> 621 raw_data = self._get_raw_data(shape, code, offset)
622 raw_data.dtype = raw_data.dtype.newbyteorder('>')
623
/opt/local/anaconda/anaconda-2.2.0/lib/python2.7/site-packages/astropy/io/fits/hdu/base.py in _get_raw_data(self, shape, code, offset)
566 offset=offset)
567 elif self._file:
--> 568 return self._file.readarray(offset=offset, dtype=code, shape=shape)
569 else:
570 return None
/opt/local/anaconda/anaconda-2.2.0/lib/python2.7/site-packages/astropy/io/fits/file.py in readarray(self, size, offset, dtype, shape)
272
273 return np.ndarray(shape=shape, dtype=dtype, offset=offset,
--> 274 buffer=self._mmap)
275 else:
276 count = reduce(lambda x, y: x * y, shape)
TypeError: buffer is too small for requested array
平均数组(2,4096,4096)效果很好:
In [40]: hdulist2 = fits.open('/randpath/randname1avg.fits')
In [41]: hdulist2.info()
Filename: /randpath/randname1avg.fits
No. Name Type Cards Dimensions Format
0 PRIMARY PrimaryHDU 10 (2, 4096, 4096) float32
In [42]: scidata2 = hdulist2[0].data
有任何想法吗?似乎大小由于某些原因很重要。 MATLAB无法打开第一个拟合文件:
Warning: Seek failed, 'Offset is bad - after end-of-file or last character written.'. File may be an
invalid FITS file or corrupt. Output structure may not contain complete file information.
> In fitsinfo>skipHduData (line 721)
In fitsinfo (line 226)
In fitsread (line 99)
Error using fitsiolib
CFITSIO library error (108): error reading from FITS file
Error in matlab.io.fits.readImg (line 85)
imgdata = fitsiolib('read_subset',fptr,fpixel,lpixel,inc);
Error in fitsread>read_image_hdu (line 438)
data = fits.readImg(fptr);
Error in fitsread (line 125)
data = read_image_hdu(info,1,raw,pixelRegion);
出于不清楚的原因,IDL可以。运行astropy.io工作流程时,数组大小是否有限制?可以毫无问题地生成相同大小的随机矩阵。我目前正在使用具有256 GB RAM的计算机,因此内存不应该起作用吗?感谢您的所有帮助!
更新:第一次加载hdulist时,Python实际上给出了一些更有用的错误消息:
警告:文件可能已被截断:实际文件长度(4160755136)小于预期的大小(21340624320)[astropy.io.fits.file]
实际上,文件大小仅为3.9 GB,与预期的〜20 GB相反。我必须仔细检查(对IDL没有太多经验),但由于某种原因,它(writefits)无法正确创建fits文件。
更新2:问题已解决。 IDL 6.2(计算机上安装的旧版本)显然不能处理太大的文件,IDL 8.3(也已安装)可以处理太大的文件。不知道为什么。
最佳答案
该问题与IDL 6.2有关,并且不再出现在IDL 8.3中。因此,可以通过使用当前的IDL版本来生成fits文件来避免这种情况。
关于python - Python开放大适合数据缓冲区大小错误,我们在Stack Overflow上找到一个类似的问题:https://stackoverflow.com/questions/40385209/