what reliable method to save huge numpy arrays

Vote count:

0

I saved some arrays using numpy.savez_compressed(). One of the arrays is gigantic, it has the shape (120000,7680), type float32.

Trying to load the array gave me the error below (message caught using Ipython).

Is seems like this is a Numpy limitation: Numpy: apparent memory error

What are other ways to save such a huge array? (I had problems with cPickle as well)

In [5]: t=numpy.load('humongous.npz')
In [6]: humg = (t['arr_0.npy'])


/usr/lib/python2.7/dist-packages/numpy/lib/npyio.pyc in __getitem__(self, key)
229             if bytes.startswith(format.MAGIC_PREFIX):
230                 value = BytesIO(bytes)
--> 231                 return format.read_array(value)
232             else:
233                 return bytes

/usr/lib/python2.7/dist-packages/numpy/lib/format.pyc in read_array(fp)
456             # way.

457             # XXX: we can probably chunk this to avoid the memory hit.

--> 458             data = fp.read(int(count * dtype.itemsize))
459             array = numpy.fromstring(data, dtype=dtype, count=count)
460 

SystemError: error return without exception set

System: Ubuntu 12.04 64 bit, Python 2.7, numpy 1.6.1

asked 3 mins ago

ng0323

26

what reliable method to save huge numpy arrays