9 ms·
Loading NumPy arrays from disk: mmap() vs. Zarr/HDF5
- butterthebuddha 7y agommap is great. I worked on a project a couple months ago where we had ~20 threads concurrently fetching data over the wire and dumping it into a file. We just ftruncated the file to ~4TB (bigger than we'd ever need it to be) and mmapped'd it all into memory, initializing the first 8 bytes to be an atomic integer that kept track of the actual size. The data needed to end up on disk anyway, and we were more than happy to let the OS take care of that us. We also had very little random access (mostly sequential writes) so it worked really well.
- beagle3 7y agoDid you ever down-truncate it? Did your backup program take forever to crunch through non-existent 4TB (and take as much space to restore?) Atomic access wise, that’s fine as long as you don’t map it through a network file system if any sort; also, $DIETY help you recover a consistent state if the power goes out or you kernel panic Mmap is awesome, I use it all the time. But writing into mmaps robustly is hard. Best example to learn from I am familiar with is symas LMDB source code.
- dekhn 7y agoUnwritten pages in files that are extended don't actually take up space or appear as blocks; a good backup program should respect that. See https://en.wikipedia.org/wiki/Sparse_file https://en.wikipedia.org/wiki/Sparse_file
- beagle3 7y agoIndeed, but many programs, backup or otherwise, or unaware of it. IIRC gzip/bzip/zstd don’t (will spend time reading and writing all the missing pages) whereS gnu tar does.
- terrelln 7y agoZstd by default will write sparse files when it detects runs of zeros during decompression, but isn't aware of it when reading a file.
- butterthebuddha 7y agoWe down-truncated on process exit. I'll take a look at the symas LMDB source code, thanks for pointing that out.
- downerending 7y agoBeware of multi-host filesystems (e.g., GPFS). The required locking and sync can be far slower than you might get on a local filesystem. If nothing else, it's very useful to have a "don't use mmap" option in your code.
- microtherion 7y agoAnother risk is that if the file system on which the file resides can get unmounted for any reason, access to the mmap'ed data will fail in nasty ways.
- deleted 7y ago[deleted]
- jofer 7y agoThere seems to be a common misconception that HDF5 somehow isn't chunked like zarray is... HDF5 will happily chunk things internally and treat them like a single large dataset. It's an index of chunks under the hood. It's _very_ much meant to encourage and allow chunking large datasets transparently. The big advantage of zarray is when reading/writing to a cloud bucket, where having a single file containing chunks/indicies is impractical and you'd rather work with direct references to objects in a bucket. Otherwise, HDF5 offers every single advantage that zarray has and is much more mature, stable, better documented, and has better support. If you're working in a situation where it makes sense to have things on disk or some sort of NFS share, use HDF5. If you're working with objects in a cloud bucket, you'll incur additional overhead with HDF5, as you'll have to read its table of indices, then make range requests to each chunk. Zarr is optimized for the cloud use case.
- itamarst 7y agoThe article, at least, doesn't say anything about HDF5 not be chunked. I recommended Zarr on the basis that threading seems to work much better and compression options are better.
- marsokod 7y agoHow is compression better with zarr? Pytables has been supporting quite a lot of compression format for hdf5 for quite a while. I've been using it with zstandard without any issue so far.
- wdobbels 7y agoNo mention of dask? So far I haven't really had a need for larger-than-memory arrays, but dask would be my go-to choice.
- itamarst 7y agoDask, while great, doesn't solve this problem at all. You still need some way to load the data. And for that Dask supports Zarr, and HDF5, and pre-existing arrays (where mmap() comes in), and a few other formats: https://docs.dask.org/en/latest/array-creation.html# https://docs.dask.org/en/latest/array-creation.html#
- jszymborski 7y agoI wonder how this compares to bcolz [0]. I've been using it for a recent DL project and it's been a lifesaver... haven't tried zarr or mmap however. [0] https://github.com/Blosc/bcolz https://github.com/Blosc/bcolz
- godelski 7y agoI thought I'd mention another library, since it is a competitor to HDF5. In HPC ADIOS2[0] and HDF5 are commonly user. ADIOS2 helps with streaming data if you want to do in situ work, but it can also read and write HDF5 files. [0] https://github.com/ornladios/ADIOS2 https://github.com/ornladios/ADIOS2
- stavrospap 7y agoHi all, we'd love to hear your thoughts about TileDB (https://github.com/TileDB-Inc/TileDB https://github.com/TileDB-Inc/TileDB), which offers the key advantages of both HDF5 and Zarr, such as chunked dense arrays, numerous compression filters and native cloud support. But TileDB is more as it also supports sparse arrays and integrates with more languages, databases and data science tools. If you'd like your data to be readable by multiple languages and tools or need array versioning, it is worth taking a look. Docs: https://docs.tiledb.com/developer/ https://docs.tiledb.com/developer/ Website: https://tiledb.com/ https://tiledb.com/ Earlier HN post: https://news.ycombinator.com/item?id=15547749 https://news.ycombinator.com/item?id=15547749 Disclosure: I am a member of the TileDB team.
- Majromax 7y ago> which offers the key advantages of both HDF5 and Zarr, such as chunked dense arrays, numerous compression filters and native cloud support. Taking a look at the compression filters, it seems that they are mostly suited for byte-oriented and integer data? The bit width reduction filter, for example, seems to act only on integers, and it doesn't look like any floating-point-specific compressors (like fpzip) are implemented -- just general-purpose ones like bzip2.
- just_testing 7y agoI am replying here with some constructive criticism, because I found it awesome. As far as I can tell, TileDB is like a better SQLite and better Parquet, is that it? The webpage does not tell me exactly what TileDB is, so I'm having a hard time getting understanding what's underneath all the marketing mumbo-jumbo. But if it is indeed a better SQLite/Parquet, DAMN, I'm so going to spread the gospel of this to all my students.
- dekhn 7y agoThe simplest way to think about TileDB and zarr is that they are zip files of arrays (either dense or sparse). Not like sqlite, not exactly like parquet. For a lot of what I do, I want a hierarchical containment system- the equivalent of folders with files. And the files themselves are leaves in the hierarchy, containing multidimensional array data. WOrks great when the arrays are composed of fairly straightforward payloads, like float[x][y][z] but also works if your array values are structs. Much of the value in zarr and tiledb comes from specifically how they arrange the arrays, for convenient read access to slices of the arrays. Access is going to look like: ages = root["user"]["age"][100:100:2] Parquet is mostly a column file format, but with nesting. I'd use it to store large amounts of structured data with a relatively straightforward schema, although the schema itself can be fairly nested so some records have very complex structure. Access would often be in a loop over all records: for record in records: if record.user.has_age(): print("User age:", record.user.age) SQLIte is a library/CLI that implements a relational database. It has a SQL interface and stores data using classic relational DB approaches, including secondary indices, etc, and permitting joins directly within the engine: SELECT age FROM user WHERE user.country == 'Bulgaria'
- heartbeats 7y ago> The data has to be on the filesystem. You can’t load data from a blob store like AWS S3. You sure about this? You can mount S3 locally, and the filesystem should be transparent to the kernel.
- itamarst 7y agoThere are two ways of doing this that I know of: 1. FUSE filesystems. This means sending really slow queries to S3, and so you're much better off using compression to get better performance. 2. Syncing to filesystem with AWS DataSync. This would work, yes, but it's a S3 specific feature and not all blob storage systems have it.
- jedbrown 7y agoHDF5 has some S3-native features emerging (https://www.hdfgroup.org/wp-content/uploads/2019/11/SC19-HDF5-BOF.pdf https://www.hdfgroup.org/wp-content/uploads/2019/11/SC19-HDF...), and there's also this relevant comparison reading HDF5 files as efficiently as native Zarr: https://medium.com/pangeo/cloud-performant-reading-of-netcdf4-hdf5-data-using-the-zarr-library-1a95c5c92314 https://medium.com/pangeo/cloud-performant-reading-of-netcdf...
- khc 7y ago> 1. FUSE filesystems. This means sending really slow queries to S3, and so you're much better off using compression to get better performance. As author of https://github.com/kahing/goofys/ https://github.com/kahing/goofys/ I respectfully disagree :-)
- itamarst 7y agoI don't mean the filesystem implementation is inherently slow, necessarily: I'm talking about bandwidth. If you get 3× compression, read times from S3 will be 3× as fast—but that means no mmap().
- makapuf 7y agoI'd much rather have a performance oriented blog post have at least a few performance data points (even if put into context): simple vs fast ... but by how much ?
- dalke 7y agoGiven the discussion on Zarr, HDF5 and TileDB, I hope I can get feedback from a problem I have. I want to switch from a format I made (the "FPB" format described at http://chemfp.com/fpb_format/ http://chemfp.com/fpb_format/ ) to one which scales to more than 250 million entries. (See https://jcheminf.biomedcentral.com/articles/10.1186/s13321-019-0398-8 https://jcheminf.biomedcentral.com/articles/10.1186/s13321-0... for my recent publication on the system, which implements similarity search for cheminformatics fingerprints.) I can't figure out if one of those three methods can do what I want. I want to include a multi-valued lookup table so given a string id I can find the 0 or more corresponding records. (I can handle single-valued lookup tables, if multi-valued isn't available.) I looked at Parquet, but it doesn't seem to handle id->index table lookups. Or 256 byte fields. Do any of Zarr, HDF5 and TileDB offer a better fit? Background: it's hard to compare molecules directly. Instead, there are hashing methods to turn a molecular structure into a fixed-length bitstring of length nearly always between 166 and 4096 bits in length. Bitstrings are easy to compare using Jaccard similarity, which gives a decent-enough and very fast way to compare molecular similarity. With help from another HN user (who I paid), we were able to get the performance to within a factor of 2 of the theoretical memory bandwidth. Going faster requires compressed RAM->uncompressed cache transfer, better indexing, and distributed computing. I don't know how to do it using one of the existing Big Data formats, and developing my own format is hard. My FPB file format is a FourCC format with several chunks. One chunk stores all N fingerprints, cache size aligned. These are ordered by popcount, so in principle some should be easily compressible - I haven't tried, and can't figure out how BLOSC is supposed to be used. Another chunk stores the corresponding fingerprint identifiers, UTF-8/non-fixed-length strings, in the same order (fingerprint[i] corresponds to id[i]). A third chunk stores a multi-valued hash table so I can look up fingerprint(s) by name, instead of by index. The hash table is a variant of the djb's "cdb" hash table. This is a 32-bit hash, which means for large data sets I get many collisions. (I designed it for <100M entries. Some people now want to handle >1B entries.) I'm hoping to use an existing format rather than roll my own. Again.
- jmuhlich 7y agoHi Andrew! Looking into similar requirements I came across Feather and fst. They both basically let you efficiently slice into compressed on-disk DataFrames. Feather already supports Python, but fst is just for C++ and R at the moment. https://blog.rstudio.com/2016/03/29/feather/ https://blog.rstudio.com/2016/03/29/feather/ http://www.fstpackage.org/ http://www.fstpackage.org/ https://github.com/fstpackage/fstlib https://github.com/fstpackage/fstlib
- tbenst 7y agoPyTables with Zstd compression is likely the fastest performance option of them all. In my benchmarks, PyTables is up to 10x faster than H5PY
- mborg 7y agoThe article leaves out a pretty important question you should ask yourself: "What fraction of the data will I need to access?" If the answer is "much less than the whole file", accessing the data over the network makes sense. Otherwise, you should almost certainly bring it to a local disk, where mmap shines.