62 ms·
Given the discussion on Zarr, HDF5 and TileDB, I hope I can get feedback from a problem I have. I want to switch from a format I made (the "FPB" format describe
by dalke 7y ago
Given the discussion on Zarr, HDF5 and TileDB, I hope I can get feedback from a problem I have. I want to switch from a format I made (the "FPB" format described at http://chemfp.com/fpb_format/ http://chemfp.com/fpb_format/ ) to one which scales to more than 250 million entries. (See https://jcheminf.biomedcentral.com/articles/10.1186/s13321-019-0398-8 https://jcheminf.biomedcentral.com/articles/10.1186/s13321-0... for my recent publication on the system, which implements similarity search for cheminformatics fingerprints.)
I can't figure out if one of those three methods can do what I want. I want to include a multi-valued lookup table so given a string id I can find the 0 or more corresponding records. (I can handle single-valued lookup tables, if multi-valued isn't available.)
I looked at Parquet, but it doesn't seem to handle id->index table lookups. Or 256 byte fields. Do any of Zarr, HDF5 and TileDB offer a better fit?
Background: it's hard to compare molecules directly. Instead, there are hashing methods to turn a molecular structure into a fixed-length bitstring of length nearly always between 166 and 4096 bits in length.
Bitstrings are easy to compare using Jaccard similarity, which gives a decent-enough and very fast way to compare molecular similarity. With help from another HN user (who I paid), we were able to get the performance to within a factor of 2 of the theoretical memory bandwidth.
Going faster requires compressed RAM->uncompressed cache transfer, better indexing, and distributed computing. I don't know how to do it using one of the existing Big Data formats, and developing my own format is hard.
My FPB file format is a FourCC format with several chunks. One chunk stores all N fingerprints, cache size aligned. These are ordered by popcount, so in principle some should be easily compressible - I haven't tried, and can't figure out how BLOSC is supposed to be used.
Another chunk stores the corresponding fingerprint identifiers, UTF-8/non-fixed-length strings, in the same order (fingerprint[i] corresponds to id[i]).
A third chunk stores a multi-valued hash table so I can look up fingerprint(s) by name, instead of by index. The hash table is a variant of the djb's "cdb" hash table. This is a 32-bit hash, which means for large data sets I get many collisions. (I designed it for <100M entries. Some people now want to handle >1B entries.)
I'm hoping to use an existing format rather than roll my own. Again.
- jmuhlich 7y agoHi Andrew! Looking into similar requirements I came across Feather and fst. They both basically let you efficiently slice into compressed on-disk DataFrames. Feather already supports Python, but fst is just for C++ and R at the moment. https://blog.rstudio.com/2016/03/29/feather/ https://blog.rstudio.com/2016/03/29/feather/ http://www.fstpackage.org/ http://www.fstpackage.org/ https://github.com/fstpackage/fstlib https://github.com/fstpackage/fstlib
- jdnier 7y agoLatest info on Feather and related pyarrow: http://arrow.apache.org/docs/python/ipc.html#feather-format http://arrow.apache.org/docs/python/ipc.html#feather-format
- dalke 7y agoThanks Jeremy. I looked at Feature, and the underlying Arrow format, but couldn't figure out how to implement a row-id-label-to-row-index dictionary. The only "dictionary" I saw was for categorical data. I had the same issue with fstlib - how do I handle id lookups? Any pointers?
- jmuhlich 7y agoSorry, I thought Feather supported random row access but it turns out it only supports random column access. For fst, I only played with the R interface, which would be called like this to retrieve row 12345 from the "fingerprint" column: read_fst("library.fst", columns="fingerprint", from=12345, to=12345) However fst didn't offer a raw/binary column datatype last I checked, which is frustrating. It has chr (string) but R can't have embedded NUL bytes in strings, so that was a dead-end for efficient storage of binary fingerprints for R. I didn't check if the underlying fstlib structures accept NULs in string columns.
- dalke 7y agoThanks for confirming that. Looks like I'll need to look elsewhere for the next chemfp fingerprint file format.
- Shelnutt2 7y agoTileDB can handle most of your requirements today and we are working on the "row id labels" which we call axes labels[1]. We are looking to get axes labels support completed in the next 2-3 months. TileDB supports fixed size and variable length attributes so you have flexibility in how to handle your 256 byte and UTF-8 fields. On the distributed computing front, TileDB has integrations with Dask[2] and Spark[3]. We are happy to answer any further questions you might have on TileDB or how to adapt it to your use case. [1] https://feedback.tiledb.com/tiledb-core/p/support-axes-labels https://feedback.tiledb.com/tiledb-core/p/support-axes-label... [2] https://docs.tiledb.com/developer/dask/quickstart https://docs.tiledb.com/developer/dask/quickstart [3] https://docs.tiledb.com/developer/spark/quickstart https://docs.tiledb.com/developer/spark/quickstart Disclosure: I am a member of the TileDB team.