8 ms·
TileDB: Storing massive dense and sparse multi-dimensional array data
- ttd 9y agoLooks interesting... Here's the publication that it appears to have formed from: https://dl.acm.org/citation.cfm?id=3025117 https://dl.acm.org/citation.cfm?id=3025117 (VLDB 2016).
- styx31 9y agoAlso available directly here: [PDF, VLDB 2017] https://people.csail.mit.edu/stavrosp/papers/vldb2017/VLDB17_TileDB.pdf https://people.csail.mit.edu/stavrosp/papers/vldb2017/VLDB17...
- dx034 9y agoI would like to see some information on compression ratios. Compression can often significantly improve performance (less data to load/save, more in memory or cache) and esp dense arrays should compress easily. They mention that compression is supported but I'm not sure if that's done internally or just applied to the files they create.
- jbooth 9y agoFloating points compress poorly, though.
- dogruck 9y agoNot always poorly -- especially if you preprocess by calculating running diffs of consecutive values.
- redcalx 9y ago> running diffs of consecutive values https://en.wikipedia.org/wiki/Delta_encoding https://en.wikipedia.org/wiki/Delta_encoding
- wenc 9y agoNot always [1]. Also in many time-series applications involving sensors whose readings don't fluctuate that much, process historians often apply deadband compression (i.e. store only one value if it is within a certain band). The type of compression is lossy and sometimes a bit controversial for high-fidelity uses, but often results in efficient storage. [1] zfp, fpzip: https://computation.llnl.gov/projects/floating-point-compression https://computation.llnl.gov/projects/floating-point-compres...
- srean 9y agoThere have been a few responses along the lines of 'not always' but what you say is indeed largely true. There is another thing that is worth considering and that is the algorithms (and even the theory) that works well for compression of discrete sources are not well suited for compressing real numbers (floating point numbers aren't, but, they are the poor man's reals). On the theory side, this bothered Claude Shannon enough that he decided to revisit this later in his career to create rate distortion theory, he knew that there was some unfinished business in information theory. We do have sort of a chicken and egg problem here, especially when we want store a lot floating points for a ML workload. Learning how to compress and learning the underlying distribution are equivalent problems. If we have already learned the model, then yes we could compress the data well. But when we haven't, then by definition we wouldn't have the knowledge to do a good job of storing the data in a well compressed form. After we have acquired the knowledge to compress well, we don't really need the compressed data anymore to learn the model, we already have it. One way to address this would be to do both incrementally and simultaneously.
- jakebol 9y agoJake from TileDB, Inc. wenc and srean are right. techniques such as those used in zfp and fpzip wenc mentioned are also used to compress real world las file (point cloud) datasets. For the moment we are only focused on lossless compression (scientists are paranoid about losing data), but there is definitely room to explore integration with lossy compression as well. Machine learning applications often do not need full precision so intelligent forms of lossy compression are useful. Another cool research application of TileDB that extends the storage library with the VP9 codec can be seen here: https://homes.cs.washington.edu/~magda/papers/haynes-sigmod17-demo.pdf https://homes.cs.washington.edu/~magda/papers/haynes-sigmod1...
- dx034 9y agoYou can get very good lossless compression with floating point numbers, Facebook's Gorilla paper comes to mind. I usually use it for delta-of-delta encoding which provides very high compression for time series. While that won't really help in your case, their floating point encoding could help compressing matrices quite efficiently. http://www.vldb.org/pvldb/vol8/p1816-teller.pdf http://www.vldb.org/pvldb/vol8/p1816-teller.pdf [page 5]
- JFFalcon 9y agoThe Learn More section has a very nice description of the concepts. The compression page [0] seems to say that they group each attribute for a fragment together and then apply compression. [0] https://docs.tiledb.io/v1.0/docs/compression https://docs.tiledb.io/v1.0/docs/compression
- davedx 9y agoSounds ideal for storing training/intermediate data for machine learning. Niche competitor for Cassandra in this space?
- speedplane 9y agoMost models I've seen have at most a few hundred MB of parameters, have complex connections that can only be modeled as a set of many different 1 or 2D arrays with different sizes. Further, this DB stresses that it handles sparse sparse data, any most ML data is not sparse. It doesn't seem like the best application. On one of their pages it mentions they ingest BAM records, which is for biological sequences. I'm guessing some DNA storage applications.
- wenc 9y ago> most ML data is not sparse. This brings up a question: in what fields does one find heavy use of large, sparse matrices that need to be persisted and queryable? In my mind, sparse matrices typically occur in the context of graphs/relationships, e.g. PageRank, logistic networks, adjacency matrices, etc. They also tend to be a property of Hessian matrices (2nd order derivative for a multivariate system). But typically these are intermediate quantities that are discarded after a computation completes.
- jakebol 9y agoJake from TileDB, Inc. Genomics is a big field where sparse matrix storage is needed. Human genomes are stored as a diff off of a reference, which as you indicated forms a graph which can be represented as a sparse matrix. In other fields of genomics, such as metagenomics, fragments of DNA when analyzed also have a graph like structure. TileDB supports both dense and sparse arrays. It was designed around the concept of handling sparse arrays but dense arrays can be thought of a degenerate case of sparse array storage in TileDB. For dense arrays tile extents are contiguous and we don't materialize the coordinate values. This way all the concepts are the same and we can capture both use-cases. Sparse annotations to dense array values, such as NA or Null handling can also be captured as a sparse array fragment layered over a backing dense array. I agree with you that for most use cases, storage will be dense. But it is useful to have one system that can handle both representations efficiently, with the sparse case not added on as an afterthought (it also makes the system simpler).
- marco-lavag 9y agoI wish they had python bindings.
- castis 9y agoTheir Python, R, Matlab, and Excel bindings are in-progress.
- Kiro 9y agoWould it make sense to store a tilemap for a game in this? Normally I use NoSQL for that and just store the tilemap as a JSON array. I need to be able to update and retrieve single cells easily (this is for a backend in a multiplayer game).
- k__ 9y agoI don't know, tiledb seems rather low-level and many databases have geospatial indexing.
- mrdrozdov 9y agoCould I use this for deep learning research? It’s common for my models to be anywhere from 100mb to a few gb. Although, I’d find it more useful for reading batched training data.
- stavrospap 9y agoStavros from TileDB, Inc. here: TileDB could be useful for storing your training data (in some storage backend of your choice) as well as your intermediate data (as davedx pointed out). But this would make sense only if your data are truly large and cannot fit in main memory. In fact, we are looking forward to seamlessly integrating with systems like TensorFlow, but we would rather wait until we can bring some value to applications with very large storage requirements.
- rajnathani 9y agoThe underlying technology [1] for TileDB came up with MIT and Intel working together. And TileDB, the company, formed out of it, has recently received funding [2] from Intel in Intel's latest tranche of investments. [1] https://people.csail.mit.edu/stavrosp/papers/vldb2017/VLDB17_TileDB.pdf https://people.csail.mit.edu/stavrosp/papers/vldb2017/VLDB17... [2] https://techcrunch.com/2017/10/19/data-is-the-name-of-the-game-as-intel-capital-puts-60m-in-15-startups-566m-in-2017-overall https://techcrunch.com/2017/10/19/data-is-the-name-of-the-ga...
- grabcocque 9y agoGood to know private shareholders are getting even richer off all that taxpayer research money that's pumped into MIT.
- mac01021 9y agoGiven that this is an open-source project, Intel's involvement doesn't seem too sinister to me.
- ensignavenger 9y agoI think it was MITs involvement the above poster was concerned about... but the fact that it is open source is indeed relevant!
- stavrospap 9y agoStavros from TileDB, Inc. here: This comment gives us the opportunity to explain a bit the history of the company and our thesis. TileDB was my research project while I was working as a full-time researcher at Intel Labs and was stationed at MIT. This was part of a collaboration between Intel Labs and MIT, called Intel Science and Technology Center (ISTC) for Big Data. Intel funded MIT (and several other universities) for 5 years, while mandating that any tech produced in the center should be open-source with a non-restrictive license (like our MIT license), and without any strings attached. So ISTC was for pure research. We created TileDB, Inc. so that we can continue working on TileDB beyond the graceful termination of the ISTC, since we believed that the tech (and future vision) can significantly contribute to the scientific community (there was already some early evidence through our work with the Broad Institute on genomics). The company will continue to respect and contribute to the open-source community, while trying to bring together very talented people to transition the tech from a research project to a production-ready software.
- _pmf_ 9y ago> Python, R, Matlab and Excel Mentioning Matlab and Excel immediately puts the product in the category "they know what they are doing" as opposed to "another group of sophomores trying to reinvent data science". I'm still waiting for a raw data dump with "avoid copies at all cost" access for very raw, very verbose vehicle and manufacturing data that is in 99.9 per cent never accessed, but must be analyzed when errors are detected late in the process. I.e. there's practically no transformation to apply during storage, but upon access, transformation must be done. If TileDB is kind of like a more structured, more low-level struct oriented Redis, it is a very welcome addition.
- srean 9y agoApart from a diminishing population of old farts and a clutch of national labs and universities that Mathworks keeps well greased, is MATLAB even relevant anymore ?
- _pmf_ 9y agoPervasive in the German automotive industry.
- jakebol 9y agoJake from TileDB, Inc. Engineering as a discipline is conservative (for good reason). The tools and the processes they use change slowly. Matlab is still entirely relevant both in the sciences and in industry. There are people with huge amounts of domain knowledge (who may only know Matlab or Excel) that are increasingly called upon to analyze and interpret larger amounts of data. These "old farts" are the people engineering, designing, and debugging our modern world. Empowering people with domain knowledge to answer data driven questions is what the democratization of data science is all about. There is tremendous value in building bridges across communities and across generations here. I say this as someone who helped in small ways to develop open source alternatives to Matlab.
- srean 9y ago> These "old farts" are the people engineering and designing our modern world. Empowering people with domain knowledge to answer data driven questions is what the democratization of data science is all about. BTW I don't begrudge that TileDB has MATLAB support. I see plenty of modern scientific/engineering workload in my day job. From what I see around myself it is usually a handful of people set in their ways that are holding others back by keeping a ridiculously overpriced tool alive that has comparable if not better alternatives. I gather from comments that it is different in Europe.
- brootstrap 9y agoanyone have experience using one of these array data stores to handle large amounts of weather forecast data? At my job we've come up with our own clever postgres solution to handle the large amounts of gridded binary data. Essentially we are inserting large 3D arrays (a series of images where each pixel represents a geographic location and a value, like windspeed or temperature or rainfall) into our postgres. Our solution is solid but am always on the lookup for new novel approaches.
- deleted 9y ago[deleted]
- deleted 9y ago[deleted]
- jakebol 9y agoJake from TileDB, Inc. I think this would be an ideal workload for array data stores (NetCDF a standard in this area uses HDF5 under the hood). You have <N> number of attributes that you want per grid-point over time (and you want to append to the time dimension). If you are ingesting Grib2 files then you can take advantage of compression as well. An array data store like TileDB should offer advantages for fast access, as you can get a pointer directly to the stored array and do not have to access the (serialized) data over a socket, especially if you are only interested in a subarray of the dataset.
- oliviervg1 9y agoHi both, this is exactly something that I’m looking at doing. We’ve got about 10TB of netcdf data coming in everyday and we’re looking for a cost efficient data store to provide fast access to individual grid points. S3 has proven to be too slow. Any chance I could pick your brains about using either Postgres or TileDB? Thanks!
- jakebol 9y agoAbsolutely! Drop us a line at hello@tiledb.io and tell us a little more about the problem you are trying to solve and we can go from there.
- ankitrohatgi 9y agoI am curious about how the query performance compares to working with JSON files in Spark for ~100GB data.
- maxpert 9y agoThey have some benchmarks in paper https://people.csail.mit.edu/stavrosp/papers/vldb2017/VLDB17_TileDB.pdf https://people.csail.mit.edu/stavrosp/papers/vldb2017/VLDB17...
- jakebol 9y agoJake from TileDB, Inc. here: Depending on the structure of the JSON files you are querying you maybe able to take advantage of columnar compression and massively reduce the dataset size (especially if the json files contain numeric data). Also, repeat queries will not have to re-parse the JSON files. This may speed up queries quite a lot, but it depends on the specifics of your problem.
- StanSeltser 9y agoStanislav Seltser, Petacube you are talking comparing structured workload(array-based TileDB) to unstructured one (JSON+Spark). Once you convert your JSON to sparce array structure (one time conversion) TileDB will beat Spark+JSON by several orders of magnitude. Caveat: assuming your spark+json workoad is a some heayy processing not a lightweight one.
- maxpert 9y agoLooks nice, still have to read the whole paper. Seems like it's most useful for sparse arrays. Maybe we will get a golang port :P
- kuwze 9y agoI wonder how it compares to SciDB[0] which is also used for storing multidimensional data. [0]: https://en.wikipedia.org/wiki/SciDB https://en.wikipedia.org/wiki/SciDB
- fisherjeff 9y agoThey cover that in their paper: https://people.csail.mit.edu/stavrosp/papers/vldb2017/VLDB17_TileDB.pdf https://people.csail.mit.edu/stavrosp/papers/vldb2017/VLDB17...
- jakebol 9y agoJake from TileDB, Inc. In addition to the differences pointed out in the paper, I think SciDB and TileDB are very different philosophically. SciDB is architected very much like a traditional RDMS while TileDB is much more lightweight. SciDB encourages you to use their own query language (AQL), TileDB wants to integrate and extend the high level tools you already use (Python, R, etc.) with as little overhead as possible.
- StanSeltser 9y agoStanislav Seltser, Petacube kuwze: the following things are fundamentally different: license, software size,data model, sparsity support SciDB --- license: Affero ,software size: 5GB ,data model: ACID ,focus on: dense ,dimensions: integer ------ TileDB license MIT ,software size <1MB , data model: eventual consistency via fragments , focus on: dense, sparse ,dimensions are integer, floats
- sandGorgon 9y agoAnyone know how it compares with Apache Arrow (that pandas is moving to) ? http://wesmckinney.com/blog/apache-arrow-pandas-internals/ http://wesmckinney.com/blog/apache-arrow-pandas-internals/ http://arrow.apache.org/ http://arrow.apache.org/
- anilshanbhag 9y agoArrow is a columnar format. TileDB does clustering across multiple dimensions based. Arrow is great when you request specific columns. TileDb is great when you want rectangular sections of the space (think getting tiles from a map).
- stavrospap 9y agoStavros from TileDB, Inc. here: Arrow employs a columnar format to store objects like data frames. TileDB is also columnar (so you can sub-select attributes and perform analytics with very similar optimizations to Arrow), but the first-class citizens are (the more general) dense and sparse multi-dimensional arrays. Moreover, TileDB focuses on optimizing for the persistent storage backend, so that it can handle out-of-core analytics on massive datasets that cannot fit in main memory (e.g., genomics), or offer the same performance using less RAM (leading to cost savings in the cloud). Nevertheless, we are quite fond of Arrow, so we hope to work together at some point and integrate as seamlessly as possible.
- polskibus 9y agoWhat do you mean by out-of-core?
- stavrospap 9y agoIn-core algorithms require the entire array(s) in main memory to perform some computation. Out-of-core algorithms are typically block-based and stream the array blocks from persistent storage to main memory on demand, working on parts of the array(s) at a time, thus minimizing the memory requirements. If this is done asynchronously and carefully, for some CPU-bound algorithms you may be able to completely hide the storage-to-memory cost, thus saving memory without losing performance.
- acidflask 9y agoHow long until we can have Julia wrappers?
- leethargo 9y agoSince jakebol is involved in both TileDB and Julia, this might be up and coming?
- Osmium 9y agoLooks exciting. What's the easiest way to get started with this via Python/NumPy? Looks like this is a design goal, but not currently supported.
- stavrospap 9y agoStavros from TileDB, Inc. here: We have elevated the Python/NumPy bindings as our top priority and have already started development. We will try to ship them asap. :)
- Osmium 9y agoExcellent! Good to hear :) Also saw in your documentation that you're concentrating on lossless compression right now, which makes complete sense. However, as a scientist, I just want to put a vote in for lossy compression too: it's not uncommon to work with large datasets given in float64 (because float64 is used for any intermediate processing steps), but that actual final precision we need to store is much less than that, but we're stuck with these huge binary files.
- stavrospap 9y agoThe way the code is currently architected allows us to add support for pretty much any compressor (compatible with our MIT license) with minimal effort. So, please do send us suggestions about your preferred compressor and we will add it pretty quickly. Thanks!
- polskibus 9y agoHow do I slice & dice TileDB with a query? Can TileDB compute aggregations or do I need to copy cells into main program memory out of TileDB and then perform all data manipulation?
- stavrospap 9y agoStavros from TileDB, Inc. here: Currently, TileDB is a storage manager, so it offers only efficient slicing. Support for aggregation queries (e.g., similar to what you can do with Pandas) is in our roadmap. We plan to work on it as soon as we ship the Python bindings.
- sixdimensional 9y agoFor posterity sake, it reminds me of MUMPS persistent sparse arrays, but built to scale much larger. This comment is not meant in any way to take away from this achievement, but rather to muse on where we were and where we're going. It seems like the days we were stuck in particular or limited ways of thinking about databases/persistent storage are finally well and truly behind us! Now we have many awesome tools to choose from, better to have more tools in the belt than less.
- jakebol 9y agoHaha, fully agree! It's good to separate MUMPS the language from MUMPS the storage layer / database which was innovative and pioneering in many ways. I don't think we need to re-hash MUMPS the language :)
- sixdimensional 9y agoAgreed! :)
- lowsenberg 9y agoThis looks very interesting. Currently we are storing our dense simulation (and experimental) data in NetCDF/HDF5. Given correct chunking, this seems to be pretty efficient both performance and compression wise. What would we gain using TileDB? How does performance compare with HDF5?
- jakebol 9y agoJake from TileDB, Inc. Performance wise I would look at the referenced paper in this thread which provides benchmarks for various workloads. As to what advantages TileDB may offer you that is problem dependent, esp. compared to dense simulation output data which is the use case HDF5 was designed for. If you have specific suggestions for ways to improve HDF5 for your use case we would love to hear about them.
- stavrospap 9y agoStavros from TileDB, Inc. here: HDF5 is a great software and TileDB was heavily inspired by it. HDF5 probably works great for your use case. TileDB matches the HDF5 performance in the dense case, but in addition it addresses some important limitations of HDF5, which may or may not be relevant to your use case. These include: sparse array support (not relevant to you), multiple readers multiple writers through thread- and process-safety (HDF5 does not have full thread-safety, whereas also it does not support parallel writes with compression - I am assuming you are using MPI and a single writer though, so still HDF5 should work well for you), efficient writes in a log-structured manner that enables multi-versioning and fault tolerance (HDF5 may suffer from file corruption upon error and file fragmentation - you are probably not updating, so still not very relevant to you). Having said that and echoing Jake's comment, we would love to hear from you how TileDB could be adapted to serve your case better. A general comment: TileDB’s vision goes beyond that of the HDF5 (or any scientific) format. Considering though the quantities of HDF5 data out there (and the fact that we like the software), we are thinking about building some integration with HDF5 (and NetCDF). For instance, you may be able to create a TileDB array by “pointing” to an HDF5 dataset, without unnecessarily ingesting the HDF5 files but still enjoying the TileDB API and extra features.