5 ms·

TileDB 2.0 and the Future of Data Science

81 points by dalke 6y ago

dalke 6y ago
The highlighted new features for version 2.0 are: 1) Support for heterogeneous dimensions in sparse arrays (that is, full dataframes), 2) Support for string dimensions in sparse arrays, 3) Google Cloud Storage and Azure Blob Storage support, and 4) Completely revamped R API. They write: "These features mark an important step in our journey to deliver a universal data storage engine for all applications and data science tools."
- goodSyntax 6y ago
  i agree, the title and article is mostly hyperbole
  - stavrospap 6y ago
    Stavros form TileDB here. Here is a more verbose explanation. Up until before 2.0, TileDB was already powerful for the main applications we targeted at: geospatial and genomics. The support for both dense and sparse arrays and the way it handles data versioning made it quite unique vs. HDF5 and Zarr. But we noticed that most of the data scientists we were working with had a lot of data beyond genomic variants, LiDAR points and rasters. They had tons of dataframes. And they were using at least two storage engines, TileDB for arrays, and Parquet or a relational database for dataframes. If you are in a large organization, this a big pain. In TileDB 2.0 we made a huge refactoring to support something seemingly simple: dimensions in sparse arrays that can have different types and that could even be strings. This allowed us to model any dataframe as a sparse array, effectively making TileDB act as a primary multi-dimensional index. In relational databases, this means that your data is sorted in an order on disk that favors your multi-column slicing enormously, so range search becomes rapid. Therefore, what we are telling the community with this release is that you can have dense arrays, sparse arrays, and dataframes in a single embeddable library being integrated with pretty much every data science tool out there, so that data scientists never have to worry about backends, files, updates, or anything other than their scientific analysis. In other words, we believe the future of data science is more science.
    - bionhoward 6y ago
      Show us code usage examples please!
      - stavrospap 6y ago
        Full developer docs here: https://docs.tiledb.com/main/ https://docs.tiledb.com/main/ Specific dataframe examples coming up shortly.
xyzzy_plugh 6y ago
This seems similar to the direction Apache Arrow is going in, but I don't see the use case for this once Arrow dataframes are prevalent.
- tpetry 6y ago
  One use case is that is running within postgres, so you dont need to install another (complex?) software and backup it.
- Shelnutt2 6y ago
  Disclosure: I am a member of the TileDB team. TileDB is designed primarily for persistent storage, similar to HDF5, Zarr and Parquet. TileDB is a cloud-optimized, dense and sparse multi-dimensional array storage engine, which is broader than Arrow's Dataframe. In addition TileDB also handles updates, time traveling and partitioning at the library level, which are not possible with the Arrow project's current choice of persistent format, Parquet. TileDB removes the need for using extra services like Delta Lake for updates or Hive for cataloging and partitioning, as you get all this functionality in a single, embeddable library.
  - missosoup 6y ago
    I feel like you guys need to create some additional engineering-level documentation to convey all this. The landing page and few links I navigated to were all too fluffy.
    - MiroF 6y ago
      For things that start as academic work, often the original paper is quite helpful. I find it an enjoyable read if you have a background in databases (it actually was a reading for my grad systems class) https://people.csail.mit.edu/stavrosp/papers/vldb2017/VLDB17_TileDB.pdf https://people.csail.mit.edu/stavrosp/papers/vldb2017/VLDB17... The basic jist is: TileDB is substantially faster than alternatives for random writes/reads - this allows for more sophisticated parallel linear algebra algorithms as well as evaluation of predicates.
xiaodai 6y ago
After reading it, it's not clear it is the future of data science.
ipsum2 6y ago
I don't know if the creators of TileDB visit HN, but if they do - it would be nice to see a code example that showcases what makes TileDB unique. Right now, I don't see any benefit versus serializing a data frame in hdf5 and storing it in Redis or cloud blob storage. (Reading https://tiledb.com/developer https://tiledb.com/developer, the Github, or the blog - none of it shows me anything amazing) Edit: browsing around the site, I finally found out what tileDB offers, and I'm still confused (too much fluff language) To save people time: https://docs.tiledb.com/main/handling-dataframes https://docs.tiledb.com/main/handling-dataframes Multi-column slicing: By defining any subset of the columns as dimensions and due to TileDB's tiling flexibility, you can increase the pruning effectiveness of multi-column slicing, thus leading to better overall read performance. Essentially, a TileDB array acts as a primary multi-dimensional index on the columns selected as the array dimensions. Data updates and versioning: TileDB offers rapid, parallel, cloud-optimized updates. All the update logic is pushed into the storage engine and is completely transparent to the user. TileDB also exposes useful time traveling functionality, such as reading arrays at time snapshots, effectively implementing data versioning built into a single embeddable library. Partitioning: TileDB enables balanced partitioning, without limiting each partition to single column values. Moreover, we will soon expose API functions for dynamically selecting different partitioning schemes (e.g., on different subsets of columns with different orders), without the need for reorganizing/rewriting the array. Sorting: All sorting is taken care of by TileDB internally with multi-threading.
- dalke 6y ago
  I heard about TileDB because one of the authors commented at https://news.ycombinator.com/item?id=22423632 https://news.ycombinator.com/item?id=22423632 . That discussion includes what you mentioned, plus a mention that it integrates with MariaDB, PrestoDB and Spark, including the ability to use an embedded MariaDB.
  - stavrospap 6y ago
    Thanks for the post!
- stavrospap 6y ago
willvarfar 6y ago
So, basically, tiledb is a file format like paraquet, avro and orc are formats? And it comes with a nice library that makes accessing the data fairly storage-transparent?
- Shelnutt2 6y ago
  Disclosure: I am a member of the TileDB team. Yes you are correct, TileDB introduces a storage format, but is also a full-fledged storage engine that implements it. Having the complete storage engine is what allows us to make accessing the data transparent regardless of the API or application you use to access the data. We are able to integrate into different computational tools without having to reimplement or sacrifice features.
  - willvarfar 6y ago
    How do you compare to Iceberg?
    - Shelnutt2 6y ago
      Disclosure: I am a member of the TileDB team. Iceberg, similar to Delta Lake, is another layer you have to add on top of Parquet, which limits your mobility of compute. You can currently use it either with Spark or Presto, but not directly via language APIs or other compute engines (e.g., pandas, Dask, MariaDB, etc). For example, the pandas "read_parquet" will not be able to read your dataframe, you will have to use pyspark or presto + sql query to fetch the data into pandas. TileDB has updates and data versioning built into its format and storage engine, therefore reading an array directly via a TileDB language API, or Spark, Dask, PrestoDB or MariaDB will have the same behavior. You will be able to see the dataframe after all the changes, or time travel and "activate" only some of the changes in time.
      - willvarfar 6y ago
        How does tiledb deal with appends and upserts etc in s3? Is there success files, atomic folder renaming and things? Or?
jiangplus 6y ago
Looking at the TileDB docs, I like the idea that you can store and manipulate array efficiently. It works like a columnar storage, but organized in tiles. I am not clear how it could scale to multiple nodes and how to do queries. Maybe it should be integrated into postgres.
- Shelnutt2 6y ago
  Disclosure: I am a member of the TileDB team. TileDB fully supports multi-writer/multi-reader, so scaling compute to multiple nodes is mostly about choosing the computation layer. We have integrations with Spark, Dask and PrestoDB for distributed compute. In our cloud product we also offer serverless UDFs and SQL (via MariaDB) to allow for scaling out computations elastically without managing your own clusters. On the postgres side, we are looking at eventually adding a postgres storage engine, now that the new storage support is in PostgresSQL 11. To start with we have built a storage engine for MariaDB, as MariaDB has an excellent storage engine API, which I have past experience with. We hope to upstream the storage engine to MariaDB after the MariaDB 10.5 release. MariaDB integration: https://docs.tiledb.com/mariadb/ https://docs.tiledb.com/mariadb/ Spark integration: https://docs.tiledb.com/spark/ https://docs.tiledb.com/spark/ Dask integration: https://docs.tiledb.com/dask/ https://docs.tiledb.com/dask/ PrestoDB integration: https://docs.tiledb.com/prestodb/ https://docs.tiledb.com/prestodb/ Cloud Docs: https://docs.tiledb.com/cloud/ https://docs.tiledb.com/cloud/

81 points by dalke 6y ago

The highlighted new features for version 2.0 are: 1) Support for heterogeneous dimensions in sparse arrays (that is, full dataframes), 2) Support for string dimensions in sparse arrays, 3) Google Cloud Storage and Azure Blob Storage support, and 4) Completely revamped R API. They write: "These features mark an important step in our journey to deliver a universal data storage engine for all applications and data science tools."

goodSyntax 6y ago
i agree, the title and article is mostly hyperbole
- stavrospap 6y ago
  Stavros form TileDB here. Here is a more verbose explanation. Up until before 2.0, TileDB was already powerful for the main applications we targeted at: geospatial and genomics. The support for both dense and sparse arrays and the way it handles data versioning made it quite unique vs. HDF5 and Zarr. But we noticed that most of the data scientists we were working with had a lot of data beyond genomic variants, LiDAR points and rasters. They had tons of dataframes. And they were using at least two storage engines, TileDB for arrays, and Parquet or a relational database for dataframes. If you are in a large organization, this a big pain. In TileDB 2.0 we made a huge refactoring to support something seemingly simple: dimensions in sparse arrays that can have different types and that could even be strings. This allowed us to model any dataframe as a sparse array, effectively making TileDB act as a primary multi-dimensional index. In relational databases, this means that your data is sorted in an order on disk that favors your multi-column slicing enormously, so range search becomes rapid. Therefore, what we are telling the community with this release is that you can have dense arrays, sparse arrays, and dataframes in a single embeddable library being integrated with pretty much every data science tool out there, so that data scientists never have to worry about backends, files, updates, or anything other than their scientific analysis. In other words, we believe the future of data science is more science.
  - bionhoward 6y ago
    Show us code usage examples please!
    - stavrospap 6y ago
      Full developer docs here: https://docs.tiledb.com/main/ https://docs.tiledb.com/main/ Specific dataframe examples coming up shortly.

xyzzy_plugh 6y ago

This seems similar to the direction Apache Arrow is going in, but I don't see the use case for this once Arrow dataframes are prevalent.

tpetry 6y ago
One use case is that is running within postgres, so you dont need to install another (complex?) software and backup it.
Shelnutt2 6y ago
Disclosure: I am a member of the TileDB team. TileDB is designed primarily for persistent storage, similar to HDF5, Zarr and Parquet. TileDB is a cloud-optimized, dense and sparse multi-dimensional array storage engine, which is broader than Arrow's Dataframe. In addition TileDB also handles updates, time traveling and partitioning at the library level, which are not possible with the Arrow project's current choice of persistent format, Parquet. TileDB removes the need for using extra services like Delta Lake for updates or Hive for cataloging and partitioning, as you get all this functionality in a single, embeddable library.
- missosoup 6y ago
  I feel like you guys need to create some additional engineering-level documentation to convey all this. The landing page and few links I navigated to were all too fluffy.
  - MiroF 6y ago
    For things that start as academic work, often the original paper is quite helpful. I find it an enjoyable read if you have a background in databases (it actually was a reading for my grad systems class) https://people.csail.mit.edu/stavrosp/papers/vldb2017/VLDB17_TileDB.pdf https://people.csail.mit.edu/stavrosp/papers/vldb2017/VLDB17... The basic jist is: TileDB is substantially faster than alternatives for random writes/reads - this allows for more sophisticated parallel linear algebra algorithms as well as evaluation of predicates.

xiaodai 6y ago

After reading it, it's not clear it is the future of data science.

ipsum2 6y ago

I don't know if the creators of TileDB visit HN, but if they do - it would be nice to see a code example that showcases what makes TileDB unique. Right now, I don't see any benefit versus serializing a data frame in hdf5 and storing it in Redis or cloud blob storage. (Reading https://tiledb.com/developer https://tiledb.com/developer, the Github, or the blog - none of it shows me anything amazing) Edit: browsing around the site, I finally found out what tileDB offers, and I'm still confused (too much fluff language) To save people time: https://docs.tiledb.com/main/handling-dataframes https://docs.tiledb.com/main/handling-dataframes Multi-column slicing: By defining any subset of the columns as dimensions and due to TileDB's tiling flexibility, you can increase the pruning effectiveness of multi-column slicing, thus leading to better overall read performance. Essentially, a TileDB array acts as a primary multi-dimensional index on the columns selected as the array dimensions. Data updates and versioning: TileDB offers rapid, parallel, cloud-optimized updates. All the update logic is pushed into the storage engine and is completely transparent to the user. TileDB also exposes useful time traveling functionality, such as reading arrays at time snapshots, effectively implementing data versioning built into a single embeddable library. Partitioning: TileDB enables balanced partitioning, without limiting each partition to single column values. Moreover, we will soon expose API functions for dynamically selecting different partitioning schemes (e.g., on different subsets of columns with different orders), without the need for reorganizing/rewriting the array. Sorting: All sorting is taken care of by TileDB internally with multi-threading.

dalke 6y ago
I heard about TileDB because one of the authors commented at https://news.ycombinator.com/item?id=22423632 https://news.ycombinator.com/item?id=22423632 . That discussion includes what you mentioned, plus a mention that it integrates with MariaDB, PrestoDB and Spark, including the ability to use an embedded MariaDB.
- stavrospap 6y ago
  Thanks for the post!
stavrospap 6y ago

willvarfar 6y ago

So, basically, tiledb is a file format like paraquet, avro and orc are formats? And it comes with a nice library that makes accessing the data fairly storage-transparent?

Shelnutt2 6y ago
Disclosure: I am a member of the TileDB team. Yes you are correct, TileDB introduces a storage format, but is also a full-fledged storage engine that implements it. Having the complete storage engine is what allows us to make accessing the data transparent regardless of the API or application you use to access the data. We are able to integrate into different computational tools without having to reimplement or sacrifice features.
- willvarfar 6y ago
  How do you compare to Iceberg?
  - Shelnutt2 6y ago
    Disclosure: I am a member of the TileDB team. Iceberg, similar to Delta Lake, is another layer you have to add on top of Parquet, which limits your mobility of compute. You can currently use it either with Spark or Presto, but not directly via language APIs or other compute engines (e.g., pandas, Dask, MariaDB, etc). For example, the pandas "read_parquet" will not be able to read your dataframe, you will have to use pyspark or presto + sql query to fetch the data into pandas. TileDB has updates and data versioning built into its format and storage engine, therefore reading an array directly via a TileDB language API, or Spark, Dask, PrestoDB or MariaDB will have the same behavior. You will be able to see the dataframe after all the changes, or time travel and "activate" only some of the changes in time.
    - willvarfar 6y ago
      How does tiledb deal with appends and upserts etc in s3? Is there success files, atomic folder renaming and things? Or?

jiangplus 6y ago

Looking at the TileDB docs, I like the idea that you can store and manipulate array efficiently. It works like a columnar storage, but organized in tiles. I am not clear how it could scale to multiple nodes and how to do queries. Maybe it should be integrated into postgres.

Shelnutt2 6y ago
Disclosure: I am a member of the TileDB team. TileDB fully supports multi-writer/multi-reader, so scaling compute to multiple nodes is mostly about choosing the computation layer. We have integrations with Spark, Dask and PrestoDB for distributed compute. In our cloud product we also offer serverless UDFs and SQL (via MariaDB) to allow for scaling out computations elastically without managing your own clusters. On the postgres side, we are looking at eventually adding a postgres storage engine, now that the new storage support is in PostgresSQL 11. To start with we have built a storage engine for MariaDB, as MariaDB has an excellent storage engine API, which I have past experience with. We hope to upstream the storage engine to MariaDB after the MariaDB 10.5 release. MariaDB integration: https://docs.tiledb.com/mariadb/ https://docs.tiledb.com/mariadb/ Spark integration: https://docs.tiledb.com/spark/ https://docs.tiledb.com/spark/ Dask integration: https://docs.tiledb.com/dask/ https://docs.tiledb.com/dask/ PrestoDB integration: https://docs.tiledb.com/prestodb/ https://docs.tiledb.com/prestodb/ Cloud Docs: https://docs.tiledb.com/cloud/ https://docs.tiledb.com/cloud/