6 ms·

It’s difficult to tell from the landing page, but what exactly is this? It says a lot of things that it’s not, but that leaves it sounding like it’s an object s

It’s difficult to tell from the landing page, but what exactly is this? It says a lot of things that it’s not, but that leaves it sounding like it’s an object store with a tightly coupled map reduce framework allowing the “pluggable computer layer”. How wrong is this assessment?

aroch 6y ago
In genomics land its less an object store and more DB built to house giant but very sparse columnar data (think the position of genetic variants in hundreds of thousands of people) with support for interval operations and some other genomics things that traditional DBs don't support. That said, we evaluated TileDB for genomics recently and found it lacking for our use case.
- stavrospap 6y ago
  Stavros from TileDB here. Great description of the genomics use case for TileDB. We'd be interested in learning what limitations you've found. Happy to discuss over email as well (stavros@tiledb.com).
  - aroch 6y ago
    Hey Stavros. We were looking for a data-store to integrate into a clinical genomics LIMS that supports in-system analysis. We deal with de novo sequenced clinical samples (and not genotyped samples, which seems to be what TileDB-vcf had in mind?). There are some edge cases that TileDB-vcf explicitly disallows (updates/reinserts to the same sample, overlapping variants) that are not edge cases for us but rather common occurrences.
    - stavrospap 6y ago
      This is an API issue with TileDB-VCF. The core TileDB library supports inserts/appends/overwrites without issues and we just need to expose those operations in the TileDB-VCF APIs. Added to our backlog, thanks!
  - deleted 6y ago
    [deleted]
- qeternity 6y ago
  Sure but that’s a very specific use case and not a “universal database”. Object store after all is just cloud speak for KV store which is the foundation of data retrieval. So I’m still unclear what this actually does. Has it invented something new or is it tying together mature tech in a really powerful way?
- hobofan 6y ago
  > That said, we evaluated TileDB for genomics recently and found it lacking for our use case. Could you share what solution you landed on instead?
  - aroch 6y ago
    Hail.is is what we landed on. There are some things we don't like with it too but weren't deal breakers
    - stavrospap 6y ago
      TileDB and Hail are rather complementary. We have customers that use TileDB to store and manage their variants, and Hail to perform GWAS (by exporting from TileDB to Hail format). We are currently designing a tighter integration with Hail. This expands on our vision for a universal data engine that integrates with pretty much everything out there and does not lock you in a single framework (e.g., Spark).
      - aroch 6y ago
        That was our feeling about the two products as well, the limitations w/ TileDB-vcf though sort of forced our hands. I was (and still am) of the opinion TileDB would be a good variant store since it does do so many of the things we want and does them well
    - johnc1231 6y ago
      Hail team member here. I'd be curious to hear what the things you don't like about hail are. Hopefully they're things that are already on our road map
- kristoff_it 6y ago
  Take a look at Redis. With the modules API it's possible to load pre-existing C libraries and consume them through simple Redis commands and there's Redis clients for each and every language out there. I did a GSoC project years ago and I hated having to use MongoDB for manpulating VCF files (it was the time where "nosql" was all the rage and all project ideas had to have mongo somewhere). Redis Modules would have been the one thing that would have made sense (that API was introduced much later).
  - aroch 6y ago
    Oh, hmm, that's very interesting and not something I would have thought about! :)
gas9S9zw3P9c 6y ago
I'm in the same camp. I'm quite interested since it mentions asset data as an example, but I have no idea what this does from looking at the landing page. Does someone have an end-to-end example? Since this stores arrays, is this kind of like Apache Arrow but with a persistence layer? Is this suited for large amounts (~1TB) of time series data?
- Shelnutt2 6y ago
  Please see my comment above in the thread for a full description of TileDB[1]. Compared to Apache Arrow we have some similarities but also some significant differences. Arrow as a project has many components, the most directly comparable are the in-memory data structure and parquet for on-disk storage. For the in-memory data structure TileDB has similar goals of doing zero-copy for moving data between libraries and applications. In-fact we even use Arrow in our TileDB-VCF[2] genomics project for zero copy into spark and python (pyarrow). We are looking to expand support for arrow into other integrations where appropriate. For parquet, a brief comparison is that parquet is a one dimensional columnar storage format, where TileDB is multi-dimensional. TileDB subsumes parquet in that we include all of its functionality and more. TileDB natively handles the eventual consistency of cloud object stores, and natively handles updates through its MVCC design. TileDB is a complete storage engine, not only a file format. That said parquet does have some advantages, a primary one it has a several year headstart on TileDB in being integrated into many tools, so its more well known. > Is this suited for large amounts (~1TB) of time series data? Yes, it's well suited for time series data. We natively support timestamp/datetime fields, in the core library and in many of our integrations (TileDB-Py, TileDB-R, Spark, MariaDB to name a few). We allow for fast sub-slicing on the dimension. You also have configurable tiling[3] so you can shape the array to fit your timestamp granularity and volume. The support for updates also can help if your timeseries data ever gets updated. Many timeseries databases don't recommend updates to records, or they recommend no primary keys and to have duplicates. TileDB supports fast and efficient updates (and duplicates) so you have full control of your design and implementation. [1] https://news.ycombinator.com/item?id=23897086 https://news.ycombinator.com/item?id=23897086 [2] https://github.com/TileDB-Inc/TileDB-VCF https://github.com/TileDB-Inc/TileDB-VCF [3] https://docs.tiledb.com/main/performance-tips/choosing-tiling-and-cell-layout https://docs.tiledb.com/main/performance-tips/choosing-tilin...
esquire_900 6y ago
The landing page seems to be a bit confusing. This page does a better job at explaining it imo: https://tiledb.com/embedded/ https://tiledb.com/embedded/ Without experience with the product, it's core seems to be a wrapper around stored data (sql, hdf, etc.) over several protocols (mysql, static files (s3) etc).
d__k 6y ago
I doubt that it is like an object store. I think that it is built around the notion of multidimensional array (matrix) by focusing on data science tools. Yet, they use the term "table" which is somewhat confusing because there are significant differences between tables and arrays. Maybe they use (sparse) arrays at physical level and then support other data models at the level of operations. For me it is not clear.
Shelnutt2 6y ago
Seth from TileDB here. The foundational invention is the TileDB universal storage engine based on dense and sparse multi-dimensional arrays. Genomic variants are 2D sparse arrays. Images and video are dense arrays. KV are interestingly sparse arrays as well[1]. TileDB is the first solution that can efficiently model all data with a single powerful data structure. TileDB universal storage engine natively supports cloud object stores, as well as local filesystems and we natively handle the eventual consistency through our MVCC design. On top of the TileDB storage engine we have built and worked on integrations with a range of existing computation tools and frameworks. Our goal is to make the computation pluggable and provide fast and efficient (zero-copy or apache arrow where possible) integrations to allow you to continue to use your favorite computation tools in your favorite language. A few examples are our integrations with numpy/Pandas, Spark, MariaDB, gdal and more. In addition, we built TileDB Cloud on our storage engine, which has two more innovations: (1) easy data sharing and logging at global scale (beyond organizations), and (2) a complete serverless infrastructure for scaling out compute, very similar to the DAG approach of dask.delayed. [1] https://docs.tiledb.com/main/handling-key-value-stores https://docs.tiledb.com/main/handling-key-value-stores [2] https://docs.tiledb.com/cloud/client-api/task-graphs https://docs.tiledb.com/cloud/client-api/task-graphs
- ben509 6y ago
  > TileDB is the first solution that can efficiently model all data with a single powerful data structure. How do you implement constraints in that model? Something like a foreign key constraint, for instance.
  - eloff 6y ago
    It doesn't seem like a relational database, and eventual consistency is going to make relational constraints problematic.
    - ben509 6y ago
      No doubt, but people need constraints. So they'll cobble together ad hoc constraints in their applications to prevent their stuff from crashing or being wrong. A DBMS's raison d'etre is to take those ad hoc code patterns and formalize them.

aroch 6y ago

In genomics land its less an object store and more DB built to house giant but very sparse columnar data (think the position of genetic variants in hundreds of thousands of people) with support for interval operations and some other genomics things that traditional DBs don't support. That said, we evaluated TileDB for genomics recently and found it lacking for our use case.

stavrospap 6y ago
Stavros from TileDB here. Great description of the genomics use case for TileDB. We'd be interested in learning what limitations you've found. Happy to discuss over email as well (stavros@tiledb.com).
- aroch 6y ago
  Hey Stavros. We were looking for a data-store to integrate into a clinical genomics LIMS that supports in-system analysis. We deal with de novo sequenced clinical samples (and not genotyped samples, which seems to be what TileDB-vcf had in mind?). There are some edge cases that TileDB-vcf explicitly disallows (updates/reinserts to the same sample, overlapping variants) that are not edge cases for us but rather common occurrences.
  - stavrospap 6y ago
    This is an API issue with TileDB-VCF. The core TileDB library supports inserts/appends/overwrites without issues and we just need to expose those operations in the TileDB-VCF APIs. Added to our backlog, thanks!
- deleted 6y ago
  [deleted]
qeternity 6y ago
Sure but that’s a very specific use case and not a “universal database”. Object store after all is just cloud speak for KV store which is the foundation of data retrieval. So I’m still unclear what this actually does. Has it invented something new or is it tying together mature tech in a really powerful way?
hobofan 6y ago
> That said, we evaluated TileDB for genomics recently and found it lacking for our use case. Could you share what solution you landed on instead?
- aroch 6y ago
  Hail.is is what we landed on. There are some things we don't like with it too but weren't deal breakers
  - stavrospap 6y ago
    TileDB and Hail are rather complementary. We have customers that use TileDB to store and manage their variants, and Hail to perform GWAS (by exporting from TileDB to Hail format). We are currently designing a tighter integration with Hail. This expands on our vision for a universal data engine that integrates with pretty much everything out there and does not lock you in a single framework (e.g., Spark).
    - aroch 6y ago
      That was our feeling about the two products as well, the limitations w/ TileDB-vcf though sort of forced our hands. I was (and still am) of the opinion TileDB would be a good variant store since it does do so many of the things we want and does them well
  - johnc1231 6y ago
    Hail team member here. I'd be curious to hear what the things you don't like about hail are. Hopefully they're things that are already on our road map
kristoff_it 6y ago
Take a look at Redis. With the modules API it's possible to load pre-existing C libraries and consume them through simple Redis commands and there's Redis clients for each and every language out there. I did a GSoC project years ago and I hated having to use MongoDB for manpulating VCF files (it was the time where "nosql" was all the rage and all project ideas had to have mongo somewhere). Redis Modules would have been the one thing that would have made sense (that API was introduced much later).
- aroch 6y ago
  Oh, hmm, that's very interesting and not something I would have thought about! :)

gas9S9zw3P9c 6y ago

I'm in the same camp. I'm quite interested since it mentions asset data as an example, but I have no idea what this does from looking at the landing page. Does someone have an end-to-end example? Since this stores arrays, is this kind of like Apache Arrow but with a persistence layer? Is this suited for large amounts (~1TB) of time series data?

Shelnutt2 6y ago
Please see my comment above in the thread for a full description of TileDB[1]. Compared to Apache Arrow we have some similarities but also some significant differences. Arrow as a project has many components, the most directly comparable are the in-memory data structure and parquet for on-disk storage. For the in-memory data structure TileDB has similar goals of doing zero-copy for moving data between libraries and applications. In-fact we even use Arrow in our TileDB-VCF[2] genomics project for zero copy into spark and python (pyarrow). We are looking to expand support for arrow into other integrations where appropriate. For parquet, a brief comparison is that parquet is a one dimensional columnar storage format, where TileDB is multi-dimensional. TileDB subsumes parquet in that we include all of its functionality and more. TileDB natively handles the eventual consistency of cloud object stores, and natively handles updates through its MVCC design. TileDB is a complete storage engine, not only a file format. That said parquet does have some advantages, a primary one it has a several year headstart on TileDB in being integrated into many tools, so its more well known. > Is this suited for large amounts (~1TB) of time series data? Yes, it's well suited for time series data. We natively support timestamp/datetime fields, in the core library and in many of our integrations (TileDB-Py, TileDB-R, Spark, MariaDB to name a few). We allow for fast sub-slicing on the dimension. You also have configurable tiling[3] so you can shape the array to fit your timestamp granularity and volume. The support for updates also can help if your timeseries data ever gets updated. Many timeseries databases don't recommend updates to records, or they recommend no primary keys and to have duplicates. TileDB supports fast and efficient updates (and duplicates) so you have full control of your design and implementation. [1] https://news.ycombinator.com/item?id=23897086 https://news.ycombinator.com/item?id=23897086 [2] https://github.com/TileDB-Inc/TileDB-VCF https://github.com/TileDB-Inc/TileDB-VCF [3] https://docs.tiledb.com/main/performance-tips/choosing-tiling-and-cell-layout https://docs.tiledb.com/main/performance-tips/choosing-tilin...

esquire_900 6y ago

The landing page seems to be a bit confusing. This page does a better job at explaining it imo: https://tiledb.com/embedded/ https://tiledb.com/embedded/ Without experience with the product, it's core seems to be a wrapper around stored data (sql, hdf, etc.) over several protocols (mysql, static files (s3) etc).

d__k 6y ago

I doubt that it is like an object store. I think that it is built around the notion of multidimensional array (matrix) by focusing on data science tools. Yet, they use the term "table" which is somewhat confusing because there are significant differences between tables and arrays. Maybe they use (sparse) arrays at physical level and then support other data models at the level of operations. For me it is not clear.

Shelnutt2 6y ago

Seth from TileDB here. The foundational invention is the TileDB universal storage engine based on dense and sparse multi-dimensional arrays. Genomic variants are 2D sparse arrays. Images and video are dense arrays. KV are interestingly sparse arrays as well[1]. TileDB is the first solution that can efficiently model all data with a single powerful data structure. TileDB universal storage engine natively supports cloud object stores, as well as local filesystems and we natively handle the eventual consistency through our MVCC design. On top of the TileDB storage engine we have built and worked on integrations with a range of existing computation tools and frameworks. Our goal is to make the computation pluggable and provide fast and efficient (zero-copy or apache arrow where possible) integrations to allow you to continue to use your favorite computation tools in your favorite language. A few examples are our integrations with numpy/Pandas, Spark, MariaDB, gdal and more. In addition, we built TileDB Cloud on our storage engine, which has two more innovations: (1) easy data sharing and logging at global scale (beyond organizations), and (2) a complete serverless infrastructure for scaling out compute, very similar to the DAG approach of dask.delayed. [1] https://docs.tiledb.com/main/handling-key-value-stores https://docs.tiledb.com/main/handling-key-value-stores [2] https://docs.tiledb.com/cloud/client-api/task-graphs https://docs.tiledb.com/cloud/client-api/task-graphs

ben509 6y ago
> TileDB is the first solution that can efficiently model all data with a single powerful data structure. How do you implement constraints in that model? Something like a foreign key constraint, for instance.
- eloff 6y ago
  It doesn't seem like a relational database, and eventual consistency is going to make relational constraints problematic.
  - ben509 6y ago
    No doubt, but people need constraints. So they'll cobble together ad hoc constraints in their applications to prevent their stuff from crashing or being wrong. A DBMS's raison d'etre is to take those ad hoc code patterns and formalize them.