3 ms·
Jake from TileDB, Inc. Genomics is a big field where sparse matrix storage is needed. Human genomes are stored as a diff off of a reference, which as you indic
by jakebol 9y ago
Jake from TileDB, Inc. Genomics is a big field where sparse matrix storage is needed. Human genomes are stored as a diff off of a reference, which as you indicated forms a graph which can be represented as a sparse matrix. In other fields of genomics, such as metagenomics, fragments of DNA when analyzed also have a graph like structure.
TileDB supports both dense and sparse arrays. It was designed around the concept of handling sparse arrays but dense arrays can be thought of a degenerate case of sparse array storage in TileDB. For dense arrays tile extents are contiguous and we don't materialize the coordinate values. This way all the concepts are the same and we can capture both use-cases. Sparse annotations to dense array values, such as NA or Null handling can also be captured as a sparse array fragment layered over a backing dense array.
I agree with you that for most use cases, storage will be dense. But it is useful to have one system that can handle both representations efficiently, with the sparse case not added on as an afterthought (it also makes the system simpler).