3 ms·

> most ML data is not sparse. This brings up a question: in what fields does one find heavy use of large, sparse matrices that need to be persisted and queryab

by wenc 9y ago

> most ML data is not sparse. This brings up a question: in what fields does one find heavy use of large, sparse matrices that need to be persisted and queryable? In my mind, sparse matrices typically occur in the context of graphs/relationships, e.g. PageRank, logistic networks, adjacency matrices, etc. They also tend to be a property of Hessian matrices (2nd order derivative for a multivariate system). But typically these are intermediate quantities that are discarded after a computation completes.

jakebol 9y ago
Jake from TileDB, Inc. Genomics is a big field where sparse matrix storage is needed. Human genomes are stored as a diff off of a reference, which as you indicated forms a graph which can be represented as a sparse matrix. In other fields of genomics, such as metagenomics, fragments of DNA when analyzed also have a graph like structure. TileDB supports both dense and sparse arrays. It was designed around the concept of handling sparse arrays but dense arrays can be thought of a degenerate case of sparse array storage in TileDB. For dense arrays tile extents are contiguous and we don't materialize the coordinate values. This way all the concepts are the same and we can capture both use-cases. Sparse annotations to dense array values, such as NA or Null handling can also be captured as a sparse array fragment layered over a backing dense array. I agree with you that for most use cases, storage will be dense. But it is useful to have one system that can handle both representations efficiently, with the sparse case not added on as an afterthought (it also makes the system simpler).
stavrospap 9y ago
Stavros form TileDB, Inc. here: Another area of great interest to us is point cloud data, which are essentially 3D points in a super sparse space. In fact, any application dealing with spatio-temporal data (which are 2D/3D/4D data that tend to be sparse and skewed) can take advantage of TileDB.

by wenc 9y ago

jakebol 9y ago

Jake from TileDB, Inc. Genomics is a big field where sparse matrix storage is needed. Human genomes are stored as a diff off of a reference, which as you indicated forms a graph which can be represented as a sparse matrix. In other fields of genomics, such as metagenomics, fragments of DNA when analyzed also have a graph like structure. TileDB supports both dense and sparse arrays. It was designed around the concept of handling sparse arrays but dense arrays can be thought of a degenerate case of sparse array storage in TileDB. For dense arrays tile extents are contiguous and we don't materialize the coordinate values. This way all the concepts are the same and we can capture both use-cases. Sparse annotations to dense array values, such as NA or Null handling can also be captured as a sparse array fragment layered over a backing dense array. I agree with you that for most use cases, storage will be dense. But it is useful to have one system that can handle both representations efficiently, with the sparse case not added on as an afterthought (it also makes the system simpler).

stavrospap 9y ago

Stavros form TileDB, Inc. here: Another area of great interest to us is point cloud data, which are essentially 3D points in a super sparse space. In fact, any application dealing with spatio-temporal data (which are 2D/3D/4D data that tend to be sparse and skewed) can take advantage of TileDB.