4 ms·

Interesting it doesn’t cover technologies for storing data. It does mention analytical operations though. Anyone have a suggestion for storing large amounts of

Interesting it doesn’t cover technologies for storing data. It does mention analytical operations though. Anyone have a suggestion for storing large amounts of ML data, training sets or otherwise? I’ve been using FeatureBase and weaviate, and would be interested in learning about other solutions.

kacperlukawski 4y ago
If you want to perform vector search over your data, then Qdrant (https://qdrant.tech https://qdrant.tech) is worth checking out.
gschoeni 4y ago
We've been working on an open source tool called Oxen to help store large ML datasets. It's optimized for large sets of unstructured data ie images, video, audio, text, as well as parquet or arrow style DataFrames. Would love to get some feedback on it! https://github.com/Oxen-AI/oxen-release#-oxen https://github.com/Oxen-AI/oxen-release#-oxen
wongarsu 4y ago
"ML data" could be just about anything: images, sounds, graphs, tabular data, vectors. Depending on what your data is, you might end up with very different storage solutions. Though I think one common approach is to just dump most data in an S3-compatible datastore of your choice (there's seaweedfs or ceph, or the cloud provider of your choice). Specialized databases make a lot of sense when you need features like vector similarity search though.
fzliu 4y ago
Vector databases in general are good for storing large amounts of unstructured data by first converting them into embeddings via ML models. There's also feature stores, which store and organize features for later use in model training or predictive analytics. Feature stores generally come in _before_ models get trained, while vector databases generally come _after_ (i.e. they use trained models). Milvus (https://milvus.io https://milvus.io) and Feast (https://feast.dev/ https://feast.dev/) are two of the most well known vector databases and feature stores, respectively.

kacperlukawski 4y ago

If you want to perform vector search over your data, then Qdrant (https://qdrant.tech https://qdrant.tech) is worth checking out.

gschoeni 4y ago

We've been working on an open source tool called Oxen to help store large ML datasets. It's optimized for large sets of unstructured data ie images, video, audio, text, as well as parquet or arrow style DataFrames. Would love to get some feedback on it! https://github.com/Oxen-AI/oxen-release#-oxen https://github.com/Oxen-AI/oxen-release#-oxen

wongarsu 4y ago

"ML data" could be just about anything: images, sounds, graphs, tabular data, vectors. Depending on what your data is, you might end up with very different storage solutions. Though I think one common approach is to just dump most data in an S3-compatible datastore of your choice (there's seaweedfs or ceph, or the cloud provider of your choice). Specialized databases make a lot of sense when you need features like vector similarity search though.

fzliu 4y ago

Vector databases in general are good for storing large amounts of unstructured data by first converting them into embeddings via ML models. There's also feature stores, which store and organize features for later use in model training or predictive analytics. Feature stores generally come in _before_ models get trained, while vector databases generally come _after_ (i.e. they use trained models). Milvus (https://milvus.io https://milvus.io) and Feast (https://feast.dev/ https://feast.dev/) are two of the most well known vector databases and feature stores, respectively.