3 ms·
You don't write your own libc to talk to the filesystem, do you? Same vibe here imo. Use zerofs, juicefs. Use picomq, automq, s3stream. Use slatedb. Use lanced
by jauntywundrkind 1mo ago
You don't write your own libc to talk to the filesystem, do you?
Same vibe here imo. Use zerofs, juicefs. Use picomq, automq, s3stream. Use slatedb. Use lancedb, fusion, tonbo, iceberg, vertex. Use duckdb, data fusion, polars. These systems all natively speak to object storage. Use Qwikwit, rising wave / hummock, greptine mito, use warp stream, neon. Use celld. Over half of the projects here are very specifically about object storage.
Many deal with uniqueness, transactions, indicies. Which is extra impressive because its across machines. Managing data on a box creates a huge array of challenges and difficulties, backup and scaling and HA, etc etc. A huge reason to face object storage head on is that often you are headed towards that fray one way or another, and object storage makes an excellent easy to scale and manage disaggregation layer that keeps your problems from complecting together, is a clear, well known separation point. With an incredible ecosystem around it.