4 ms·
It's an interesting question! The problem is that the data is spread everywhere - no choice about that. So with that in mind, how do you query that data? Today
by TheIronYuppie 3y ago
It's an interesting question!
The problem is that the data is spread everywhere - no choice about that. So with that in mind, how do you query that data? Today, the idea is that you HAVE to put it into a central location. With tools like Bacalhau[1] and DuckDB [2], you no longer have to - a single query can be sharded amongst all your data - EFFECTIVELY giving you a lot of what you want from a data lake.
It's not a replacement, but if you can do a few of these items WITHOUT moving the data, you will be able to see really significant cost and time savings.
Disclosure: I co-founded Bacalhau
[1] https://github.com/bacalhau-project/bacalhau https://github.com/bacalhau-project/bacalhau
[2] https://github.com/duckdb/duckdb https://github.com/duckdb/duckdb