4 ms·
Nobody seems to bother but why anybody who use a cloud object storage will care about latency?
by up2isomorphism 3y ago
Nobody seems to bother but why anybody who use a cloud object storage will care about latency?
- Gasp0de 3y agoHow would you store petabytes of data with a low latency instead? We didn't really find a better solution.
- awiesenhofer 3y agoWhats your use case if I might ask? What kind of data is it?
- Gasp0de 3y agoIt is sensor data from tens of thousands of distributed sensors. Each sensor sends a reading every few seconds. We have a live view.in the frontend which is why latency plays a role.
- up2isomorphism 3y agoWould like to hear about the use case, we mostly found they are throughput sensitive rather than latency sensitive.
- Gasp0de 3y agoIt is sensor data from tens of thousands of distributed sensors. Each sensor sends a reading every few seconds. We have a live view.in the frontend which is why latency plays a role.
- wenc 3y agoFor me, I want to be able to query data faster. SQL queries on S3 data are affected by read latencies. When you're doing huge aggregations and joins say on Parquet files, the engine has to read the Parquet metadata multiple times, and read the relevant portions of the data. There are multiple reads going on, and every read incurs latency. Implementing a caching layer really helps, but it's not out of the box.
- Proven 3y ago[dead]
- klooney 3y agoBecause people are re-architecting their databases around cloud object storage, the new hotness is to try and get all your persistence there, with on-box just as a cache. This lets you offload all of the hard parts of persistence onto the AWS team.