4 ms·
The challenge with this is that S3 and friends are object stores, meaning you upload or download the whole file each time. As you can imagine, this will cost yo
by tn1 7y ago
The challenge with this is that S3 and friends are object stores, meaning you upload or download the whole file each time. As you can imagine, this will cost you tremendous bandwidth even to insert one row.
Furthermore, it doesn't solve the multiple writers problem, because (afaik) there's no way to lock a file on S3.
- pm90 7y ago> As you can imagine, this will cost you tremendous bandwidth even to insert one row. Is this the cost of network transfers changed by AWS or GCP? What if I'm hosting my app on e.g. EKS or GKE respectively?
- kondro 7y agoIt costs roughly $5/million PUTs and $0.40/million GETs on S3 in addition to the bandwidth and storage you use. S3 objects are also immutable. Once they’re written, they can’t be updated. A read-only version of this might be useful, but probably wouldn’t work in-place. Something that might be of interest is S3 SELECT support that lets you query a single (optionally compressed) CSV, JSON or Parquet file server-side at the same cost of a regular S3 GET. https://docs.aws.amazon.com/AmazonS3/latest/dev/selecting-content-from-objects.html https://docs.aws.amazon.com/AmazonS3/latest/dev/selecting-co... And if you really want relational (i.e. JOINs, aggregations and sub-queries) semantics on a bucket full of CSV, JSON, Parquet, ORC or regular-expression-describable files in a cost effective way that has great performance on buckets containing 100’s of TBs of data, definitely look at Athena which is only $5/TB of data scanned during a query.