3 ms·
we use partitioned parquet files in s3. we use a csv in the bucket root to track the files. i’m sure there’s a better way but for now the 2tb of data are stored
by swasheck 2y ago
we use partitioned parquet files in s3. we use a csv in the bucket root to track the files. i’m sure there’s a better way but for now the 2tb of data are stored cheaply and we get fast reads by only reading the partitions we need to read.
- Incipient 2y agoI'm curious how much simpler to build, manage, and run vs cost it would be to simply running a database on a large vultr/DO instance and paying for 2tb of storage? I feel like you'd get away with the whole thing for around $500/mo depending on how much compute was needed?
- ramraj07 2y agoYou just need to try it once to see the issue. Merely loading this amount of data onto a Postgres db will be hell.
- swasheck 2y agowell that's not the infrastructure we have. we are primarily an aws shop so we use the resources available to us in the context of our infrastructure decisions. it would be a hard sell to buy something outside of that ecosystem.
- Incipient 2y agoI understand that's the infrastructure you have. But that's more describing vendor lock-in haha. Most of my work is with clients that don't have any set infrastructure yet, so was curious if anyone had any anecdotes.