3 ms·

Opinion: Don’t put your business data in s3 objects! It can work if all you will ever need is a key value store, but as soon as you need SQL-like queries (join

Opinion: Don’t put your business data in s3 objects! It can work if all you will ever need is a key value store, but as soon as you need SQL-like queries (joins, filtering, transactions) you will be stuck reimplementing a sql engine in your app code. I recommend to try SQLite with some kind of replication (either Linode disk snapshots, Litestream etc). Having your state observable and local to your application instead of a slow http hop away makes development much easier.

cweagans 5y ago
+1 for SQLite. You might take a look at https://github.com/rqlite/rqlite https://github.com/rqlite/rqlite if you really need replication. SQLite can go pretty far on its own though.
- otoolep 5y ago
  rqlite author here, happy to answer questions about it. I should note that rqlite is a distributed database that uses SQLite as its database engine. This means in practice it does replicate a SQLite database to each node on the rqlite cluster -- but it's not a SQLite replication system per-se. https://github.com/rqlite/rqlite/blob/master/DOC/FAQ.md https://github.com/rqlite/rqlite/blob/master/DOC/FAQ.md
Hermitian909 5y ago
> you will be stuck reimplementing a sql engine in your app code. Only if you're intent on keeping the s3-only architecture when it no longer makes sense, and the transition to SQL need need not be hard if you've been disciplined about your data structures from the get go. I've been part of a transition from s3 only to s3/sql when business requirements changed, it only took us a week to set up a db, set up a workflow manager to insert necessary data into the db when something uploaded to s3, and ran some batch jobs for data that already existed. Total data in s3 was ~1pb. IMO it's all about whether your problem space fits well with s3-only, and sometimes it does!
Terretta 5y ago
You might not have to make a SQL engine immediately. - S3 Select: https://docs.aws.amazon.com/AmazonS3/latest/API/API_SelectObjectContent.html https://docs.aws.amazon.com/AmazonS3/latest/API/API_SelectOb... - Athena: https://towardsdatascience.com/query-data-from-s3-files-using-aws-athena-686a5b28e943 https://towardsdatascience.com/query-data-from-s3-files-usin...
- anothernewdude 5y ago
  Athena is a bad tool that if you have too much data or your queries take too long will just error out and give you empty results, and charge you for the privilege. It's cheaper and less pain to host your own Presto if you need it. How much is too much data? It depends. On things you won't be told. Varies day to day. Totally random.
deleted 5y ago
[deleted]

cweagans 5y ago

+1 for SQLite. You might take a look at https://github.com/rqlite/rqlite https://github.com/rqlite/rqlite if you really need replication. SQLite can go pretty far on its own though.

otoolep 5y ago
rqlite author here, happy to answer questions about it. I should note that rqlite is a distributed database that uses SQLite as its database engine. This means in practice it does replicate a SQLite database to each node on the rqlite cluster -- but it's not a SQLite replication system per-se. https://github.com/rqlite/rqlite/blob/master/DOC/FAQ.md https://github.com/rqlite/rqlite/blob/master/DOC/FAQ.md

Hermitian909 5y ago

> you will be stuck reimplementing a sql engine in your app code. Only if you're intent on keeping the s3-only architecture when it no longer makes sense, and the transition to SQL need need not be hard if you've been disciplined about your data structures from the get go. I've been part of a transition from s3 only to s3/sql when business requirements changed, it only took us a week to set up a db, set up a workflow manager to insert necessary data into the db when something uploaded to s3, and ran some batch jobs for data that already existed. Total data in s3 was ~1pb. IMO it's all about whether your problem space fits well with s3-only, and sometimes it does!

Terretta 5y ago

You might not have to make a SQL engine immediately. - S3 Select: https://docs.aws.amazon.com/AmazonS3/latest/API/API_SelectObjectContent.html https://docs.aws.amazon.com/AmazonS3/latest/API/API_SelectOb... - Athena: https://towardsdatascience.com/query-data-from-s3-files-using-aws-athena-686a5b28e943 https://towardsdatascience.com/query-data-from-s3-files-usin...

anothernewdude 5y ago
Athena is a bad tool that if you have too much data or your queries take too long will just error out and give you empty results, and charge you for the privilege. It's cheaper and less pain to host your own Presto if you need it. How much is too much data? It depends. On things you won't be told. Varies day to day. Totally random.

deleted 5y ago

[deleted]