3 ms·
Your example is way better than what AWS has on their page (marketing, analytics) which made me think immediately: you shouldn't use a file store for that but a
by barbazoo 6y ago
Your example is way better than what AWS has on their page (marketing, analytics) which made me think immediately: you shouldn't use a file store for that but a database.
- lambda_obrien 6y agoS3 is basically a database. You can store everything there and interact with it via SQL with Athena, for example. I could see utilizing all of Amazon's "infinitely scalable" services to build very simple and powerful software really easily.
- barbazoo 6y agoI think Athena has to get a lot less expensive for that to become true.
- kryptk 6y agoFor small numbers of large queries, Athena has been an incredible technology stack .. I pay pennies a month to run my analytics jobs on it (aggregates across a few hundred million rows, joins with tens of millions of rows into hundreds of thousands of rows). I start my data as .csv.gz but the first step is a CTAS to extract columns and convert to compressed parquet. This step basically costs the most but gives a 10x data size reduction to downstream steps. Athena does not work at all if you perform large numbers of small indexed read queries, definitely use a traditional database for that.
- lambda_obrien 6y agoYes, it's great if you setup the data structures properly, we pay a few bucks a month to run a lot of infra on it to query telemetry data over a few hours to a few days at a time. I think an equivalent cost in just a database server instance to run the same load would be at least 25 to 50 a month. You're not really locked into athena, either, it just takes standard sql, so you could easily transition to a dedicated box later.
- chrisandchris 6y agoI copied an example from the AWS blog post and adapted to our use-case. I did never work with it, but there is Athena that allows to query S3 in something like SQL. I think there _might_ be some cases were S3 is a useful database, as one can use pre-signed URLs to let any client write to the database (S3). That‘s something I‘m not sure how to set up with a classic database and without any API in between. IMHO, I think it‘s all about the same functionality with a simpler architecture.