4 ms·

While we're joking about AWS adventures, has anyone here seriously contemplated using S3 keys as a cheap blob store? It looks like they're a generous kilobyte a

by hansvm 4y ago

While we're joking about AWS adventures, has anyone here seriously contemplated using S3 keys as a cheap blob store? It looks like they're a generous kilobyte a pop, and prefix queries are supported, so you could hypothetically chunk your data into a [key|index|data_index| scheme and store a bunch of empty files in slightly under 1KB chunks -- ingress is nearly free (API calls, though you can batch them to keep this from being terrible), egress is expensive, and storage is free. Does that work?

csunbird 4y ago
That is why you also pay per request :)
berkle4455 4y ago
This is a terrible idea. S3 charges $0.005 per 1,000 PUT operations. So let's say you want to store 1GB (1e9 bytes) 1,000 bytes at a time, that's 1,000,000 keys. $0.005 * (1,000,000 PUT operations / 1,000) = $5.00 to write 1GB. The on-going storage cost I believe would be free since the files themselves are empty. Meanwhile writing a single 1GB file to S3? Costs you $0.023 per month storage, and the single write cost is $0.000005. It'll take 217 months to cost $5.00
- scarface74 4y ago
  Trivia: this API is I believe the only documented time that AWS has ever raised prices. It was done just for that reason.
  - dexterdog 4y ago
    If you track the extortion that AWS charges for egress bandwidth against both inflation and what bandwidth costs on average then they are always raising that rate.
  - QuinnyPig 4y ago
    They changed the pricing dimension when it was in beta. They've also raised SNS prices in some markets where regulatory fees increased, boosted pricing for domain registration several times, redid the pricing model for IoT Core when it was in beta, and changed the pricing model for AWS Config in ways that are kinda... unfortunate, for a lot of us.
    - scarface74 4y ago
      How do you know? Who are you suppose to be? I’m kidding, point taken. I listen to your podcast all of the time. You also called a project I was heavily involved in at AWS “This may be the worst AWS naming of 2021.” I took it as a point of nerd pride. Out of all of our bad naming convention, that said a lot.
- hansvm 4y ago
  Isn't that partially mitigated via S3 Batch?
  - berkle4455 4y ago
    Not to my knowledge? Can you link me to the S3 Batch documentation around PUT'ing multiple files from an external source onto S3?
dexterdog 4y ago
Is it cheaper than just using route53?
jwestbury 4y ago
Having worked at AWS for about half a decade in the past, I can guarantee that if you've thought of a way to abuse^W leverage their services for alternative purposes, someone has thought of it internally, as well. I knew someone back in 2014 who was working on a project to build a data store on tags, since tags were free. (The reality is that the tagging infrastructure would never have supported this kind of large-scale use, though. I remember a customer who was applying multiple unique tags to every single EC2 instance they launched, and by the time they had launched hundreds of thousands of instances, they were seeing very real latency when interacting with tags. I think the underlying performance issue was resolved, but these sorts of gotchas are all over the place.)

by hansvm 4y ago

csunbird 4y ago

That is why you also pay per request :)

berkle4455 4y ago

This is a terrible idea. S3 charges $0.005 per 1,000 PUT operations. So let's say you want to store 1GB (1e9 bytes) 1,000 bytes at a time, that's 1,000,000 keys. $0.005 * (1,000,000 PUT operations / 1,000) = $5.00 to write 1GB. The on-going storage cost I believe would be free since the files themselves are empty. Meanwhile writing a single 1GB file to S3? Costs you $0.023 per month storage, and the single write cost is $0.000005. It'll take 217 months to cost $5.00

scarface74 4y ago
Trivia: this API is I believe the only documented time that AWS has ever raised prices. It was done just for that reason.
- dexterdog 4y ago
  If you track the extortion that AWS charges for egress bandwidth against both inflation and what bandwidth costs on average then they are always raising that rate.
- QuinnyPig 4y ago
  They changed the pricing dimension when it was in beta. They've also raised SNS prices in some markets where regulatory fees increased, boosted pricing for domain registration several times, redid the pricing model for IoT Core when it was in beta, and changed the pricing model for AWS Config in ways that are kinda... unfortunate, for a lot of us.
  - scarface74 4y ago
    How do you know? Who are you suppose to be? I’m kidding, point taken. I listen to your podcast all of the time. You also called a project I was heavily involved in at AWS “This may be the worst AWS naming of 2021.” I took it as a point of nerd pride. Out of all of our bad naming convention, that said a lot.
hansvm 4y ago
Isn't that partially mitigated via S3 Batch?
- berkle4455 4y ago
  Not to my knowledge? Can you link me to the S3 Batch documentation around PUT'ing multiple files from an external source onto S3?

dexterdog 4y ago

Is it cheaper than just using route53?

jwestbury 4y ago

Having worked at AWS for about half a decade in the past, I can guarantee that if you've thought of a way to abuse^W leverage their services for alternative purposes, someone has thought of it internally, as well. I knew someone back in 2014 who was working on a project to build a data store on tags, since tags were free. (The reality is that the tagging infrastructure would never have supported this kind of large-scale use, though. I remember a customer who was applying multiple unique tags to every single EC2 instance they launched, and by the time they had launched hundreds of thousands of instances, they were seeing very real latency when interacting with tags. I think the underlying performance issue was resolved, but these sorts of gotchas are all over the place.)