5 ms·

Relevant tweet [1]: "Do you know S3 throughput is higher than a SATA 3 controller and almost as much as PCIe 4.0 2x? Depending on the usecase, mounting S3 as a

by andr 7y ago

Relevant tweet [1]: "Do you know S3 throughput is higher than a SATA 3 controller and almost as much as PCIe 4.0 2x? Depending on the usecase, mounting S3 as a filesystem, not only makes sense, but it saves a LOT of money (and time) as well. You just need to use it a special kind of filesystem." [1] [1] https://twitter.com/dialtone_/status/1115130667677786112 https://twitter.com/dialtone_/status/1115130667677786112

hedora 7y ago
S3 latency and availability are many (3, 4, 5?) orders of magnitude worse than those other things. In other news, smart cars and supercars have comparable leg room, and are therefore interchangeable, depending on your use case.
- sacheendra 7y ago
  Latency I can see. I find it hard to believe S3 has a lower availability than a sata drive. Given that S3 is replicated.
  - cafxx 7y ago
    S3 maybe has better availability. Your DSL/fiber and router, that are on the critical path, definitely not.
    - sacheendra 7y ago
      I was speaking in the context of the cloud. If you are using S3, I presume you are using AWS EC2 or other services. Otherwise network egress costs dwarf everything else for most use cases. Except maybe as a repo of database backups.
  - khc 7y ago
    Depends on what you mean by availability. AWS S3 does return 500 errors from time to time and you are supposed to retry. There are also opaque rate limiting. These are mostly hidden from you if you use the AWS sdk but still is something to be aware of if you are using it in scale. And we haven't even started to talk about eventual consistency. I am the author of goofys (which exposes s3 like a filesystem) so obviously I believe this can work, but you do need to understand the tradeoffs (or you use goofys and hope that the tradeoffs I've made are correct :-P)
marknadal 7y ago
I do similar with my database ( https://github.com/amark/gun https://github.com/amark/gun ) and get great performance even out of free Heroku instances. 0 dev ops to manage too, can actually sleep at night. 100GB / day (100M records) for $10 all costs: https://youtu.be/x_WqBuEA7s8 https://youtu.be/x_WqBuEA7s8 TrailDB much better than ours at this though.

by andr 7y ago

hedora 7y ago

S3 latency and availability are many (3, 4, 5?) orders of magnitude worse than those other things. In other news, smart cars and supercars have comparable leg room, and are therefore interchangeable, depending on your use case.

sacheendra 7y ago
Latency I can see. I find it hard to believe S3 has a lower availability than a sata drive. Given that S3 is replicated.
- cafxx 7y ago
  S3 maybe has better availability. Your DSL/fiber and router, that are on the critical path, definitely not.
  - sacheendra 7y ago
    I was speaking in the context of the cloud. If you are using S3, I presume you are using AWS EC2 or other services. Otherwise network egress costs dwarf everything else for most use cases. Except maybe as a repo of database backups.
- khc 7y ago
  Depends on what you mean by availability. AWS S3 does return 500 errors from time to time and you are supposed to retry. There are also opaque rate limiting. These are mostly hidden from you if you use the AWS sdk but still is something to be aware of if you are using it in scale. And we haven't even started to talk about eventual consistency. I am the author of goofys (which exposes s3 like a filesystem) so obviously I believe this can work, but you do need to understand the tradeoffs (or you use goofys and hope that the tradeoffs I've made are correct :-P)

marknadal 7y ago

I do similar with my database ( https://github.com/amark/gun https://github.com/amark/gun ) and get great performance even out of free Heroku instances. 0 dev ops to manage too, can actually sleep at night. 100GB / day (100M records) for $10 all costs: https://youtu.be/x_WqBuEA7s8 https://youtu.be/x_WqBuEA7s8 TrailDB much better than ours at this though.