3 ms·

> Speed - S3 is not very fast, even accounting for network delays. Latencies of 20-50ms are par for the course I've used S3 in past projects. These numbers are

> Speed - S3 is not very fast, even accounting for network delays. Latencies of 20-50ms are par for the course I've used S3 in past projects. These numbers are in line with my experience. And frankly, I'm good with them. S3 is dirt cheap compared to EBS, and it IMO doesn't generate a lot of production incidents (compared to, say, AWS RDS MySQL). The latency is on the high side compared to some things, but if the user is doing an "upload" operation of some sort, 50ms is tolerable, IMO. And it always seems to be way more efficient that just about everything else, in particular, it's usually next to some Mongo/SQL DB that is stuggling under the load of feature creep. S3 has basically never been on my bottleneck radar. The biggest thing I find people doing wrong, particularly in Python, is a. failing to keep the connection open (and so paying the TCP & TLS handshakes repeatedly when they don't need to, which adds a significant latency hit) and b. making boto calls they don't need to, such as "get_bucket" which in boto will check for the bucket's existence. You almost always can guarantee the bucket exists as a precondition of normal system operation… and the next call to GET/PUT object will fail anyways if something is amiss.

fpereiro 7y ago
Fully agree. 20-50ms for something like S3 is fantastic and proof that we live in the future. S3 is only slow when compared to interacting with redis or a FS layer powered by node (by about an order of magnitude). That's why I feel that complementing it with a self-managed FS can work well in certain settings. And interestingly enough, because S3 is so solid, you don't have to be so paranoid with your own FS layer, since S3 is there to bail you out in case of data loss. Regarding S3 pricing, ~0.09 USD per downloaded GB could potentially become a significant bill (~90 USD per TB). I'm building a service that serves pictures and limiting the financial downside is very important for the economics of the business. More than the actual magnitude, the certainty that costs are a linear and predictable function of storage makes me sleep better. An implicit point that I just realized by re-reading your comment: if you have a significant amount of data (let's say, > 100GB) a local FS is only cheaper if you're using instances or dedicated servers with large disks - and probably this is prohibitively expensive in AWS (EBS). I'm particularly considering Hetzner dedicated servers with large disks.
- say_it_as_it_is 7y ago
  Can you break down the financials for us?
  - fpereiro 7y ago
    At some point I'll do a public writeup of my planned cost structure. Hetzner's EX62 servers (https://www.hetzner.com/dedicated-rootserver https://www.hetzner.com/dedicated-rootserver) are about 10 USD/mo/TB of disk and it would be hard to go over their bandwidth limits . Even allowing for RAID and/or multiple locations/servers for a file node, the cost is still low. The main advantage in terms of cost is the possibility of not spending money on outgoing bandwidth.
    - ablekh 7y ago
      Hetzner's pricing for dedicated servers is impressive. However, I'm curious about how do you deal with (or whether you are worried about) the lack of their presence in North America, Asia, Oceania? Unless most of your customers are in Europe, latency might be an issue ... What is your experience in this regard?
      - fpereiro 7y ago
        That's a good question. I'm going to try first with Hetzner and measure latencies from different places; if this becomes an issue, I'll definitely consider complementing the infrastructure from elsewhere. But I'd rather start simple. I currently run a web service from North American servers and the performance here in Europe is quite good. I think this experience - very, very limited - gives me confidence that the wires across the world are good enough, provided your data center is close to the main pipes.
        ablekh 7y ago
        All right, good luck! Please share your Hetzner experiences here - I will be watching this thread.

fpereiro 7y ago

Fully agree. 20-50ms for something like S3 is fantastic and proof that we live in the future. S3 is only slow when compared to interacting with redis or a FS layer powered by node (by about an order of magnitude). That's why I feel that complementing it with a self-managed FS can work well in certain settings. And interestingly enough, because S3 is so solid, you don't have to be so paranoid with your own FS layer, since S3 is there to bail you out in case of data loss. Regarding S3 pricing, ~0.09 USD per downloaded GB could potentially become a significant bill (~90 USD per TB). I'm building a service that serves pictures and limiting the financial downside is very important for the economics of the business. More than the actual magnitude, the certainty that costs are a linear and predictable function of storage makes me sleep better. An implicit point that I just realized by re-reading your comment: if you have a significant amount of data (let's say, > 100GB) a local FS is only cheaper if you're using instances or dedicated servers with large disks - and probably this is prohibitively expensive in AWS (EBS). I'm particularly considering Hetzner dedicated servers with large disks.

say_it_as_it_is 7y ago
Can you break down the financials for us?
- fpereiro 7y ago
  At some point I'll do a public writeup of my planned cost structure. Hetzner's EX62 servers (https://www.hetzner.com/dedicated-rootserver https://www.hetzner.com/dedicated-rootserver) are about 10 USD/mo/TB of disk and it would be hard to go over their bandwidth limits . Even allowing for RAID and/or multiple locations/servers for a file node, the cost is still low. The main advantage in terms of cost is the possibility of not spending money on outgoing bandwidth.
  - ablekh 7y ago
    Hetzner's pricing for dedicated servers is impressive. However, I'm curious about how do you deal with (or whether you are worried about) the lack of their presence in North America, Asia, Oceania? Unless most of your customers are in Europe, latency might be an issue ... What is your experience in this regard?
    - fpereiro 7y ago
      That's a good question. I'm going to try first with Hetzner and measure latencies from different places; if this becomes an issue, I'll definitely consider complementing the infrastructure from elsewhere. But I'd rather start simple. I currently run a web service from North American servers and the performance here in Europe is quite good. I think this experience - very, very limited - gives me confidence that the wires across the world are good enough, provided your data center is close to the main pipes.
      - ablekh 7y ago
        All right, good luck! Please share your Hetzner experiences here - I will be watching this thread.