5 ms·

IMO, it's not so much that they overcharge as that the service was created and priced for a different purpose. It's possible to serve huge files to tens of thou

by ufmace 6y ago

IMO, it's not so much that they overcharge as that the service was created and priced for a different purpose. It's possible to serve huge files to tens of thousands of people, but it's not really what it's meant for. Naturally, the pricing isn't really optimized for that, and a service that's designed for that can be an order of magnitude cheaper. Seems to me S3 is more designed for highly reliable data storage. If you wanna store a bunch of data in a way that's highly reliable and resilient to hardware failures and also not have to worry about managing RAIDs and clusters of servers for the data size, S3 is just the thing, and probably priced pretty fairly. It's also a capable and flexible service. It's possible to use it for things it isn't really designed for and have it behave pretty well. Well enough that you can mostly ignore that the other thing isn't really what it's designed around. The "mostly" is important, though. If you start hitting any extremes while using it in a different way, then you certainly can run into some pathological cases in the pricing structure and pay way too much for something that would have been cheaper in a service dedicated to that.

fxtentacle 6y ago
S3 is much cheaper than the egress traffic. As such, this isn't so much about S3 because you'd be paying the same price for that traffic if you were using EC2 instead of S3. Looking purely at the traffic price, I believe the word choice "overcharging" is warranted if you can buy the exact same product elsewhere for 90% less.
raiyu 6y ago
They over charge. Their cost is less than .01 cents per gb and look at their published pricing for customers. And I’m including networking equipment as well as head count in my cost, not just the cost of pipes.
- cortesoft 6y ago
  They charge so much for it as a disincentive to use it for CDN purposes. It is not architected to act as a CDN for people, and if they don't overcharge for egress, people will use it as a CDN because it is easy.
  - acdha 6y ago
    In that case, why isn’t EC2 egresa pricing more competitive with other services?
    - cortesoft 6y ago
      Same reason?
      - acdha 6y ago
        It's dubious that they don't want you using S3 for one of the advertised use-cases but even if we accept that, the logic wouldn't apply to EC2 since there's even less of a chance of that impacting other users. Building your own CDN on EC2 isn't going to produce a different workload than many other activities which are actively encouraged.
  - fxtentacle 6y ago
    Probably not. They have a tutorial on how to host static data with S3, so it appears to be an intended use of the service: https://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html https://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosti...
manquer 6y ago
If serving static files to thousands and millions of users is not cloudfront is for then I am not sure what else would a CDN be used for . You don't need to manage any storage to get your bill to drop, S3 can be fronted by your cached proxy (minio or just plain nginx works easily) it is trivial to setup on OVH or hetzner or any other VPS provider. You get all the benefits of S3 without the egress bill of a major cloud
- bigiain 6y ago
  > I am not sure what else would a CDN be used for He used a CDN, just didn't read the fineprint: “Things I should’ve known but didn’t.” Did you know that “The maximum file size Cloudflare’s CDN caches is 512MB for Free, Pro, and Business customers and 5GB for Enterprise customers.” That’s right, Cloudflare saw requests for a 13.7 GB file and sent them straight to origin every time BY DESIGN. Ouch! It's another case of misunderstanding and misusing a tool/service in my opinion. Cloudflare (and Cloudfront) are intended to be website CDNs. Their use case is _not_ serving 13.7GB objects - and so they didn't. (Whether "passing those requests straight back to the origin every time" is a better/more customer friendly thing than " blocking those requests since they're not being caches due to an object size rule" is a good question...) If you want "inexpensive large file hosting/serving", you don't use a combinations of "a service with 11 nines of durability" and "a web-infrastructure and website-security company, providing content-delivery-network". As you rightly point out, duct taping this together on VPS providers competing on low-cost bandwidth is the "right thing" for this use case. The gold-plating of 11 nines of durability and a CDN with over 200 globally distributed POPs almost (perhaps "should have"?) cost this guy $2.5k for "taking the easy way out without thinking through the consequences". In my head, it's like he had a Tesla Model S in the driveway, and needed to transport 6 tons of lead bricks across town, and thought "I know, I'll just load them up on the back seat and drive 50 miles!" and then getting surprised the car needed expensive repairs afterwards.
  - ufmace 6y ago
    Yup that's exactly what I was saying. Cloudfare is designed for website resources, mostly smallish images and bits of JS and CSS. They would probably regard 5MB as a rather large resource. I'm not gonna blame them for having no idea what to do when somebody tries to run a 13.7GB download through them.
illumin8 6y ago
If you're using it properly you can even boot VMs off of it, in the same region (S3 data transfer in region to EC2 is free) like this article is talking about. Just don't enable public read on your bucket to avoid a surprise bill.

by ufmace 6y ago

fxtentacle 6y ago

S3 is much cheaper than the egress traffic. As such, this isn't so much about S3 because you'd be paying the same price for that traffic if you were using EC2 instead of S3. Looking purely at the traffic price, I believe the word choice "overcharging" is warranted if you can buy the exact same product elsewhere for 90% less.

raiyu 6y ago

They over charge. Their cost is less than .01 cents per gb and look at their published pricing for customers. And I’m including networking equipment as well as head count in my cost, not just the cost of pipes.

cortesoft 6y ago
They charge so much for it as a disincentive to use it for CDN purposes. It is not architected to act as a CDN for people, and if they don't overcharge for egress, people will use it as a CDN because it is easy.
- acdha 6y ago
  In that case, why isn’t EC2 egresa pricing more competitive with other services?
  - cortesoft 6y ago
    Same reason?
    - acdha 6y ago
      It's dubious that they don't want you using S3 for one of the advertised use-cases but even if we accept that, the logic wouldn't apply to EC2 since there's even less of a chance of that impacting other users. Building your own CDN on EC2 isn't going to produce a different workload than many other activities which are actively encouraged.
- fxtentacle 6y ago
  Probably not. They have a tutorial on how to host static data with S3, so it appears to be an intended use of the service: https://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html https://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosti...

manquer 6y ago

If serving static files to thousands and millions of users is not cloudfront is for then I am not sure what else would a CDN be used for . You don't need to manage any storage to get your bill to drop, S3 can be fronted by your cached proxy (minio or just plain nginx works easily) it is trivial to setup on OVH or hetzner or any other VPS provider. You get all the benefits of S3 without the egress bill of a major cloud

bigiain 6y ago
> I am not sure what else would a CDN be used for He used a CDN, just didn't read the fineprint: “Things I should’ve known but didn’t.” Did you know that “The maximum file size Cloudflare’s CDN caches is 512MB for Free, Pro, and Business customers and 5GB for Enterprise customers.” That’s right, Cloudflare saw requests for a 13.7 GB file and sent them straight to origin every time BY DESIGN. Ouch! It's another case of misunderstanding and misusing a tool/service in my opinion. Cloudflare (and Cloudfront) are intended to be website CDNs. Their use case is _not_ serving 13.7GB objects - and so they didn't. (Whether "passing those requests straight back to the origin every time" is a better/more customer friendly thing than " blocking those requests since they're not being caches due to an object size rule" is a good question...) If you want "inexpensive large file hosting/serving", you don't use a combinations of "a service with 11 nines of durability" and "a web-infrastructure and website-security company, providing content-delivery-network". As you rightly point out, duct taping this together on VPS providers competing on low-cost bandwidth is the "right thing" for this use case. The gold-plating of 11 nines of durability and a CDN with over 200 globally distributed POPs almost (perhaps "should have"?) cost this guy $2.5k for "taking the easy way out without thinking through the consequences". In my head, it's like he had a Tesla Model S in the driveway, and needed to transport 6 tons of lead bricks across town, and thought "I know, I'll just load them up on the back seat and drive 50 miles!" and then getting surprised the car needed expensive repairs afterwards.
- ufmace 6y ago
  Yup that's exactly what I was saying. Cloudfare is designed for website resources, mostly smallish images and bits of JS and CSS. They would probably regard 5MB as a rather large resource. I'm not gonna blame them for having no idea what to do when somebody tries to run a 13.7GB download through them.

illumin8 6y ago

If you're using it properly you can even boot VMs off of it, in the same region (S3 data transfer in region to EC2 is free) like this article is talking about. Just don't enable public read on your bucket to avoid a surprise bill.