5 ms·

I think there is really no doubt that the storage prices itself ( for all types) are pretty amazing. But let's face it: traffic costs are the huge elephant in t

by stemuk 10y ago

I think there is really no doubt that the storage prices itself ( for all types) are pretty amazing. But let's face it: traffic costs are the huge elephant in the room. By charging 12ct and more per GB, the traffic costs easily become your biggest expense, and make the storage price reduction from 2.6 to 2ct per GB almost forgetable. For me this is in no way acceptable and it seems to be a vicious attempt to sneak in some extra profit without having the customer noticing this upfront. Shure, these harsh words seam like a big exaggeration, but I literally never ever hear or read something about traffic costs in Googles fancy blog posts, and I would make the assumption that only a fraction of the HN community is aware of this fact. 120 bucks for a sloppy TB of traffic is just way too high.

kyledrake 10y ago
This x 1000. Even when you throw your own CDN on top of this, the bandwidth markup is nasty. Neocities would be unsustainable if we used GCS for our hosting. I'm paying about $0.01/GB right now, and I've seen market rate at half that. And that's not even directly using IP transit providers. You can get a gigabit unmetered for $450-1000/mo which you can shove a theoretical 324TB through every month. The difference in the numbers is so staggering I sometimes wonder if I'm even doing the math right. Perhaps their bandwidth is better somehow (prove it), but 12-18x better it is probably not, and having truffle shavings added to your IP transit really adds up when you're hauling a lot of traffic. If you're doing something with heavy BW usage and low margins, be careful with stuff like this. It quickly becomes much more expensive than doing it yourself. I'd love to be wrong here. I'm sitting next to 60 pounds of storage servers I'm setting up for a data center, they're taking up my entire living room. I would love to get out of the data persistence business forever. But at these BW rates, it's never going to happen.
- bedros 10y ago
  out of curiosity, what storage do you use? any experience using backblaze b2 as hot storage, and glacier or google coldline as backup
  - kyledrake 10y ago
    I'm planning to build a CephFS cluster at this point. I've given up on finding a cloud storage provider that will work well for us. This will require a fairly high fixed monthly cost to get space and transit at a datacenter, but after ~10-30TB BW you start to save a lot of money. There are trade offs. Aside from more upfront costs and a fixed monthly, Ceph is ridiculously complicated. Interface wise, they need some much better abstractions. B2 was a strong candidate, their BW rates are a little high still but approaching reasonable. The one issue is that they seem to have inconsistent latency. Not a problem for most use cases, but I need nearly all requests to come back in <100ms consistently, as I'm using this for web hosting. Their use case seems more focused on "hot standby backup" than on high availability ATM. I'm strongly considering them as the backup provider for my storage cluster. FWIW, GCS is not winning any best-of-show awards for latency either. S3 at the time I ran tests was doing a better job.
    - bedros 10y ago
      I read a lot of good stuff on CephFS, but never tried it.
    - SysArchitect 10y ago
      What makes Ceph complicated? The underlying idea and how Ceph works is fairly simple.
      - kyledrake 10y ago
        This is the "quick installation": http://docs.ceph.com/docs/master/start/ http://docs.ceph.com/docs/master/start/ It really shouldn't be this complex. I would love to just be able to boot an executable with a simple config file and be done with it. SeaweedFS shines a light on how this could be improved: https://github.com/chrislusf/seaweedfs https://github.com/chrislusf/seaweedfs
        rakoo 10y ago
        ... which naturally brings the question: why choose Ceph and not SeaweedFS ? I've read a bit about the latter (especially the Facebook paper), and the design and operability seem simple enough that it might be a good starting point
- Veratyr 10y ago
  You can get gigabit + a full cabinet of colocation from Hurricane Electric for $400/mo so you know: https://he.net/colocation.html https://he.net/colocation.html I'm sharing it with a few other people and it's great.
  - kyledrake 10y ago
    I've looked at this. Easily the best deal in town. A few notes for people considering it: - It's IP transit and a DC from HE, which means it's possibly not BGP peering through other transit providers? If this is true, if HE's network fails, so does your server's internet connection. Typically you want to be multi-homed to ensure redundancy, unless you're doing something like an Anycast CDN and you don't care if it craps out for a few hours. - HE is the cheapest transit for reasons. It may not be a big deal for what you're doing, but be aware of the differences https://news.ycombinator.com/item?id=5348624 https://news.ycombinator.com/item?id=5348624 - They only provide a 15A circuit (likely only 80% usable) for a 42U rack, which is pretty ridiculous. You'll blow through that pretty quickly if you're using dual Xeon servers. It's cheaper under their pricing to get a second 42U rack than it is to get the proper amount of power needed for a full cabinet. - If you're doing Anycast, be aware that HE doesn't provide any BGP communities except blackholing, which could make it hard to tune your network. (If any of this sounds annoying to deal with and think about, you know exactly why I'd prefer that the cloud providers had better BW costs so I could not have to do this anymore. Again, if the business model works with GCS, congratulations, use it.)
    - Veratyr 10y ago
      > It's IP transit and a DC from HE, which means it's possibly not BGP peering through other transit providers? The gigabit transit that comes with the cabinet is HE-only, yep. You can however buy transit and an interconnect from any other carrier in the datacenter (there are quite a few in FMT2 at least). > They only provide a 15A circuit for a 42U rack, which is pretty ridiculous. Yeah, it's pretty shitty. I ended up going with Xeon D for the power efficiency. Works pretty well.
  - chrisabrams 10y ago
- boulos 10y ago
  At our highest egress tier, our CDN offering around GCE (or GCS in Alpha), starts being competitive [1]. I agree though that if you just want some bandwidth, no major public cloud provider comes close to the kind of transit rates you'll find for "this isn't VOIP or gaming...". How much egress does Neocities do a month? Feel free to ping me if you'd prefer to discuss offline. Disclosure: I work for Google Cloud. [1] https://cloud.google.com/cdn/pricing#cache_egress_pricing_table https://cloud.google.com/cdn/pricing#cache_egress_pricing_ta...
  - kyledrake 10y ago
    $0.02 for North America is a rate where I start to go "okay, that could make sense" for entry level. I would call it priced right if it was extremely high quality bandwidth, as in you made some good peering arrangements with the eyeball ISPs and had some strong low-saturation IP transit in the mix. You would have to convince me of it's quality. My traceroutes should show a lot of direct regional peering to the major ISPs and Level 3 caliber transit. I really think you could differentiate a from your competition hugely by just using that $0.02/GB point for all your customers, regardless of the amount of egress they're using. It's a big chicken-and-egg problem to have those bandwidth costs initially as a startup, but then be able to handle them as a bigger organization. It made us move to our own infrastructure, and once we're here, I already have my infrastructure in a datacenter at that point, so why bother with the cloud? Even better (for me anyways) would be to provide an option for the "this isn't VOIP or gaming" bandwidth. Not everybody needs that. Actually, probably most people don't. For what I would use GCS for I only need transit to other datacenters. That bandwidth is much cheaper than major ISP peering bandwidth because there aren't monopolies like Comcast extorting everybody for peering.
donavanm 10y ago
There are a couple of interesting infrastructure/business trends driving this across providers. The call out of 12ct for throughput and 2ct for storage actually reflects a typical blended data environment. In short, data access frequency definitely follows the pareto principle. The vast majority of data is write only, with very little of it being accessed frequently. For those generalized storage populations the pricing actually aligns. Related is the issue of "stalled" throughput of dense storage like HDD. 4 years ago it was 4TB spindles at 5-7,200 RPM. We're up to 10TB per spindle these days. But rotational latency is still 5-7K rpm, limiting throughput to say 20-200MB/s depending on how much you like latency and queueing. As I recall HDDs are going to be up to ~20TB per spindle over the next few years. And still 5-7K rpm. Storage keeps getting cheaper, throughput is not. And lastly its really important that youre not paying for traffic. Youre paying for your providers network. That's their datacenters, their backbone, their leased fiber investments, their hundreds of millions in capital. It's not something as simple as "oh I can get cogent for $0.2/mbs." It's an old post, but for idea see https://news.ycombinator.com/item?id=7479030 https://news.ycombinator.com/item?id=7479030.
user5994461 10y ago
WTF are you talking about??? "Egress to a different Google Cloud Platform service within the same region => No charge" Source: https://cloud.google.com/compute/pricing#network https://cloud.google.com/compute/pricing#network

by stemuk 10y ago

kyledrake 10y ago

This x 1000. Even when you throw your own CDN on top of this, the bandwidth markup is nasty. Neocities would be unsustainable if we used GCS for our hosting. I'm paying about $0.01/GB right now, and I've seen market rate at half that. And that's not even directly using IP transit providers. You can get a gigabit unmetered for $450-1000/mo which you can shove a theoretical 324TB through every month. The difference in the numbers is so staggering I sometimes wonder if I'm even doing the math right. Perhaps their bandwidth is better somehow (prove it), but 12-18x better it is probably not, and having truffle shavings added to your IP transit really adds up when you're hauling a lot of traffic. If you're doing something with heavy BW usage and low margins, be careful with stuff like this. It quickly becomes much more expensive than doing it yourself. I'd love to be wrong here. I'm sitting next to 60 pounds of storage servers I'm setting up for a data center, they're taking up my entire living room. I would love to get out of the data persistence business forever. But at these BW rates, it's never going to happen.

bedros 10y ago
out of curiosity, what storage do you use? any experience using backblaze b2 as hot storage, and glacier or google coldline as backup
- kyledrake 10y ago
  I'm planning to build a CephFS cluster at this point. I've given up on finding a cloud storage provider that will work well for us. This will require a fairly high fixed monthly cost to get space and transit at a datacenter, but after ~10-30TB BW you start to save a lot of money. There are trade offs. Aside from more upfront costs and a fixed monthly, Ceph is ridiculously complicated. Interface wise, they need some much better abstractions. B2 was a strong candidate, their BW rates are a little high still but approaching reasonable. The one issue is that they seem to have inconsistent latency. Not a problem for most use cases, but I need nearly all requests to come back in <100ms consistently, as I'm using this for web hosting. Their use case seems more focused on "hot standby backup" than on high availability ATM. I'm strongly considering them as the backup provider for my storage cluster. FWIW, GCS is not winning any best-of-show awards for latency either. S3 at the time I ran tests was doing a better job.
  - bedros 10y ago
    I read a lot of good stuff on CephFS, but never tried it.
  - SysArchitect 10y ago
    What makes Ceph complicated? The underlying idea and how Ceph works is fairly simple.
    - kyledrake 10y ago
      This is the "quick installation": http://docs.ceph.com/docs/master/start/ http://docs.ceph.com/docs/master/start/ It really shouldn't be this complex. I would love to just be able to boot an executable with a simple config file and be done with it. SeaweedFS shines a light on how this could be improved: https://github.com/chrislusf/seaweedfs https://github.com/chrislusf/seaweedfs
      - rakoo 10y ago
        ... which naturally brings the question: why choose Ceph and not SeaweedFS ? I've read a bit about the latter (especially the Facebook paper), and the design and operability seem simple enough that it might be a good starting point
Veratyr 10y ago
You can get gigabit + a full cabinet of colocation from Hurricane Electric for $400/mo so you know: https://he.net/colocation.html https://he.net/colocation.html I'm sharing it with a few other people and it's great.
- kyledrake 10y ago
  I've looked at this. Easily the best deal in town. A few notes for people considering it: - It's IP transit and a DC from HE, which means it's possibly not BGP peering through other transit providers? If this is true, if HE's network fails, so does your server's internet connection. Typically you want to be multi-homed to ensure redundancy, unless you're doing something like an Anycast CDN and you don't care if it craps out for a few hours. - HE is the cheapest transit for reasons. It may not be a big deal for what you're doing, but be aware of the differences https://news.ycombinator.com/item?id=5348624 https://news.ycombinator.com/item?id=5348624 - They only provide a 15A circuit (likely only 80% usable) for a 42U rack, which is pretty ridiculous. You'll blow through that pretty quickly if you're using dual Xeon servers. It's cheaper under their pricing to get a second 42U rack than it is to get the proper amount of power needed for a full cabinet. - If you're doing Anycast, be aware that HE doesn't provide any BGP communities except blackholing, which could make it hard to tune your network. (If any of this sounds annoying to deal with and think about, you know exactly why I'd prefer that the cloud providers had better BW costs so I could not have to do this anymore. Again, if the business model works with GCS, congratulations, use it.)
  - Veratyr 10y ago
    > It's IP transit and a DC from HE, which means it's possibly not BGP peering through other transit providers? The gigabit transit that comes with the cabinet is HE-only, yep. You can however buy transit and an interconnect from any other carrier in the datacenter (there are quite a few in FMT2 at least). > They only provide a 15A circuit for a 42U rack, which is pretty ridiculous. Yeah, it's pretty shitty. I ended up going with Xeon D for the power efficiency. Works pretty well.
- chrisabrams 10y ago
boulos 10y ago
At our highest egress tier, our CDN offering around GCE (or GCS in Alpha), starts being competitive [1]. I agree though that if you just want some bandwidth, no major public cloud provider comes close to the kind of transit rates you'll find for "this isn't VOIP or gaming...". How much egress does Neocities do a month? Feel free to ping me if you'd prefer to discuss offline. Disclosure: I work for Google Cloud. [1] https://cloud.google.com/cdn/pricing#cache_egress_pricing_table https://cloud.google.com/cdn/pricing#cache_egress_pricing_ta...
- kyledrake 10y ago
  $0.02 for North America is a rate where I start to go "okay, that could make sense" for entry level. I would call it priced right if it was extremely high quality bandwidth, as in you made some good peering arrangements with the eyeball ISPs and had some strong low-saturation IP transit in the mix. You would have to convince me of it's quality. My traceroutes should show a lot of direct regional peering to the major ISPs and Level 3 caliber transit. I really think you could differentiate a from your competition hugely by just using that $0.02/GB point for all your customers, regardless of the amount of egress they're using. It's a big chicken-and-egg problem to have those bandwidth costs initially as a startup, but then be able to handle them as a bigger organization. It made us move to our own infrastructure, and once we're here, I already have my infrastructure in a datacenter at that point, so why bother with the cloud? Even better (for me anyways) would be to provide an option for the "this isn't VOIP or gaming" bandwidth. Not everybody needs that. Actually, probably most people don't. For what I would use GCS for I only need transit to other datacenters. That bandwidth is much cheaper than major ISP peering bandwidth because there aren't monopolies like Comcast extorting everybody for peering.

donavanm 10y ago

There are a couple of interesting infrastructure/business trends driving this across providers. The call out of 12ct for throughput and 2ct for storage actually reflects a typical blended data environment. In short, data access frequency definitely follows the pareto principle. The vast majority of data is write only, with very little of it being accessed frequently. For those generalized storage populations the pricing actually aligns. Related is the issue of "stalled" throughput of dense storage like HDD. 4 years ago it was 4TB spindles at 5-7,200 RPM. We're up to 10TB per spindle these days. But rotational latency is still 5-7K rpm, limiting throughput to say 20-200MB/s depending on how much you like latency and queueing. As I recall HDDs are going to be up to ~20TB per spindle over the next few years. And still 5-7K rpm. Storage keeps getting cheaper, throughput is not. And lastly its really important that youre not paying for traffic. Youre paying for your providers network. That's their datacenters, their backbone, their leased fiber investments, their hundreds of millions in capital. It's not something as simple as "oh I can get cogent for $0.2/mbs." It's an old post, but for idea see https://news.ycombinator.com/item?id=7479030 https://news.ycombinator.com/item?id=7479030.

user5994461 10y ago

WTF are you talking about??? "Egress to a different Google Cloud Platform service within the same region => No charge" Source: https://cloud.google.com/compute/pricing#network https://cloud.google.com/compute/pricing#network