7 ms·

This x 1000. Even when you throw your own CDN on top of this, the bandwidth markup is nasty. Neocities would be unsustainable if we used GCS for our hosting. I

This x 1000. Even when you throw your own CDN on top of this, the bandwidth markup is nasty. Neocities would be unsustainable if we used GCS for our hosting. I'm paying about $0.01/GB right now, and I've seen market rate at half that. And that's not even directly using IP transit providers. You can get a gigabit unmetered for $450-1000/mo which you can shove a theoretical 324TB through every month. The difference in the numbers is so staggering I sometimes wonder if I'm even doing the math right. Perhaps their bandwidth is better somehow (prove it), but 12-18x better it is probably not, and having truffle shavings added to your IP transit really adds up when you're hauling a lot of traffic. If you're doing something with heavy BW usage and low margins, be careful with stuff like this. It quickly becomes much more expensive than doing it yourself. I'd love to be wrong here. I'm sitting next to 60 pounds of storage servers I'm setting up for a data center, they're taking up my entire living room. I would love to get out of the data persistence business forever. But at these BW rates, it's never going to happen.

bedros 10y ago
out of curiosity, what storage do you use? any experience using backblaze b2 as hot storage, and glacier or google coldline as backup
- kyledrake 10y ago
  I'm planning to build a CephFS cluster at this point. I've given up on finding a cloud storage provider that will work well for us. This will require a fairly high fixed monthly cost to get space and transit at a datacenter, but after ~10-30TB BW you start to save a lot of money. There are trade offs. Aside from more upfront costs and a fixed monthly, Ceph is ridiculously complicated. Interface wise, they need some much better abstractions. B2 was a strong candidate, their BW rates are a little high still but approaching reasonable. The one issue is that they seem to have inconsistent latency. Not a problem for most use cases, but I need nearly all requests to come back in <100ms consistently, as I'm using this for web hosting. Their use case seems more focused on "hot standby backup" than on high availability ATM. I'm strongly considering them as the backup provider for my storage cluster. FWIW, GCS is not winning any best-of-show awards for latency either. S3 at the time I ran tests was doing a better job.
  - bedros 10y ago
    I read a lot of good stuff on CephFS, but never tried it.
  - SysArchitect 10y ago
    What makes Ceph complicated? The underlying idea and how Ceph works is fairly simple.
    - kyledrake 10y ago
      This is the "quick installation": http://docs.ceph.com/docs/master/start/ http://docs.ceph.com/docs/master/start/ It really shouldn't be this complex. I would love to just be able to boot an executable with a simple config file and be done with it. SeaweedFS shines a light on how this could be improved: https://github.com/chrislusf/seaweedfs https://github.com/chrislusf/seaweedfs
      - rakoo 10y ago
        ... which naturally brings the question: why choose Ceph and not SeaweedFS ? I've read a bit about the latter (especially the Facebook paper), and the design and operability seem simple enough that it might be a good starting point
        kyledrake 10y ago
        Ceph is more production-ready, and has a lot of things SeaweedFS I believe doesn't yet have (such as consistency checking and rebuilding of failed nodes). It also has FS layer capability. SeaweedFS can use a filer but it currently doesn't support indexing (so you can't do subdirectory lookups for example, you would need to know the full filename). SeaweedFS is fundamentally trying to do a different thing than Ceph is, and it benefits certain use cases. Reading the Facebook Haystack paper will give you an idea of the differences. That said, I'm extremely impressed with how simple the interface is. It's very easy to get started.
  - brianwski 10y ago
- stemuk 10y ago
  I am currently using the Hetzner SX131 in RAID 1 and http://minio.io http://minio.io. For about 200€ you get (2x) 30 TB and 50 TB traffic included. Thats about 0.007€ per GB with "free" traffic.
  - bedros 10y ago
    is minio.io a service or server like cephfs?
    - stemuk 10y ago
      It is similar to cephfs, but I found it to be more performant and easier to set up.
    - Veratyr 10y ago
      Minio is basically a local implementation of the S3 API so you can access your stuff remotely and integrate with S3 compatible services using it.
    - bedros 10y ago
      reading the documentation, it looks like more of an NFS replacement with S3 API; I did not see any mention of clustering and redundancy
Veratyr 10y ago
You can get gigabit + a full cabinet of colocation from Hurricane Electric for $400/mo so you know: https://he.net/colocation.html https://he.net/colocation.html I'm sharing it with a few other people and it's great.
- kyledrake 10y ago
  I've looked at this. Easily the best deal in town. A few notes for people considering it: - It's IP transit and a DC from HE, which means it's possibly not BGP peering through other transit providers? If this is true, if HE's network fails, so does your server's internet connection. Typically you want to be multi-homed to ensure redundancy, unless you're doing something like an Anycast CDN and you don't care if it craps out for a few hours. - HE is the cheapest transit for reasons. It may not be a big deal for what you're doing, but be aware of the differences https://news.ycombinator.com/item?id=5348624 https://news.ycombinator.com/item?id=5348624 - They only provide a 15A circuit (likely only 80% usable) for a 42U rack, which is pretty ridiculous. You'll blow through that pretty quickly if you're using dual Xeon servers. It's cheaper under their pricing to get a second 42U rack than it is to get the proper amount of power needed for a full cabinet. - If you're doing Anycast, be aware that HE doesn't provide any BGP communities except blackholing, which could make it hard to tune your network. (If any of this sounds annoying to deal with and think about, you know exactly why I'd prefer that the cloud providers had better BW costs so I could not have to do this anymore. Again, if the business model works with GCS, congratulations, use it.)
  - Veratyr 10y ago
    > It's IP transit and a DC from HE, which means it's possibly not BGP peering through other transit providers? The gigabit transit that comes with the cabinet is HE-only, yep. You can however buy transit and an interconnect from any other carrier in the datacenter (there are quite a few in FMT2 at least). > They only provide a 15A circuit for a 42U rack, which is pretty ridiculous. Yeah, it's pretty shitty. I ended up going with Xeon D for the power efficiency. Works pretty well.
- chrisabrams 10y ago
boulos 10y ago
At our highest egress tier, our CDN offering around GCE (or GCS in Alpha), starts being competitive [1]. I agree though that if you just want some bandwidth, no major public cloud provider comes close to the kind of transit rates you'll find for "this isn't VOIP or gaming...". How much egress does Neocities do a month? Feel free to ping me if you'd prefer to discuss offline. Disclosure: I work for Google Cloud. [1] https://cloud.google.com/cdn/pricing#cache_egress_pricing_table https://cloud.google.com/cdn/pricing#cache_egress_pricing_ta...
- kyledrake 10y ago
  $0.02 for North America is a rate where I start to go "okay, that could make sense" for entry level. I would call it priced right if it was extremely high quality bandwidth, as in you made some good peering arrangements with the eyeball ISPs and had some strong low-saturation IP transit in the mix. You would have to convince me of it's quality. My traceroutes should show a lot of direct regional peering to the major ISPs and Level 3 caliber transit. I really think you could differentiate a from your competition hugely by just using that $0.02/GB point for all your customers, regardless of the amount of egress they're using. It's a big chicken-and-egg problem to have those bandwidth costs initially as a startup, but then be able to handle them as a bigger organization. It made us move to our own infrastructure, and once we're here, I already have my infrastructure in a datacenter at that point, so why bother with the cloud? Even better (for me anyways) would be to provide an option for the "this isn't VOIP or gaming" bandwidth. Not everybody needs that. Actually, probably most people don't. For what I would use GCS for I only need transit to other datacenters. That bandwidth is much cheaper than major ISP peering bandwidth because there aren't monopolies like Comcast extorting everybody for peering.

bedros 10y ago

out of curiosity, what storage do you use? any experience using backblaze b2 as hot storage, and glacier or google coldline as backup

kyledrake 10y ago
I'm planning to build a CephFS cluster at this point. I've given up on finding a cloud storage provider that will work well for us. This will require a fairly high fixed monthly cost to get space and transit at a datacenter, but after ~10-30TB BW you start to save a lot of money. There are trade offs. Aside from more upfront costs and a fixed monthly, Ceph is ridiculously complicated. Interface wise, they need some much better abstractions. B2 was a strong candidate, their BW rates are a little high still but approaching reasonable. The one issue is that they seem to have inconsistent latency. Not a problem for most use cases, but I need nearly all requests to come back in <100ms consistently, as I'm using this for web hosting. Their use case seems more focused on "hot standby backup" than on high availability ATM. I'm strongly considering them as the backup provider for my storage cluster. FWIW, GCS is not winning any best-of-show awards for latency either. S3 at the time I ran tests was doing a better job.
- bedros 10y ago
  I read a lot of good stuff on CephFS, but never tried it.
- SysArchitect 10y ago
  What makes Ceph complicated? The underlying idea and how Ceph works is fairly simple.
  - kyledrake 10y ago
    This is the "quick installation": http://docs.ceph.com/docs/master/start/ http://docs.ceph.com/docs/master/start/ It really shouldn't be this complex. I would love to just be able to boot an executable with a simple config file and be done with it. SeaweedFS shines a light on how this could be improved: https://github.com/chrislusf/seaweedfs https://github.com/chrislusf/seaweedfs
    - rakoo 10y ago
      ... which naturally brings the question: why choose Ceph and not SeaweedFS ? I've read a bit about the latter (especially the Facebook paper), and the design and operability seem simple enough that it might be a good starting point
      - kyledrake 10y ago
        Ceph is more production-ready, and has a lot of things SeaweedFS I believe doesn't yet have (such as consistency checking and rebuilding of failed nodes). It also has FS layer capability. SeaweedFS can use a filer but it currently doesn't support indexing (so you can't do subdirectory lookups for example, you would need to know the full filename). SeaweedFS is fundamentally trying to do a different thing than Ceph is, and it benefits certain use cases. Reading the Facebook Haystack paper will give you an idea of the differences. That said, I'm extremely impressed with how simple the interface is. It's very easy to get started.
- brianwski 10y ago
stemuk 10y ago
I am currently using the Hetzner SX131 in RAID 1 and http://minio.io http://minio.io. For about 200€ you get (2x) 30 TB and 50 TB traffic included. Thats about 0.007€ per GB with "free" traffic.
- bedros 10y ago
  is minio.io a service or server like cephfs?
  - stemuk 10y ago
    It is similar to cephfs, but I found it to be more performant and easier to set up.
  - Veratyr 10y ago
    Minio is basically a local implementation of the S3 API so you can access your stuff remotely and integrate with S3 compatible services using it.
  - bedros 10y ago
    reading the documentation, it looks like more of an NFS replacement with S3 API; I did not see any mention of clustering and redundancy

Veratyr 10y ago

You can get gigabit + a full cabinet of colocation from Hurricane Electric for $400/mo so you know: https://he.net/colocation.html https://he.net/colocation.html I'm sharing it with a few other people and it's great.

kyledrake 10y ago
I've looked at this. Easily the best deal in town. A few notes for people considering it: - It's IP transit and a DC from HE, which means it's possibly not BGP peering through other transit providers? If this is true, if HE's network fails, so does your server's internet connection. Typically you want to be multi-homed to ensure redundancy, unless you're doing something like an Anycast CDN and you don't care if it craps out for a few hours. - HE is the cheapest transit for reasons. It may not be a big deal for what you're doing, but be aware of the differences https://news.ycombinator.com/item?id=5348624 https://news.ycombinator.com/item?id=5348624 - They only provide a 15A circuit (likely only 80% usable) for a 42U rack, which is pretty ridiculous. You'll blow through that pretty quickly if you're using dual Xeon servers. It's cheaper under their pricing to get a second 42U rack than it is to get the proper amount of power needed for a full cabinet. - If you're doing Anycast, be aware that HE doesn't provide any BGP communities except blackholing, which could make it hard to tune your network. (If any of this sounds annoying to deal with and think about, you know exactly why I'd prefer that the cloud providers had better BW costs so I could not have to do this anymore. Again, if the business model works with GCS, congratulations, use it.)
- Veratyr 10y ago
  > It's IP transit and a DC from HE, which means it's possibly not BGP peering through other transit providers? The gigabit transit that comes with the cabinet is HE-only, yep. You can however buy transit and an interconnect from any other carrier in the datacenter (there are quite a few in FMT2 at least). > They only provide a 15A circuit for a 42U rack, which is pretty ridiculous. Yeah, it's pretty shitty. I ended up going with Xeon D for the power efficiency. Works pretty well.
chrisabrams 10y ago

boulos 10y ago

At our highest egress tier, our CDN offering around GCE (or GCS in Alpha), starts being competitive [1]. I agree though that if you just want some bandwidth, no major public cloud provider comes close to the kind of transit rates you'll find for "this isn't VOIP or gaming...". How much egress does Neocities do a month? Feel free to ping me if you'd prefer to discuss offline. Disclosure: I work for Google Cloud. [1] https://cloud.google.com/cdn/pricing#cache_egress_pricing_table https://cloud.google.com/cdn/pricing#cache_egress_pricing_ta...

kyledrake 10y ago
$0.02 for North America is a rate where I start to go "okay, that could make sense" for entry level. I would call it priced right if it was extremely high quality bandwidth, as in you made some good peering arrangements with the eyeball ISPs and had some strong low-saturation IP transit in the mix. You would have to convince me of it's quality. My traceroutes should show a lot of direct regional peering to the major ISPs and Level 3 caliber transit. I really think you could differentiate a from your competition hugely by just using that $0.02/GB point for all your customers, regardless of the amount of egress they're using. It's a big chicken-and-egg problem to have those bandwidth costs initially as a startup, but then be able to handle them as a bigger organization. It made us move to our own infrastructure, and once we're here, I already have my infrastructure in a datacenter at that point, so why bother with the cloud? Even better (for me anyways) would be to provide an option for the "this isn't VOIP or gaming" bandwidth. Not everybody needs that. Actually, probably most people don't. For what I would use GCS for I only need transit to other datacenters. That bandwidth is much cheaper than major ISP peering bandwidth because there aren't monopolies like Comcast extorting everybody for peering.