4 ms·

I would host in a datacenter of your choice and do a cross connect into AWS: https://aws.amazon.com/directconnect/pricing/ https://aws.amazon.com/directconnect/

I would host in a datacenter of your choice and do a cross connect into AWS: https://aws.amazon.com/directconnect/pricing/ https://aws.amazon.com/directconnect/pricing/ This allows you to read the data into AWS instances at no cost and process it as needed since there is 0 cost for ingress into AWS. I have some experience with this (hosting using Equinix)

philippb 5y ago
Thanks for the pointer. Never thought about this as an option. Great stuff!!!
- pickle-wizard 5y ago
  I had a similar problem at a past job. Though we only had a PB of data. We used a products called SwiftStack. It is open source, but they have paid support. I recommend getting support, as their support is really good. It is an object store like S3, but it has its own API. Though I think they now have an S3 compatible gateway now. We had about 25 Dell R730xd servers. When the cluster would start to fill up, we would just replace drives with larger drives. Upgrading drives with SwiftStack is a piece of cake. When I left we were upgrading to 10TB drives as that was the best pricing. We didn't buy the drives from Dell as they were crazy expensive. We just bought drives from Amazon/New Egg, and kept some spares onsite. We got a better warranty that way too. Dell only had a 1 year warranty, but the drives we were buying had a 5 year warranty.
  - neverartful 5y ago
    Way late to the discussion, but I second the positive remarks on SwiftStack. It's in the easy button category in this case. The core storage engine of SwiftStack is open source (OpenStack Swift). However, the nice wrap-around tooling and web dashboard is not open source.
- TechBro8615 5y ago
  I’m not an AWS pricing expert, but you should be aware you’re still on the hook for S3 requests even if you can get out of paying for bandwidth. Is AWS direct connect a pure peering arrangement? I wonder what their requirements are for that. Guess I’ll read the link :) Idk what your team’s expertise is, but I’d advise avoiding the cloud as long as possible. If you can build out an on-premise infrastructure, it will be a huge competitive advantage for your company because it will allow you to offer features that your competitors can’t. Examples of this: - Cloudflare built up their own network and infrastructure and it’s always been their biggest asset. They set the standard for free tier of CDN pricing, and nobody who builds a CDN on top of an existing cloud provider will ever beat it. - Zoom. By hosting their own servers and network, Zoom is similarly able to offer a free tier where they are not subject to variable costs from free customers losing them money on bandwidth charges. - WhatsApp. They scaled to hundreds of millions of users with less than a dozen engineers, a few dozen (?) servers, and some Erlang code. IMO defaulting to the cloud is one of the worst mistakes a young company can make. If your app is not business critical, you can probably afford up to a day of downtime or even some data loss. And that is unlikely to happen anyway, as long as you’ve got a capable team looking after it who chooses standard and robust software.
  - throwaway823882 5y ago
    I run cloud infra for a living. Have been managing infrastructure for 20 years. I would never for one second consider building my own hosting for a start-up. It would be like a grocery delivery company starting their own farm because seeds are cheap.
    - TechBro8615 5y ago
      Depends what you’re doing I suppose. I think the three companies I mentioned (CloudFlare, Zoom and WhatsApp) are good examples of infrastructure investment as a competitive advantage.
      - derefr 5y ago
        None of those are start-ups, though. They've either IPOed (CloudFlare, Zoom) or been acquired by publicly-traded companies (WhatsApp). A startup is a company that might still need to pivot to find its final business model, potentially shedding its entire existing infrastructure base in the process. Start-ups are why IaaS providers don't default to instance reservations — because, as a startup, you might suddenly realize that you won't be needing that $10k/hr of compute, but rather $10k/hr of something else.
- dsyrk 5y ago
  I’d like to add I’d agree with the parent comment and add some specifics. Buy storage servers from 45drives they basically build same hardware as Backblaze uses. Add copper 10G nics to the servers. https://www.45drives.com/ https://www.45drives.com/ Get necessary switches 10G with 40G uplink ports. Whatever your favorite. Use 10GBaseT to the servers. Install hardware in a quality data center. Like one of theirs - https://www.digitalrealty.com/ https://www.digitalrealty.com/ And get 10G virtual cross connects to AWS. Back of the envelope calculation you need 30TB raw, so about 60 servers. They aren’t really that power hungry so 10 per cabinet. 6 cabinets. at least 6+2 switches. Software wise you have lots of options with this infra. High upfront cost but low MRC vs all other options. Assuming you have skilled sys admins who know what they are doing.
- comboy 5y ago
  + some deep archive glacier? I think waiting 12h for data is acceptable if your datacenter burns down but it may not be the case for you.
aynsof 5y ago
Direct Connect isn't required from a cost perspective - ingress into AWS is free in all cases I can think of, but certainly in the case of S3 [0]. DX is useful when customers need assurances of bandwidth/throughput, or if they want to avoid their traffic routing over the internet. [0] "You pay for all bandwidth into and out of Amazon S3, except for the following: Data transferred in from the internet..." - https://aws.amazon.com/s3/pricing/ https://aws.amazon.com/s3/pricing/

philippb 5y ago

Thanks for the pointer. Never thought about this as an option. Great stuff!!!

pickle-wizard 5y ago
I had a similar problem at a past job. Though we only had a PB of data. We used a products called SwiftStack. It is open source, but they have paid support. I recommend getting support, as their support is really good. It is an object store like S3, but it has its own API. Though I think they now have an S3 compatible gateway now. We had about 25 Dell R730xd servers. When the cluster would start to fill up, we would just replace drives with larger drives. Upgrading drives with SwiftStack is a piece of cake. When I left we were upgrading to 10TB drives as that was the best pricing. We didn't buy the drives from Dell as they were crazy expensive. We just bought drives from Amazon/New Egg, and kept some spares onsite. We got a better warranty that way too. Dell only had a 1 year warranty, but the drives we were buying had a 5 year warranty.
- neverartful 5y ago
  Way late to the discussion, but I second the positive remarks on SwiftStack. It's in the easy button category in this case. The core storage engine of SwiftStack is open source (OpenStack Swift). However, the nice wrap-around tooling and web dashboard is not open source.
TechBro8615 5y ago
I’m not an AWS pricing expert, but you should be aware you’re still on the hook for S3 requests even if you can get out of paying for bandwidth. Is AWS direct connect a pure peering arrangement? I wonder what their requirements are for that. Guess I’ll read the link :) Idk what your team’s expertise is, but I’d advise avoiding the cloud as long as possible. If you can build out an on-premise infrastructure, it will be a huge competitive advantage for your company because it will allow you to offer features that your competitors can’t. Examples of this: - Cloudflare built up their own network and infrastructure and it’s always been their biggest asset. They set the standard for free tier of CDN pricing, and nobody who builds a CDN on top of an existing cloud provider will ever beat it. - Zoom. By hosting their own servers and network, Zoom is similarly able to offer a free tier where they are not subject to variable costs from free customers losing them money on bandwidth charges. - WhatsApp. They scaled to hundreds of millions of users with less than a dozen engineers, a few dozen (?) servers, and some Erlang code. IMO defaulting to the cloud is one of the worst mistakes a young company can make. If your app is not business critical, you can probably afford up to a day of downtime or even some data loss. And that is unlikely to happen anyway, as long as you’ve got a capable team looking after it who chooses standard and robust software.
- throwaway823882 5y ago
  I run cloud infra for a living. Have been managing infrastructure for 20 years. I would never for one second consider building my own hosting for a start-up. It would be like a grocery delivery company starting their own farm because seeds are cheap.
  - TechBro8615 5y ago
    Depends what you’re doing I suppose. I think the three companies I mentioned (CloudFlare, Zoom and WhatsApp) are good examples of infrastructure investment as a competitive advantage.
    - derefr 5y ago
      None of those are start-ups, though. They've either IPOed (CloudFlare, Zoom) or been acquired by publicly-traded companies (WhatsApp). A startup is a company that might still need to pivot to find its final business model, potentially shedding its entire existing infrastructure base in the process. Start-ups are why IaaS providers don't default to instance reservations — because, as a startup, you might suddenly realize that you won't be needing that $10k/hr of compute, but rather $10k/hr of something else.
dsyrk 5y ago
I’d like to add I’d agree with the parent comment and add some specifics. Buy storage servers from 45drives they basically build same hardware as Backblaze uses. Add copper 10G nics to the servers. https://www.45drives.com/ https://www.45drives.com/ Get necessary switches 10G with 40G uplink ports. Whatever your favorite. Use 10GBaseT to the servers. Install hardware in a quality data center. Like one of theirs - https://www.digitalrealty.com/ https://www.digitalrealty.com/ And get 10G virtual cross connects to AWS. Back of the envelope calculation you need 30TB raw, so about 60 servers. They aren’t really that power hungry so 10 per cabinet. 6 cabinets. at least 6+2 switches. Software wise you have lots of options with this infra. High upfront cost but low MRC vs all other options. Assuming you have skilled sys admins who know what they are doing.
comboy 5y ago
+ some deep archive glacier? I think waiting 12h for data is acceptable if your datacenter burns down but it may not be the case for you.

aynsof 5y ago

Direct Connect isn't required from a cost perspective - ingress into AWS is free in all cases I can think of, but certainly in the case of S3 [0]. DX is useful when customers need assurances of bandwidth/throughput, or if they want to avoid their traffic routing over the internet. [0] "You pay for all bandwidth into and out of Amazon S3, except for the following: Data transferred in from the internet..." - https://aws.amazon.com/s3/pricing/ https://aws.amazon.com/s3/pricing/