9 ms·

At that kind of scale, S3 makes zero sense. You should definitely be rolling your own. 10PB costs more than $210,000 per month at S3, or more than $12M after f

• warrenm 5y ago
  >RackMountPro offers a 4U server with 102 bays, similar to the BackBlaze servers, which fully configured with 12GB drives is around $11k total and stores 1.2 PB per server. (https://www.rackmountpro.com/product.php?pid=3154 https://www.rackmountpro.com/product.php?pid=3154) I dare you to buy 102 12TB drives for $11k The cheapest consumer class 12GB hdd is ~$275 a pop That's $28k just for the drives
• FireBeyond 5y ago
  Does Snowball let you exfiltrate data from AWS? I was under the impression it was only for bulk ingestion.
  • skynet-9000 5y ago
    First sentence on the linked page: "With AWS Snowball, you pay only for your use of the device and for data transfer out of AWS."
    • leetrout 5y ago
      Wow that’s up to $500,000 just to export 10PB (depending on region).
      • canucker2016 5y ago
        According to https://aws.amazon.com/snowball/pricing/ https://aws.amazon.com/snowball/pricing/, egress fees depends on the region, which can range from $0.03/GB (North America & parts of Europe) to $0.05/GB (parts of Asia and Africa). So US$300K to US$500K for egress fees + cost of Snowball devices. The major downside of Snowball in this export case is the size limit of 80TB per device - from https://aws.amazon.com/snowball/features/ https://aws.amazon.com/snowball/features/ : "Snowball Edge Storage Optimized provides 80 TB of HDD capacity for block volumes and Amazon S3-compatible object storage, and 1 TB of SATA SSD for block volumes." That'd be around 125 Snowball devices to get 10PB out. If OP actually has 10PB on S3 currently, the OP may want to fallback to leaving the existing data on S3 and accessing new data in the new location.
        • nicoburns 5y ago
          There's also the snowmobile https://aws.amazon.com/snowmobile/ https://aws.amazon.com/snowmobile/
          • canucker2016 5y ago
            The AWS Snowmobile pages only talk about migrating INTO AWS, not OUT OF. from https://aws.amazon.com/snowmobile/ https://aws.amazon.com/snowmobile/ : AWS Snowmobile is an Exabyte-scale data transfer service used to move extremely large amounts of data to AWS. You can transfer up to 100PB per Snowmobile, a 45-foot long ruggedized shipping container, pulled by a semi-trailer truck. Snowmobile makes it easy to move massive volumes of data to the cloud, including video libraries, image repositories, or even a complete data center migration. from https://aws.amazon.com/snowmobile/faqs/ https://aws.amazon.com/snowmobile/faqs/ : Q: What is AWS Snowmobile? AWS Snowmobile is the first exabyte-scale data migration service that allows you to move very large datasets from on-premises to AWS.
• darkr 5y ago
  I’ve heard reports that minio gets slow beyond the hundreds of millions of objects threshold
• user5994461 5y ago
  You realize you can't fit 10 appliances of 4U in a rack? (A rack is 42U) There's network equipment and power equipment that requires space in the rack. There's power limitations and weight limitations on the rack that prevents to fill it to the brim.
  • jedberg 5y ago
    I've put 39U of drives in a rack before. You only need 1U for a network switch, and you can get power that attaches vertically to the back, so it doesn't take up any space. If you have a cabinet with rack in front and back and all the servers have rails, the weight shouldn't be an issue. The biggest issue will be cooling depending on how hot your servers run. Specifically, it was a rack full of Xserve RAIDs, which are 3U each and about 100lbs each. So that was over 1300lbs.
    • dilyevsky 5y ago
      You run a single network switch for a rack full of drives to the brim?
      • jauer 5y ago
        Sure. A single rack is a common failure domain so you make sure to replicate across racks. E.g. Dunno about anyone else, but Facebook racks (generally) have a single switch.
        • dilyevsky 5y ago
          That seems rather unnecessary risk. Sure you stripe across the racks but another tor switch in mlag configuration is a minuscule expense compared to the costs involved here
      • jedberg 5y ago
        You could easily run two switches, there would be enough room. But normally yes, I'd run one switch per rack. Switch failure is pretty rare, and when it does happen it's pretty easy to switch it out for a spare.
• cricalix 5y ago
  The thing about fitting everything in one rack, potentially, is vibration. There have been several studies into drive performance degredation from vibration, and there's noticeable impact in some scenarios. The Open Compute "Knox" design as used by Facebook spins drives up when needed, and then back down, though whether that's for vibration impact, I don't know (their cold storage use [0]). 0: https://datacenterfrontier.com/inside-facebooks-blu-ray-cold-storage-data-center/ https://datacenterfrontier.com/inside-facebooks-blu-ray-cold... https://www.dtc.umn.edu/publications/reports/2005_08.pdf https://www.dtc.umn.edu/publications/reports/2005_08.pdf https://digitalcommons.mtu.edu/cgi/viewcontent.cgi?article=1491&context=etdr;performance https://digitalcommons.mtu.edu/cgi/viewcontent.cgi?article=1...
  • pokler 5y ago
    Here is Brendan Gregg showing how vibrations can affect disk latency: https://www.youtube.com/watch?v=tDacjrSCeq4 https://www.youtube.com/watch?v=tDacjrSCeq4
    • fakedang 5y ago
      I'm an absolute noob here, but is using SSD racks for storage a feasible option cost wise and for this issue in particular?
      • warrenm 5y ago
        Absolutely it's an option It's gonna cost more But it's also going to be nearly vibration-free (just the PSU fans), and stupidly-fast
      • chefkoch 5y ago
        No, ssds are still way to expensive If you don't need the performance.
• tinus_hn 5y ago
  You are mixing up your units, with 12GB drives and 15TB in a rack.
• cdavid 5y ago
  [disclaimer: while I have some small experience putting things in DC, including big GPU servers, I have never been anywhere near that scale, certainly not storage] 10k $ is for a server with no hard drive. W/ 12 Gb disks, and with enough RAM, we're talking closer to 40-50k$ per server. Let's say for simplicity you're going to need to buy 15 of those, and let's say you only need to replace 2 of them per year. You need 25 over five years, that's already ~ 750k $ over 5 years. And then you need to factor in the network equipment, the hosting in a colocation space, and if storage is your core value, you need to think about disaster recovery. You will need at least 2 people full time on this, in the US that means minimum 2x 150k$ of costs per year: over 5 years, that's 1.5m$. If you use software-defined storage, that's likely gonna cost you much more because of the skill demand. Altogether that's all gonna cost you much more than 500k$ over 5 years. I would say you would need at least 5x to 10x this.
  • cerved 5y ago
    yes, the TCO needs consideration, not just the metal
  • deleted 5y ago
    [deleted]
• Johnny555 5y ago
  10PB costs more than $210,000 per month at S3, or more than $12M after five years. Your pricing is off by a 2X - he said he's ok with infrequent access, 1 zone, which is $0.01/GB, or $100K/month. If he rarely needs to read most of the data, he can cut the price by 1/10th by using deep archive, $0.00099 per GB, so $10K/month, or around $600K over 5 years, not including retrieval costs.
  • Titan2189 5y ago
    Nope, can't use Deep Archive as he specified max retrieval time of 1000ms. But you're correct with S3-IA
    • Johnny555 5y ago
      For a 10X reduction in cost, things that are impossible often become possible.
    • webmaven 5y ago
      > Nope, can't use Deep Archive as he specified max retrieval time of 1000ms. If accesses can be anticipated, pre-loading data from cold storage to something warmer might make it viable.
• hpcjoe 5y ago
  After a certain size, AWS et al simply don't make sense, unless you have infinitely deep pockets. For storage that you pull from, AWS et al charge bandwidth costs. These costs are non-trivial for non-trivial IO. I worked up financial operational models for one of my previous employers, when were were looking at costs of remaining on S3 and using it, versus rolling it into our own DCs. The download costs, the DC space, staff, etc. was far less per year (and the download cost is a 1 time cost) than the cold storage costs. Up to about 1PB with infrequent use, AWS et al might be better. When you look at 10-100PB and beyond (we were at 500PB usable or so last I remembered) the costs are strongly biased towards in-house (or in-DC) vs cloud. That is, unless you have infinitely deep pockets.
  • hpcjoe 5y ago
    I should add to this comment, as it may give the impression that I'm anti cloud. I'm not. Quite pro-cloud for a number of things. The important point to understand in all of this is, is that there are cross-over points in the economics for which one becomes better than the other. Part of the economics is the speed of standing up new bits (opportunity cost of not having those new bits instantly available). This flexibility and velocity is where cloud generally wins on design, for small projects (well below 10PB). This said, if your use case appears to be rapidly blasting through these cross-over points, the economics usually dictates a hybrid strategy (best case) or a migration strategy (worst case). And while your use case may be rapidly approaching these limits (you need to determine where they are if you are growing/shrinking), there are things you can do to risk and cost reduce transitions ahead of this. Hybrid as a strategy can work well, as long as your hot tier is outside of the cloud. Hybrid makes sense also if you have to include the possibility of deplatforming from cloud providers (which, sadly, appears to be a real, and significant, risk to some business models and people). None of this analysis is trivial. You may not even need to do it, if you are below 1PB, and your cloud bills are reasonable. This is the approach that works best for many folks, though as you grow, it is as if you are a frog in ever increasing temperature water (with regard to costs). Figuring out the pain point where you need to make changes to get spending on a different (better) trajectory for your business is important then.
  • papageek 5y ago
• atomicity 5y ago
  If you have a PBs of data that you rarely access, it seems to make sense to compress it first. I've rarely seen any non-giants with PBs of data properly compressed. For example, small JSON files converted into larger, compressed parquet files will use 10-100x less space. I am not familiar with images but see no reason why encoding batches of similar images should make it hard to get similar or even better compression ratios Also, if you decide to move off later on, your transfer costs will also be cheaper if you can move it off in a compressed form first.
  • cerved 5y ago
    couple be wrong but I don't believe compression of batches of compressed images compresses well but it'd be very interested to here about techniques on this because I have a lot of space eaten up by timelapses myself
    • d110af5ccf 5y ago
      On the contrary, batches of images with a high degree of similarity compress _very_ well. You have to use an algorithm specifically designed for that task though. Video codecs are a real world example of such - consider that H. 265 is really compressing a stream of (potentially) completely independent frames under the hood. I'm not sure what the state of lossless algorithms might be for that though.
      • cerved 5y ago
        right but we're not talking about compressing a video stream but compressing individually compressed pictures, big difference
      • water8 5y ago
        A movie is a series of similar images and while it does allow temporal compression in a 3rd axis to the 2d raster, H265 is about as good as it gets at the moment but its also lossy which might not be tolerable.
        • 5y ago
• SergeAx 5y ago
  You didn't take personnel cost into account. You will need at least two system administrators to look after those racks (even if remote hands to change faulty drives are in the monthly opex). It quickly takes you to surplus of 200k/year with current prices (which will rise another 50% in 5 years). On the other hand, you may negotiate a very sizable discount from AWS for 10Pb storage for 5 years.