5 ms·
To be fair to Backblaze this level of storage density is really only possible with recent advances in disk technology (higher densities, SMR storage, etc). Als
by james_cowling 11y ago
To be fair to Backblaze this level of storage density is really only possible with recent advances in disk technology (higher densities, SMR storage, etc).
Also not everyone wants to be packing a petabyte into a box. At that level of density you need to invest a lot of effort in replication strategies, tooling, network fabric etc to handle failures with high levels of availability/durability.
- jamwt 11y agoYes, a 1PB+ failure domain only made sense because Magic Pocket is very good at automatic cluster management and repair.
- samstave 11y agoAre you using spindle or ssd or flash? (admittedly I dont know if you consider flash and ssd to be the same) What is your price per GB raw?
- jamwt 11y ago> Are you using spindle or ssd or flash? (admittedly I dont know if you consider flash and ssd to be the same) We have flash caches (NVMe) in the machines, but the long-term storage devices are spindle--PMR and SMR. > What is your price per GB raw? I cannot disclose, exactly, but it is well below any other public price I've seen.
- voltagex_ 11y agoSo NVMe is "ready for production"? Do you think you'd use it at home, or only in the enterprise?
- jamwt 11y agoI mean, if at home you need 400k IOPS, 1GB/s of writes, and 2.2GB/s of reads... go for it! I sure don't, but more power to you. :-)
- voltagex_ 11y agoI'm tempted to get one and set it up as some kind of cache in my NAS. I'm already in silly territory with it now, anyway. 1GB/s of writes sounds crazy though - I haven't got anything that can write to it that fast!
- stock_toaster 11y agoI have a PCIe NVMe card (with a Samsung 950 pro M.2 in it) in my home server, which serves as an L2arc device for a ZFS pool. It is pretty nice. Runs a bit warm though.
- voltagex_ 11y agoWe are now well offtopic but I've seen some advice recently that advises using an SLOG rather than L2ARC for a home NAS.
- chronid 11y agoI think that a home nas would have a lot of async writes and very few/none sync writes. So a ZIL dedicated device (the SLOG) is not really useful/helpful. I'm not sure even if you really need an L2ARC device, just slap 8/16Gb of RAM on it and you will be happy... Obviously if you are playing seriously with VMs and databases and whatever else both of those (SLOG/L2ARC) may become important for you, I'm going for the "i'm just using this to store my raw-format photos, backups for the taxes and other big files" kind of usage here. :)
- voltagex_ 11y agoYou don't have any contact details but I'd definitely like to talk to you more about home NAS.
- scurvy 11y agoNVMe's lower latency is great. I've never really known how "slow" regular SATA/SAS SSD's were until I used NVMe ones.
- gbrayut 11y agoWe have been primarily using Intel NVMe storage for our database servers since fall of 2014 with no major complaints. Our high end desktop/laptop systems are also using the latest Samsung NVMe M.2 drives which are screaming fast.
- rwg 11y agoJust to put a number to "screaming fast", 2.5 GB/s sequential reads averaged across the entire drive on a Samsung 950 Pro NVMe SSD: https://pbs.twimg.com/media/CaMhc2oVAAE4q3q.jpg:large https://pbs.twimg.com/media/CaMhc2oVAAE4q3q.jpg:large
- toyg 11y agoI put it in production, and it's something else. Whether I'd use it at home depends entirely on how rich I feel I am ;)
- zbjornson 11y agoAside, Google Cloud has NVMe available on almost all machine types if you want to play with it [1]. [1] https://cloud.google.com/compute/docs/disks/local-ssd https://cloud.google.com/compute/docs/disks/local-ssd
- jorangreef 11y agoSurely it's the other way round? That is, the better your software can handle repair, the smaller your failure domain can be? A larger failure domain would only make sense if you wanted to minimize the compute required per unit stored?
- jamwt 11y agoWe do want to minimize that--compute costs money, and storage is irreducible. But you can only reduce aggressively if your larger distributed system is great at repairs, since the failure of a single box kicks off 1PB of repair activity!
- jorangreef 11y agoThanks, do you have a time lapse backup of data in Magic Pocket? For example, in case of software error?
- ju-st 11y agoIn datacenters space is cheap in comparison to power so often it's simply not economical to cram more in less space.
- jamwt 11y agoSometimes; but depreciation costs are usually more than either of those (for storage).
- yardie 11y agoDid you build your own centers? Running our hosting we had far more constraints on our power than anything else. Even though we only had 20U of equipment we ended up taking a 44u rack. Our disk arrays being the largest consumers. This made the case for moving to SSDs even stronger.
- dexterdog 11y agoThat was the problem I always had in my pre-cloud days. We were usually running racks half-full because of power restrictions in the DC. It was convenient because you could keep you cold spares and tools in your expensive space, but it always seemed like you should be putting more useful stuff in the racks.
- jamwt 11y agoYou can substantially change the economics if you design your own power distribution systems.
- scurvy 11y agoAye, but even off the shelf 3ph 50A 208v circuits gets you a lot of power. That's enough for 7 or 8 4U storage nodes with 70 drives each. Then again, I don't really like putting too much weight that high up in a rack that already has 2600lbs of gear in it.
- ChuckMcM 11y agoNot sure what is more amazing, the project of this scale (love the disk drawers!) or that infrastructure for managing the fleet of drives gets top billing!
- cm2187 11y agoIs dropbox using SMR? (I never used dropbox. Do they offer incremental backup? Otherwise I would have thought they would need to be able to do random writes, unless SMR is so cheap that it is economical to keep some dead data when a user uploads a new version of a file)
- cm3 11y agoThere was a post a while back here from a Dropbox employee explaining how they manage SMR disks manually via a direct HBM card and basically doing what the firmware would do otherwise. I can't find the post but it was talking about working around the architectural deficiencies of SMR while still getting to use more disk space.
- jamwt 11y agoMagic Pocket is a block storage system, and all storage in it is append only. We'll go into more detail on the on-disk format in a blog post. But, yes, we use the SMR disks on an HBA, and we directly address the disk on a LBA (and zone) basis.