6 ms·
> NF1 SSDs enabled an undisclosed maker of servers to install 72 of such drives in a 2U rack for a 576 TB capacity That's 12PB in a 42U rack! I wonder what so
by iooi 8y ago
> NF1 SSDs enabled an undisclosed maker of servers to install 72 of such drives in a 2U rack for a 576 TB capacity
That's 12PB in a 42U rack! I wonder what sort of storage densities Google, Amazon, and Dropbox are achieving.
- _wmd 8y agoNote that 2U box needs something like 250gbit internal bus just for disk IO, assuming you wanted the full throughput of every drive.. there is a huge trade off in terms of processing capability and networking bandwidth putting so much disk in a single box, the rest of the hardware can't cope with it. These things look useful only if a tiny fraction of the stored data is hot, I imagine lower density that provides more flexibility in terms of processing room on the box, and networking around the box is probably a better option in a lot of cases. Been pondering these devices for the past 10 minutes or so, can't think of a use for them except elaborate storage servers.
- dogma1138 8y agoVideo editing? 6/8K raw video takes a lot of space.
- colejohnson66 8y agoLinusTechTips films in 8k IIRC and they only have one petabyte
- SmellyGeekBoy 8y agoOff topic but does anyone else find that guy incredibly annoying? I also get the impression he doesn't have a clue what he's talking about most of the time.
- dzhiurgis 8y agoYes indeed! He seems to have good clue about gaming hardware, which is the field he is targetting.
- crististm 8y agoMy impression is this is part of his persona and intentional. I don't find him annoying but I can see why people might. On the other hand, he has a bunch of people researching the topics so that at least he knows what he's talking about.
- AnIdiotOnTheNet 8y agoPar for the course with Youtube celebrities then.
- Dylan16807 8y agoAt that resolution you're in the gigabyte-per-second range, and when you're grabbing a gigabyte at a time there's almost no reason to prefer SSD over HDD.
- murukesh_s 8y agoWhy would that be? Can you provide more details? I thought SSDs are always faster than HDD
- ddorian43 8y agoHDD are fast ~enough on sequential read/write in this scenario.
- akvadrako 8y agoNot at all - SSDs are much faster. One of the fastest HDDs on storagereview.com is the WD Black 6TB, with a sustained 218MB/s. While one of the faster SSDs is an Intel 750 at 2400MB/s.
- ddorian43 8y agoIf you have some HDD, and you RAID them, processing video should be mostly sequential read/write, so it should work on HDD. I know SSD is faster, but it depends on what the workload requires.
- Dylan16807 8y agoSo let's say you're working on ten hours of 8k footage. If you go with 1.2TB 750s... oh you can't actually fit 20-100 of those in a single workstation.[1] So maybe we back off to a NAS attached over a 40Gbps cable. 40 of those hard drives will be a lot cheaper than 20 of those SSDs. If we grossly overestimate seek times, despite how big the frames are, we might estimate 125MB/s per drive. That's enough to completely saturate the connection. Oh, and the array has 10 times the capacity. [1] 20 assumes a lower framerate and generous compression. 100 assumes 60fps 12-bit at a lossless 3:1 compression. If "raw" is taken completely literally then 10 hours of footage could require 300 of them (and some of the math would have to be redone).
- Nexxxeh 8y agoCompared to HDD storage, isn't all the data hot, relatively speaking? I mean it's pushing the bottleneck to the bus, sure, but the alternative of HDD storage is slower by every meaningful metric. (Except maybe heat? I would assume these run at a higher temp than equivalent storage space in HDD, but I've nothing to actually base that on.) If money's no object, presumably it's not "there's no use case for them", it's "there's no reason to do anything else"?
- codefined 8y agoI think they're actually talking about other items being the bottleneck as opposed to the rate at which you can grab information off of disks, so if you have any meaningful fraction of your data being hot at any one point there is no advantage between either. Even with HDDs, if you actually want to use all your throughput you're likely going to have problems with the bus (rarely needs to be optimised in 99% of cases) vs the drives (needs to be optimised in all environments).
- pas 8y agoThis 16TB thing has Active read/write up to 8.7/10.6 W, Idle 4.0 W power consumption properties, and HDDs range from 4 to 10 W ( https://www.tomshardware.com/reviews/desktop-hdd.15-st4000dm000-4tb,3494-6.html https://www.tomshardware.com/reviews/desktop-hdd.15-st4000dm... from 2013 ), so pretty much the same. And you don't have to worry about vibrations with SSDs, so you can blow a lot more air at them.
- deleted 8y ago[deleted]
- paranoidrobot 8y agoI think AMD Epyc is a perfect match for this... 128 PCIe lanes per socket. On a dual socket Epyc system with 36 of these, each with a 4x PCIe connection 'only' takes up 144 lanes, leaving plenty of lanes free for even 8x Dual-QSFP+ 40GbE NICs. Of course, I'm not sure how you'd physically pack all of that into even a 2U form factor. Usefulness... data warehouses. If the price was right, we'd probably move to that to speed up various ingestion & reporting functionality. At 288TB (assuming RAID10) capacity, it'd be a perfect fit.
- Dylan16807 8y agoEpyc is 128 lanes period. The dual-socket configuration steals half the lanes for the interconnect. It's also PCIe 3 for now.
- paranoidrobot 8y agoAh, right. My mistake. Still, I think it might be in a better position than Xeon for raw storage IO.
- scurvy 8y agoixnay on the 40gig-ay 25/50/100 is cheaper/better on modern gear.
- paranoidrobot 8y agoFair enough, I wasn't aware of 100G adapters.
- shaklee3 8y agoUnfortunately it's not pci 4, though.
- paranoidrobot 8y agoIt looks like these are PCI3 SSDs, and with a bit of poking around I can't see any PCI4 Xeons out yet either.
- anoother 8y agoAh, but you're assuming the 2U device is a server. More likely JBOF that can be connected to multiple servers.
- akvadrako 8y agoStill needs 250 gbit/s total bandwidth, which doesn't come cheap. In this case it really makes much more sense to distribute the processors along side the storage and aim for data locality.
- _wmd 8y agoA modern Xeon is 20 lanes of PCIv3, or 273MB/sec per drive just to get data into RAM, assuming the bus was exclusively dedicated to disk, which means at full utilization you get about 69k 4k IOPs per unit rather than the rated 500k. Then you need to shunt that back out over the network - using the same bus it just arrived on, most likely after a trip via the CPU to add protocol headers. With a 40 Gbit NIC the per-drive bandwidth drops to absolute max 204MB/sec and available network bandwidth per-drive is only 86MB/sec (22k IOPs), meaning at full whack and in the best possible scenario, and assuming no other overheads, you'd still only ever see 4-5% of rated performance. In reality and after accounting for things like packetization (and the impedance mismatch of 4k reads vs. 1500 byte network MTU) and the work done by the CPU it's probably safe to knock that number down by another 20-40%, depending on the average IO size. Sure they'd kick ass for reads from a small number of clients and latency would be amazing, but it'd be so much simpler, cheaper, and flexible to just have the same disk split across 4 or more boxes. What happens if the motherboard dies? That's a megaton of disk suddenly offline, etc.
- handle0174 8y agoDropbox recently posted about their new 4U / 1.4PB chassis. https://blogs.dropbox.com/tech/2018/06/extending-magic-pocket-innovation-with-the-first-petabyte-scale-smr-drive-deployment/ https://blogs.dropbox.com/tech/2018/06/extending-magic-pocke...
- rcoveson 8y ago> an undisclosed maker of servers So Supermicro