6 ms·
Crossing the 1 tb/s throughput threshold for object storage with NVMe drives
- lettergram 7y agoThe key is “drives”, writing simultaneously across all you can get high throughput.
- BubRoss 7y agoThis doesn't say what minio actually is. It also says the benchmark cost $1000 to run. Also since this is terabits it is about 120 gigabytes per second using 256 drives.
- estsauver 7y agoMinio is an s3 interface compatible storage api.
- vxNsr 7y agowhats the use case?
- atmosx 7y agoA self hosted alternative to s3, a scalable object store service.
- BubRoss 7y agoIsn't that just called a file system?
- bradknowles 7y agoS3 is block/object storage, not file or filesystem.
- BubRoss 7y agoI thought it was a scalable object store service
- vidar 7y agoAnd a S3-api compatible http api layer
- mtmail 7y agoI looked up the pricing: Each i3en.24xlarge (96 vCPU, 768GB RAM, 8 x 7500 NVMe SSD) costs $10.848 per hour. For the test they rented 32, so around 350 USD/hour or around 250.000 USD/month.
- dx034 7y agoAt this cost, is AWS really worth it? Shouldn't colocation be much cheaper after only a few months of use?
- mrb 7y agoFabric-wise, PCIe 4.0 offers 16 Gbit/s per lane, so a single PCIe x16 device would have 256 Gbit/s of bandwidth available (in each direction), and a single server could reach 1 Tbit/s with four NVMe devices on four PCIe slots. There is the overhead of 128b-132b encoding to be considered, but that's minimal (3%). There is also the overhead of PCIe packets to consider, but if the NVMe device supports large maximum payload sizes (1kB+) it should be minimal. Are there PCIe 4.0 x16 NVMe devices commercially available?
- deleted 7y ago[deleted]
- amarshall 7y agoIt’s worth noting that M.2 & U.2 both have 4 lanes, so anything more would need to be HHHL or FHHL. Anyway, the Liqid LQD4500 has a 4.0 x16 interface and claims 24 GB/s. I don’t wanna know how much it costs, though. https://www.liqid.com/products/composable-storage/element-lqd4500-pcie-aic-ssd https://www.liqid.com/products/composable-storage/element-lq... https://www.servethehome.com/liqid-element-lqd4500-32tb-pcie-gen4-x16-nvme-ssd-oss/ https://www.servethehome.com/liqid-element-lqd4500-32tb-pcie...
- Rafuino 7y agox16 NVMe SSDs aren't very common. The most recent one I've seen is the Micron X100, but that's a x16 PCIe 3.0 device and it's not commercially available yet. Most consumer and enterprise SSDs are x4, though perhaps that may change as better media and controllers are able to saturate PCIe capabilities. Then it's a tradeoff on power requirements though to be considered...
- jordanthoms 7y agoAsus motherboards (and probably others too) support a mode where you can split the x16 into 4 x4 connections and use https://www.asus.com/us/Motherboard-Accessories/HYPER-M-2-X16-CARD/ https://www.asus.com/us/Motherboard-Accessories/HYPER-M-2-X1... to mount them.
- Nextgrid 7y agoMy question is whether this includes overhead of actually persisting the data and guaranteeing a certain SLA in terms of retrieving the data? Don’t get me wrong, this is a nice achievement and good luck to them if their objective is to build a storage company (I or my company has no need to store large amounts of data at high speed so no bias there) but there’s a difference between ingesting data at high speed and actually guaranteeing it’s going to be there in a year’s time despite potential storage drive failures and stuff.
- algo_trader 7y agoAre there any architecture papers/blogs on re-designing software for very fast storage ? Once storage is as-fast as memory, things change. Please note: i am NOT referring to cpu-on-the-memory-chip architectures
- JulianMorrison 7y agoThere's a lot of programs of late, designed to run in pure RAM. I imagine it would be similar? Except perhaps with different retention-over-restart characteristics.
- the8472 7y agoP2P-DMA is making inroads to the linux kernel. Which means software can limit itself to orchestrating transfers between storage, GPUs, accelerators and network cards and avoid having the data touch system RAM or the CPUs.
- natmaka 7y agoConfining the CPU to the most noble processing, and flanking him with highly-specialized units directly exchanging... isn't it the mainframe all over again?
- jamessb 7y agoFor databases, there was a 2016 VLDB paper that suggested using 'write-behind logging' as an alternative to traditional 'write-ahead logging' that is "designed for a hybrid storage hierarchy with NVM and DRAM": http://www.vldb.org/pvldb/vol10/p337-arulraj.pdf http://www.vldb.org/pvldb/vol10/p337-arulraj.pdf
- amarshall 7y ago> Once storage is as-fast as memory Bandwidth is only one aspect, SSD latency (even Optane) is still quite slow compared to RAM.
- mjb 7y agoLots of work, in both academia and industry. For example, check out Daniel Bittman's work at UCSC, and Tim Kraska's at MIT. Two interesting things are happening here: memory becoming non volatile, and storage and network bandwidth catching up with memory bandwidth.
- SomeHacker44 7y agoCan someone please fix the title? I presume it means TB/s or Tb/s but I am not able to load the article now. Then delete my comment.
- LawnboyMax 7y agoIt should be Tb/s