79 ms·
NVMe is not a hard disk
- kardos 5y agoNot following the leaf-and-spine figure. 40x10G is 400G, but 4x40G is 160G, so how is this completely oversubscription free? Edit: followed the link, it is 2.5:1 oversubscribed
- walrus01 5y agoit's worth noting that 40GbE is an absolute dead end in terms of technology, in the ISP world. Look at how many 40GbE members there are on many major IXes and other things. It's super cheap to buy an older Arista switch on ebay with N x 10GbE ports and a few 40GbE uplinks, because they're all getting pulled from service and sold off cheap. The upgrade path for 10GbE is 100GbE. major router and switch manufacturers aren't even making new line cards with multiple 40GbE ports on them anymore, they're tech from 6-7 years ago. You can buy either something with dense 10GbE SFP+ ports, or something with QSFP/QSFP28 or similar 100GbE ports. If you want something for a test/development lab, or are on a very tight budget, sure.
- bluedino 5y agoMeanwhile some budget homelabbers are jumping on the 2.5Gb bandwagon
- isotopp 5y agoThe image is from 2012 and describes a topology, but not a technology. Today you'd rather use Arista 7060, 7050cx orJuniper 5200 as ToR. You'd not build 1:1, but plan for it in terms of ports and cable space, then add capacity as needed. Almost nobody in an Enterprise environment actually needs 1:1, unlike hosters or hyperscalers renting nodes to others. Even then you'd probably being able to get away with a certain amount of oversubscription.
- dcminter 5y ago"Most people need their data a lot less than they think they do" - great way to put it and a thought provoking article.
- santoshalper 5y agoEspecially in a world where so much data now "lives" in the cloud. Between my dropbox, github, google photos, etc. Very little of my data only lives on a hard drive. The stuff that does lives on a Synology NAS and is mirrored to S3 Glacier weekly.
- sreeramb93 5y agoNAS mirrored to S3 Glacier. How much does it cost?
- xyzzy_plugh 5y agoIn my experience this is very cheap. I take it the parent is not retrieving from Glacier often/ever, which is where the significant costs go. It's a decent balance for disaster recovery. I sync my photos to S3 (a mix of raw and jpeg, sidecar rawtherapee files) across a few devices so Glacier is prohibitively expensive in this regard, but I still pay <$100 a year for more stuff than I could ever store locally.
- kstrauser 5y agoI did the math, and for me Glacier is great for backups where homeowners insurance is likely to be involved in the restoral. It was ferociously expensive for anything less drastic.
- laurentb 5y agohow would that process look like in practice should you need to get to call the insurance guys? as in, would you claim on the cost to retrieve the data or ? (this question is general, regardless of the actual country)
- innot 5y ago> RAID or storage replication in distributed storage <..> is not only useless, but actively undesirable I guess I'm different from most people, good news! When building my new "home server" half a year ago I made a raid-1 (based on ZFS) with 4 NVMEs. I rarely appear at that city, so I brought the fifth one and put it into an empty slot. Well, one of the 4 nvmes lasted for 3 months and stopped responding. One "zpool replace" and I'm back to normal, without any downtime, disassembly, even reboots. I think that's quite useful. When I'm there the next time I'll replace the dead one, of course.
- zxexz 5y agoWhat setup do you use to put 4 NVME in one box? I know it’s possible, I’ve just heard off so many different setups. I know there are some PCIE cards that allow for 4 NVME drives. But you have to match that with a motherboard/CPU combo with enough ones to not lose bandwidth.
- birdyrooster 5y agoThat’s exactly what they are doing. Anyone else is using proprietary controllers and ports for a server chassis
- pdpi 5y agoI've been looking into building some small/cheap storage, and this is one of the enclosures I've been looking at. https://www.owcdigital.com/products/express-4m2 https://www.owcdigital.com/products/express-4m2
- isotopp 5y agoFor distributed storage, we use this: https://www.slideshare.net/Storage-Forum/operation-unthinkable-software-defined-storage-bookingcom-peter-buschman#14 https://www.slideshare.net/Storage-Forum/operation-unthinkab... We then install SDS software, Cloudian for S3, Quobyte for File, and we used to use Datera for iSCSI. Lightbits maybe in the future, I don't know. These boxen get purchased with 4 NVME devices, but can grow to 24 NVME devices. Currently 11 TB Microns, going for 16 or more in the future. For local storage, multiple NVME hardly ever make sense.
- villgax 5y agoWish it was as swappable as RAM sticks, quite the pain to get in with a screwdriver after removing the GPU during OS installs on 2 drives without bootloader crap
- snuxoll 5y agoNVMe is just a protocol for accessing flash memory over PCIe express, do not confuse it with the M.2 form factor (which also supports SATA and USB!). My Optane P900 I use as a ZFS log device is NVMe and plugs into a standard PCIe slot on my PowerEdge R520, and servers frequently use U.2 form factor drives.
- anarazel 5y agoNot necessarily flash - the p900 you list isn't flash for one. Nor is it necessarily over PCIe...
- walrus01 5y agoI think where people commonly confuse it is because most often when they see a M.2 device, it is the variant of the slot that exposes PCI-E pins to the NVME SSD, such as for a $100 M.2 2280 SSD from Samsung, Intel, WD, etc. As you mention there's lots of other things which can be electrically connected to a motherboard's I/O buses in a M.2 slot.
- villgax 5y agoYeah agreed, but I'm clearly not referring to the protocol.
- anarazel 5y agoE.g. U.2 drives can utilize NVMe but in a form factor easier to swap (including hot plugging support). Edit: typo
- Godel_unicode 5y agoIcy dock makes an entire range of hot-swap docks for m.2 NVMe drives.
- notaplumber 5y ago> Because flash does not overwrite anything, ever. This is repeated multiple times in the article, and I refuse to believe it is true. If NVME/SSDs never overwrote anything, they would quickly run out of available blocks, especially on OSs that don't support TRIM.
- cduzz 5y agoThere's nuance to this; the deletes / overwrites are accomplished by bulk wiping entire blocks. Rather than change the paint color in a hallway you have to tear down the house and build a new house in the vacant lot next door that's a duplicate of the original, but with the new hallway paint. To optimize, you keep a bucket of houses to destroy, and a bucket of vacant lots, and whenever a neighborhood has lots of "to be flattened houses" the remaining active houses are copied to a vacant lot and the whole neighborhood is flattened. So, things get deleted, but not in the way people are used to if they imagine a piece of paper and a pencil and eraser.
- zdw 5y agoinspired by that last sentence, the analogy could be rewritten as: - lines on page - pages of paper - whole notebooks and might be easier for people to grok than the earlier houses/paint analogy.
- jazzyjackson 5y agoI don’t know, I like the drama of copying a neighborhood and tearing down the old one xD
- KaiserPro 5y agoNVME over X seems like fun. After a decade of shying away from fibre channel, we've finally got to the point where we've forgotten how awful iscsi was to look after. (and how expensive FC used to be, and how disappointing SAS switches were.)
- jabl 5y agoDon't worry. By the time the worst bugs are ironed out, and we have management tools worth crap, there will be a new shiny. That being said, nvme seems nice with no major footguns, so I guess NVME over X is in principle at least a good idea.
- isotopp 5y ago"I don't know what features the network of the future will have, but it will be called Ethernet." Ethernet is absorbing all the features of other networking technology, and adding them to it's own perfection.
- tyingq 5y ago"most database-like applications do their redundancy themselves, at the application level, so that...storage replication in distributed storage...is not only useless, but actively undesirable" I do believe that's been the author's experience. However, I think he may be unaware that's not everyone's...or even most people's, experience.
- walrus01 5y agoMy biggest take away from this is "I certainly hope this person isn't responsible for actually designing the bare metal storage infrastructure meant to underly something important". They seem to be operating from the premise that data replication at the 'cloud' level can solve all their problems.
- 411111111111111 5y agoI think you're giving him too little credit there. The parents point is really spot on though: most websites aren't at scale where every stateful api has redundant implementations. But the author's point does have merit: inevitably, something goes wrong with all systems - and when it does, your system goes down if you trust your HA config to work. If you actually did go for redundant implementations, your users likely aren't even gonna notice anything went wrong. It's however kinda unsustainable to maintain unless you have a massive development budget
- alisonkisk 5y agoBut why is his layer the only correct layer for redundancy? He's also doing wasted work his redundancy has bugs that a consumer has to worry about working around.
- deleted 5y ago[deleted]
- 411111111111111 5y agoWe're currently quite off-topic if i'm honest. I think the author was specifically talking about RAID-1 redundancy and is advocating that you can leave your systems with RAID-0 (so no redundant drives in each server), as you're gonna need multiple nodes in your cluster anyway... so if any of your systems disks break, you can just let it go down and replace the disk while the node is offline. but despite being offtopic: redundant implementations are -from my experience - not used in a failover way. they're active at all times and load is spread if you can do that, so you'd likely find the inconsistencies in the integration test layer.
- DeliriumTrigger 5y agoThis blog post is the perfect example for "I think I have a deep understanding but I really don't."
- walrus01 5y ago> "Customers of the data track have stateless applications, because they have outsourced all their state management to the various products and services of the data track." I have no idea what this is even supposed to mean. It's like somebody combined some buzzwords thought up by a fresh business school marketing graduate working in the 'cloud' industry with an attempt at actual x86-64 hardware systems engineering. The whole premise of the first half of the article seems to be 'you don't need to design a lot of redundancy and fault tolerance', the second part then goes into a weird explanation of NVME targets on CentOS. I hope this person isn't actually responsible for building storage systems at the bare metal level supporting some production business application.
- wmf 5y agoIt's basic 12-factor aka cloud native thinking: "Any data that needs to persist must be stored in a stateful backing service, typically a database."
- cowmoo728 5y agoI think the article is saying that a web server (customers) should be stateless, because everything important should be in a database (data track) on another host. And that database probably has application level handling for duplicating writes to another disk or another host. The conclusion seems to be that it's not important for hardware level data redundancy because existing database software already handles duplication in application code. I don't understand how that conclusion was reached. Hardware level redundancy like raid1 seems useful because it simplifies handling a common failure case when a single HDD or NVME fails on a database server. Hardware redundancy is just the first stage in a series of steps to handle drive failure. I do agree that a typical stateless server doesn't need raid1, but afaik it's not standard practice for a stateless web application to bother with raid1 anyway.
- bob1029 5y agoMy take on NVMe has evolved a lot after getting a chance to really play around with the capabilities of consumer-level devices. The biggest thing I have realized (as have others), is that traditional IO wait strategies don't make sense for NVMe. Even "newer" strategies like async/await do not give you want you truly want for one of these devices (still too slow). The best performance I have been able to extract from these is when I am doing really stupid busy-wait strategies. Also, single-writer principle and serializing+batching writes before you send them to disk is critical. With any storage medium where a block write costs you device lifetime, you want to put as much effective data per block as possible, and you also want to avoid editing existing blocks. Append-only log structures are what NVMe/flash devices live for. With all of this in mind, the motivations for building gigantic database clusters should start going away. One NVMe device per application instance is starting to sound a lot more compelling to me. Build clusters with app logic (not database logic). In testing of these ideas I have been able to push 2 million small writes (~64k) per second to a single Samsung 960 pro for a single business entity. I don't know of any SQL clusters that can achieve these same figures.
- isotopp 5y agoIn NVME you can get around 800.000 IOPS from a single device, but the latency gives you around 20.000 IOPS sequentially. You need to talk with deep queues or with multiple concurrent threads to the device in order to eat the entire IOPS buffet. Traditional OLTP workloads do not tend to have the concurrency to actually saturate the NVME. You would need to be 40-way parallel, but most OLTP workloads give you 4-way. Multiple instances per device are almost a must.
- bob1029 5y ago> You need to talk with deep queues or with multiple concurrent threads to the device in order to eat the entire IOPS buffet. Completely agree. There is another angle you can play if you are willing to get your hands dirty at the lowest levels. If you build a custom database engine that fundamentally stores everything as key-value, and then builds relational abstractions on top, you can leverage a lot more benefit on a per-transaction basis. For instance, if you are storing a KVP per column in a table and the table has 10 columns, you may wind up generating 10-20 KVP items per logical row insert/update/delete. And if you are careful, you can make sure this extra data structure expressiveness does not cause write amplification (single writer serializes and batches all transactions).
- Syzygies 5y ago"Flash shredders exist, too, but in order to be compliant the actual chips in their cases need to be broken. So what they produce is usually much finer grained, a “sand” of plastics and silicon." For metal, chip credit cards I can't shred, I put them on a brick in the back yard, and torch them with a weed burner. There's a psychological bonus here, if for example Amex made me change cards through no choice of my own because they were hacked. This wouldn't be "compliant" for a flash drive, but it would be effective.
- rbanffy 5y agoOne thing I don’t get in the Mach.2 design is why not use the approach IBM used in mainframe hard drives of having multiple heads over the same platter. The Mach.2 is two drives sharing parts, that will behave like a 2-drive RAID array. Having multiple heads on the same platter would allow it to read like a RAID-1 while writing like a RAID-0.
- kmeisthax 5y agoYeah, I was quite surprised to see "the drive presents itself as two disks"... I can only imagine building the drive this was was seen as the lowest risk in terms of both firmware NREs and additional hardware required. That being said, dual head stacks (as opposed to just dual actuators on a single head stack) would almost certainly not fit within the footprint that ordinary drives do, which would mean not being able to use standard disk enclosures for them... which would be a non-starter. Alternatively you shrink the platters, which means less capacity - and given that hard disk drives have been stuck at a certain areal density limit for a while I imagine nobody's particularly keen on losing lots of capacity for a small IOPS benefit. IDK - maybe if HAMR goes mainstream in hard drives then we can make that tradeoff without the drive seeming like an inferior product.
- rbanffy 5y ago> would almost certainly not fit within the footprint that ordinary drives do Yes. You'd probably need to use 2.5" platters for a 3.5" enclosure. Also not sure if the mechanics would be precise enough to allow one head to read what the other wrote. When IBM did this, densities were MUCH lower. As for capacity vs IOPS, 10K and 15K RPM drives rarely have the same densities as the slower large capacity units. I don't think I ever saw a 10+TB drive faster than 7200 RPM.
- myrandomcomment 5y agoArista was pushing leaf-spine in 2008....
- jeffinhat 5y agoI've been messing with NVMe over TCP at home lately and it's pretty awesome. You can scoop up the last-gen of 10GBe/40GBe networking on eBay for cheap and build your own fast disaggregated storage on upstream Linux. The kernel based implementation saves you some context switching over other network file systems, and you can (probably) pay to play for on-NIC implementations (especially as they're getting smarter). It seems like these solutions don't have a strong authentication/encryption-in-transit story. Are vendors building this into proprietary products or is this only being used on trusted networks? I think it'd be solid technology to leverage for container storage.
- stingraycharles 5y agoI just use iSCSI at home but using Mellanox’ RoCE which is pretty well performing. One thing I’m noticing is that most of these storage protocols do, in fact, assume converged Ethernet; that is, zero packet loss and proper flow control. Is this also the case with NVMe over TCP?
- jeffinhat 5y agoI haven't experimented with it yet, but I expect that over TCP things degrade more gracefully. It seems earlier iterations of storage over networking didn't want to pay the overhead of TCP and lost out on the general purpose benefits that it brings. IIRC some RoCE iterations aren't routable for example. In theory, you could expose your NVMe over TCP device over the internet. It seems to me applications taking advantage of NVMe are focused on building out deep queues of operations which may smooth out issues introduced by the network. But only way to know is to benchmark.
- isotopp 5y agoThe point of TCP is to provide protection against packet loss and reordering, and to provide flow control.
- effie 5y ago> build your own fast disaggregated storage on upstream Linux Why is this better than just connecting the drive via PCIe bus directly to the CPU?
- ramoz 5y agoRandom perspective: I think the author is unintentionally speaking of a future not quite here yet & specifically with AI integrated into applications & their infrastructure. Essentially the need for low data gravity with how AI works today in viable business domains. For large processing of data in general requests (maybe like a traditional SQL query) AI might typically require the retrieval data to map to vector formats. But it's not applicable to add those to RDBs (you're not doing any sql upfront, retrieval will be too slow), nor compute those ad-hoc (business app use cases, not general CV or NLP models, nor analytics). AI/ML inputs/vectors need to exist somewhere where it is fast access vs abstracted software over that access; so using embedded dbs (leveldb, rocks, badger, lmdb, etc). Anecdotally, I get top performance over 8-24 physical ssd attachments (TBs of data), nvme raid-0 single mount on individual nodes, where we need to gather anywhere from 1 - potentially 10s of thousands of vectors every request. It's never more performant to distribute that and nothing is really gained with persistence. Today's platforms are somewhat geared for fast recovery (though writing TB's of data isnt simple, you can model automation & scale horizontally). Still, app platform infrastructure isn't really geared for AI application architectures like this. E.g., Kubernetes is too abstracted, but I imagine Google/GCP will, intentionally for AI workloads, enable CSI drivers and more robustness around Local SSDs on GKE.