8 ms·
We Replaced an SSD with Storage Class Memory
- renewiltord 6y agoJesus Christ, this is insane. Almost a Terabyte of 12.6 Gbps reads? I have a bunch of geospatial entity resolution workloads that I could absolutely smash with this. For way cheaper than the fat mem instances.
- lrem 6y agoBack in the day when I interviewed for Google I had this beautiful question. The interviewer fished for a basic distributed key-value store. I just kept coming up with single machine solution to his numbers. "No, I really can have that storage+bandwidth, here's the part number." I'm still wondering if that interview costed me a level.
- lmilcin 6y agoBeing interviewer myself I can say interviewers tend to be stubborn. They have an excellent question (at least they think they have) that they have spend a lot of time thinking through and discussing with other candidates. Your answer causes him/her to miss the opportunity to have a real discussion and understand your knowledge of the topic. It is better to play along and only mention the real solution at the end, to finish on a high note.
- thechao 6y agoAlternately, the interviewer should recognize that their question is flawed & move on. I’ve had candidates nail a 30 minute discussion question with a 2 minute nonobvious answer. Shit happens: you’re not the smartest person in the room that day.
- dijit 6y agoIt should go without saying that as an interviewer; I am never the smartest person in the room. I am a person with different experiences and I need to find if you’re a fit- not if you’re unintelligent. If you got to me, you’re not unintelligent.
- AnIdiotOnTheNet 6y agoMost interviewers though aren't actually any good at it and very often don't like not being the smartest person in the room.
- lmilcin 6y agoWhether you are interviewer or interviewee, putting your ego before the goal of the interview is just going to waste the time of both parties. I can't hire a person that can't restrain their ego for the duration of the interview and an interviewer that is focusing on anything else than figuring out if the candidate is right person for the job is causing disservice to everybody.
- sukilot 6y agoThe thing to remember is that in some companies the interviewer doesn't care about hiring and is only their because they were forced to.
- dhosek 6y agoI guess it depends on where you're doing the interviewing. I've had surprisingly unintelligent people get to me in the interviewing stages. It amazes me how many developers, experienced even, who can't write a simple recursive function. And we're not even looking at, "can you come up with a recursive function as a solution to this problem?" we're looking at "write a recursive function to calculate the factorial" as the question.
- joefourier 6y ago
- ur-whale 6y agoIf the interviewer was gauging your political skills/ adaptability, it doesn't look like you fared too well indeed.
- Frost1x 6y agoThat seems like an excuse for poor behavior moreso than a deeper level of testing (testing when you know you're right but someone with more authority tells you otherwise). While possible, I'd have to think it's unlikely.
- lrem 6y agoDo you test new grads for their political skills? I don't.
- deleted 6y ago[deleted]
- asdfasgasdgasdg 6y agoEven if you can do a particular problem on a single machine, sometimes that isn't the right call. In a work scheduled cluster environment, a task that wants the entire machine may have trouble getting a slot unless it has the priority to preempt everything else going on on that machine. We call such VMs "picky" and they don't get scheduling guarantees.
- michaelt 6y agoI'm having trouble thinking of how you'd end up with a "cluster" that couldn't provide the power of a single machine?
- asdfasgasdgasdg 6y agoTo make it concrete: consider a cluster of ten machines with 256G of ram. Team A and team B both schedule jobs with a requirement of 129G of RAM and five replicas each. The replicas get scheduled, one on each machine. Team C wants to schedule a task that takes an entire 256GB of ram. If they don't have sufficient priority to preempt the other jobs, then they will not schedule. In real heterogenous-workload production clusters, every available machine likely has several VMs scheduled on it if the cluster isn't idle. There is never a full machine that's free unless some special effort has been taken to make it so.
- throwaway_pdp09 6y agohttp://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html http://www.frankmcsherry.org/graph/scalability/cost/2015/01/... "Rather than making your computation go faster, the systems introduce substantial overheads which can require large compute clusters just to bring under control. In many cases, you’d be better off running the same computation on your laptop." My limited experience fits this in that a bit of smarts on a single box beats a bunch of boxes. (the link is a very good read BTW)
- yencabulator 6y agoThe biggest problem with that write-up is ignoring availability. "Fast all the way up to the crash" can be much worse than slow and steady. Of course, for a batch job with a runtime under 11 minutes, that probably doesn't really matter too much. Just don't generalize that too much.
- dominotw 6y ago> solution to his numbers. Why did he have concrete numbers. Couldn't he simply have increased those numbers when you proposed single machine solution.
- twic 6y agoAn interviewer once gave me a problem about scraping some information out of a log file. I wrote a short shell pipeline. The interviewer asked for a more complicated analysis. I wrote a longer pipeline. The interviewer added more requirements, with aggregation, state, etc in order to push me to write an actual program. I wrote an even longer pipeline. By this point we had both realised that this was a battle of wits: could he come up with a problem that i couldn't solve with a pipeline? At the end of the interview, i had a pipeline that took up most of a piece of A4 paper to write out. I had won the battle, and was offered the job. Of course, i would not advise you to actually write a pipeline like that in production, but it's a fun exercise. Anyway, the moral of this story is that if the interviewer wants you to solve a problem a certain way, and you can solve it in a simpler way, then a good interviewer will mark you up, not down. Perhaps at Google they didn't; they don't really seem like a company that has it together.
- sukilot 6y agoYour thesis is that a company that has it together would only write software that is either a shell pipeline or can be solved in 45minues? > would not advise you to actually write a pipeline like that in production, So when you are interviewing for a production job, why would you fight for non-production quality solutions?
- sukilot 6y agoYour interviewer may have lacked skill in guiding the process, but you could have taken the question in the spirit it was given and answered "now if the usage grows 10x as much as hardware advances" or "if we need to be resilient to hardware failures" or "if we need to roll out a logic upgrade slowly" and continued to solve the problem. > I'm still wondering if that interview costed me a level. Unlikely. Candidate level is decided before the interview.
- jiggawatts 6y agoGBps, not Gbps! They were getting 12.6 gigabytes per second, which would be 100 gigabits per second almost exactly. However, even their 6-DIMM test produces only 300 Gbps, which is insufficient to saturate a modern 400 GbE network adapter for either reads or writes. This would be most relevant on a "single master" system storing some sort of simple data where consistency requirements means that the "writer" cannot be distributed. In a situation like this, the NIC and the storage bandwidth are the ultimate limits. In general, Intel SSDs and Intel Optane have poor but consistent bandwidth, and consistently low latency. Coupled with the high price and small capacity, they have their niche, but they're not a clear winner in any category. As a reference point for how crazy high bandwidths are these days, NVIDIA sells a turnkey solution with 200 GB/sec network bandwidth (1.6 Tbps!): https://www.nvidia.com/en-au/data-center/dgx-a100/ https://www.nvidia.com/en-au/data-center/dgx-a100/
- lmilcin 6y agoWhere this is useful is when the application needs to process much more data than it then needs to transmit over the network. For a database system like mongodb this could be perfect depending on workload.
- Jabbles 6y agoDoes anyone have good numbers for a) RAM, and b) the cost of each of these solutions? (RAM is obviously volatile.)
- Pelic4n 6y agoSo you need to do THAT to get decent perfs with MongoDB. Good to know!
- jiggawatts 6y agoUnless I'm missing something, their benchmark graphs at the bottom of the report show that there is no significant benefit to using SCM with MongoDB! Their internal overheads must be high enough to swamp the few microseconds gained from the faster storage.
- Pelic4n 6y agoThat was a joke. I've been using MongoDB in production for years and I'm salty. :)
- fortran77 6y agoThat was my first thought, too. You need these tricks to make MongoDB "WebScale" http://www.mongodb-is-web-scale.com/ http://www.mongodb-is-web-scale.com/ (I can get better results for Key-Value storage by using an SQL Database--Postgres or MariaDB--for key/values over MongoDB. And if you know SQL well, you can get even better results using a real relational database and optimal queries than pulling out keys/values and ad-hocking your relations in some Javascript code or whatever these web kids are doing.)
- jd_mongodb 6y agoYou don't pick a database like MongoDB purely for its key/value query performance. SQL will do really well until it does really badly, for example in cases where you exceed the limitations of the box/VM it is running on.
- lichtenberger 6y agoYou can efficiently read 256 Byte granular data (4 cache lines) with Optane Memory (due to checksums). I think it makes much more sense to read/write fine granular changes for instance at least align pages to 64 or 256 Bytes instead of 4kb pages, where you often times first of all write too much data and secondly you pollute the caches with probably unnecessary data. There's a paper about how to add cache line aligned mini-pages (16 cache-lines): https://db.in.tum.de/~leis/papers/nvm.pdf https://db.in.tum.de/~leis/papers/nvm.pdf
- ayende 6y agoWriting on a page (4KB / 8KB) boundary is almost always a good idea. Because there is an overhead per page that you have to account for. It can be in the range of 16 - 64 bytes, so having a 256 mini page is probably a bad idea. Most of the _data_ is also not going to fit in 256 bytes anyway.
- lichtenberger 6y agoIt might add some overhead, but I guess it depends on the page implementation, but 16 bytes seems to be the minimum (and Optane Memory might , I agree. That said if someone changes only one record the best thing is to write in the smallest granularity possible on the storage medium. So, it might well be that someone is only interested in only one or a few records. Why then fetch and cache a whole 4Kb page if latency is good in both cases (4kb and 256 bytes)? On the other hand I agree that you should probably cache more data from a hot page.
- lichtenberger 6y agoI think along those lines, reading and writing fine granular data, especially when storing the full history of the data is the way to go (in addition to path copying and a log-structured storage). Furthermore pointer swizzling might be needed to get almost in-memory database performance. I'm trying to address this with my versioned Open Source DBMS project, where only page-fragments are stored and bunch of them is read in-parallel to reconstruct a full page in-memory. Adding mini-page ps plus a simple cache at some point with hot data from several page fragments at least is orthogonal: https://github.com/sirixdb/sirix https://github.com/sirixdb/sirix | https://sirix.io https://sirix.io
- cm2187 6y agoStupid question. What are the use cases for such massively fast write speeds? If you are storing data to disk at that speed, you fill even the biggest optane drives in a couple of minutes. So it would be an application where you need to overwrite a huge amount of data over and over again.
- stonewhite 6y agoThat also necessitates a matching network bandwidth as well. It reminds me of the Bugatti Veyron situation, its tires lasting 15 minutes at top speed yet it runs out of fuel in 12.
- falcolas 6y agoI'd rather run out of fuel and drift to a stop, instead of having a blowout and crash while driving at top speed. I'm not sure where the problem lies.
- ReactiveJelly 6y agoIt's not a problem with the car, it's an analogy that the fuel and the network are bottlenecks where normally the tires and storage might be.
- falcolas 6y agoStill doesn’t make sense. The car is correctly configured (for safety purposes), and it would also make sense to be able to write to storage faster than your network speeds (no database writes exactly what it receives from the network, it adds metadata and structuring).
- kristianpaul 6y agoTime series databases perhaps?
- dr_zoidberg 6y agoDigital forensics is a situation where you need extreme IO but you aren't really writing all that much -- it's mostly searches over the forensic copy of computers you are analizing. As for write speed, you don't really need it unless you're doing file/data recovery, but even then you can just make indexes to the disk image. AFAIK, very few (if none) are using SCM in their forensic expert workstations, so I can't really tell if you'd saturate the storage capacity or the bandwidth. But in digital forensics, NVMe's have been a must for a while, so this would definitely help.
- georgewfraser 6y agoAndy Pavlo talks about this in his class at CMU. You shouldn’t expect to get better performance by running a disk-optimized storage engine on memory, because you’re still paying all the overhead of locks and pages to work around the latency of disk, even though that latency no longer exists. Instead, you have to build a new, simpler storage engine that skips all the bookkeeping of a disk-oriented storage engine. https://youtu.be/a70jRWLjQFk https://youtu.be/a70jRWLjQFk
- mackle_hair 6y agoi believe there are other issues with SCM arent there, for instance if you flip the same bit over and over, you'll wear out the media.. so just like with SSD's there has to be some wear leveling algorithms, but because the media has gotten so fast, the wear leveling would theoretically become a larger % of the bottleneck of using the drive. but i totally agree, rethinking how data is stored is going to be key to the adoption of these types of new media. there's a company called vast data that has built out a full storage solution utilizing the unique properties of SCM, very cool https://vastdata.com/ https://vastdata.com/
- pocket_cheese 6y agoFor anyone interested in how a database works underneath the hood, I could not recommend a better MOOC than Andy Pavlo's database courses. His intro to database course is so freaking good.