4 ms·
Your issue was exactly, and I mean exactly, what I had in mind when I said “learning experiences” upthread. I followed it closely. Linode painfully learned the
by yte15 6y ago
Your issue was exactly, and I mean exactly, what I had in mind when I said “learning experiences” upthread. I followed it closely. Linode painfully learned the scrubbing lesson as well - and DO was in a worse position to learn it at the time, being forced to throw the scrub write load at SSDs that don’t like that type of load. That’s the real issue, honestly, that they probably started noticing SMART warnings from scrubbing by default on SSDs (particularly for cheap instances that turn over often), and took the venture capital approach of taping over it to make the next round by making it opt in.
It’s a tough problem to deal with and not something you’re likely to think of designing such a product from first principles. That’s not an indictment, it’s just fundamental to experience gained doing this stuff (and it’s perilous to get wrong). Filtering unsolicited ARP to prevent domUs from hijacking default gateways was another lesson in blood, and one of the first things we tried within five minutes when we did competitive on DO around launch time (they hadn’t thought of it; it worked).
- sneak 6y agoIt's not tough to solve. The whole scrub thing was a red herring: you don't need to scrub anything to not leak data, you just thin provision. It's (mostly) fine if customer data stays on your disks after they delete. It's not fine to give it to the next customer because you don't know how disk abstraction works. My issue is mainly how they coped with it, which has nothing to do with their (at the time) technical incompetence: they simply lied about the effects. Competent or not, lying on your corporate blog isn't a good choice. It doesn't take any special skills or training to be honest on your journey from incompetence to competence, all it takes is integrity.
- yte15 6y agoYeah. Agreed. I unfairly walked back solve to “deal with” on you simply from knowing the considerations that go into bursts of write load every time a customer clicks “delete”. At the time Linode had a host-side scrub daemon that simply ate LVs as customers deleted. It’s technically simple, but drive longevity and capacity on the host (the user is probably recreating their VM, for example) is where it gets tricky. External considerations. In the end scrubbing is basically killing a few dozen inodes of user data, but the architecture in VPS usually requires you to nuke the whole image, including what is ultimately the host’s 59th copy of Ubuntu. Just managing the iops without annoying neighboring customers is a challenge. To my cloning point I made at the top of the thread, Linode was certainly no stranger to the reality distortion field when it suited, and I think that’s yet another thing DO copied. That vertical is closer to B2C since you’re usually dealing with individuals, not sales teams, and glossing over stuff is a bit easier than when your counterparty has a better engineering team than you.
- voltagex_ 6y agoWhat do you mean by "thin provision" here?
- sneak 6y agoUse an abstraction layer between the virtualized block device and the underlying storage system, and only write used blocks to disk, and return zeroes for all unused blocks. You don't need to wipe, zero, or TRIM anything to not leak customer A data to customer B.