3 ms·
That's entirely right. Products have to transition from fast-moving exploratory products to boring infrastructure. We have different goals and expectations for
by dfabulich 11mo ago
That's entirely right. Products have to transition from fast-moving exploratory products to boring infrastructure. We have different goals and expectations for an ecommerce web app vs. a database, or a database vs. the software controlling an insulin pump.
Having said that, at this point, Cloudflare's core DDOS-protection proxy should now be built more like an insulin pump than like a web app. This thing needs to never go down worldwide, much more than it needs to ship a new feature fast.
- jacquesm 11mo agoPrecisely. This is key infrastructure we're talking about not some kind of webshop.
- simlevesque 11mo agoYeah but the anti-DDOS feature needs to react to new methods all the time, it's not a static thing you build once and it works forever. An insulin pump is very different. Your human body, insulin, and physics aren't changing any time soon.
- wathef 11mo agoan insulin pump is a good metaphor; insulin as a hormone has a lot of interactions and the pump itself, if not wanting to unalive its user, should (most do not) account for external variables, such as: exercise, heart rate, sickness, etc. these variables are left for the user to deal with, and in this case, is a subpar experience in managing a condition.
- jacquesm 11mo agoYou are simplifying the control software of an insulin point to a degree that does not match reality. I'm saying that because I actually reviewed the code of one and the amount of safety consciousness on display there was off the charts compared to what you usually encounter in typical web development. You also under-estimate the dynamic nature of the environment these pumps operate in as well as the amount of contingency planning that they embody, failure modes of each and every part in the pump were taken into consideration, and there are more such parts that you are most likely aware of. This includes material defects, defects as a result from abuse, wear & tear, parts being simply out of spec and so on. To see this as the typical firmware that ships with say a calculator or a watch is to diminish the accomplishment considerably.
- joshuamorton 11mo agoI had a former coworker who moved from the medical device industry to similar-to-cloudflare-web software. While he had some appreciation for the validation and intense QA they did (they didn't use formal methods, just heavy QA and deep specs), it became very clear to him very clearly that those approaches don't work with speed-of-release as a concern (his development cycles were annual, not weekly or daily). And they absolutely don't work in contexts where user-abuse or reactivity are necessary. The contexts are just totally different.
- jacquesm 11mo agoIt is perfectly possible to engineer for faster cycles without losing control over what your code can and can not do. It is harder, for sure. But I do not think it is a matter of this absolutely not working, that's black-and-white and it never is black and white, it is always some shade of gray. For instance: validating a configuration before loading it is fairly standard practice, as are smoke tests and gradual roll-outs. Configuration fuck-ups are fairly common so you engineer with that in mind.
- mikestorrent 11mo agoThanks for spelling that out. It's so often tempting to be reductionist about things, but there is often a tremendous amount of thankless engineering inside products that we are privileged to consider as being somewhat boring. It takes a lot of work to make something so dynamic and life-critical and make it reliable enough to be considered simple, when it's anything but.
- MichaelZuo 11mo agoIf humans beings had a small chance to transform into say quadrupeds or suddenly grow tenatacles, extra hearts, organs, etc., in any given year… then wouldn’t designing a safe insulin pump literally be impossible?
- jacquesm 11mo agoCompared to what they are already doing it would be marginally more difficult.
- saghm 11mo agoAll the more reason to be careful about relying on humans to avoid making mistakes when changing it rather than moving quickly and letting things fail in production.
- b-man 11mo agoIndeed. I was trying to make that point on my concluding paragraph.
- Aperocky 11mo ago> This thing needs to never go down worldwide Quantity introduce a quality all of its own in terms of maintenance.