6 ms·
The study likely assumed that people already familiar with working in these methodologies were equally available, equally happy to do so, and equally priced as
by contingencies 11y ago
The study likely assumed that people already familiar with working in these methodologies were equally available, equally happy to do so, and equally priced as regular developers, which is unlikely to be case. A certain minimum team size is probably required to make many of these processes viable; many real world systems are written by solo coders. Also, in reality it is relatively trivial to scavenge work like seL4, Linux kernel security toolkits, software HA clustering and/or proxy implementations, hardware watchdogs, ssss and/or distribute infrastructure across disparate locations to provide what probably amounts to magnitude higher redundancy and resilience against real world attacks or component failures without paying those costs. Indeed, the implementation of such is becoming near free many cases due to convergence on SoA/microservices.
To put this more plainly, "I can get another if I break it, so a clay cup trumps a grail." - Mirza Asadullah Khan Ghalib, classical Urdu and Persian poet from the Mughal Empire ... quote #1 in my fortune db @ https://github.com/globalcitizen/taoup https://github.com/globalcitizen/taoup
- nickpsecurity 11y ago"The study likely assumed that people already familiar with working in these methodologies were equally available, equally happy to do so, and equally priced as regular developers, which is unlikely to be case." For things like Cleanroom and PSP/TSP, they were first-time users of the process. Not long-time. Just decent developers shown a better way of getting results. In high assurance, it varied from first-time to experienced users. First time users usually gained the benefits of the method but with difficulty. There'd be at least one specialist aiding the team. Best results came from people who had done it a while. So, for high that's partially true and for stuff I push on commercial sector it's the opposite. "A certain minimum team size is probably required to make many of these processes viable; many real world systems are written by solo coders. " That's a legit critique. They can still benefit as I experienced myself and some others. It requires one to divide the work into periods where you wear various hats. It takes discipline to externalize one's own work and attack it objectively. Good news is that inspection, testing, static analysis, model-checking, design-by-contract... these all catch problems that are objective more than subjective. Still get plenty of results. There's been one or two threads on here where even solo people do QA on their own stuff with good results. Still open-ended problem worth some experiments to see how much can be done at what effectiveness for a solo coder juggling work. So, good point. "in reality it is relatively trivial to scavenge " It's so trivial we had today yet another HN thread about distributed locks showing how people were copying one implementation and screwing it up. Then there was disagreement in the comments. There's threads like that on various critical things at least once a day or so on HN. Shows it's trivial to do and hard to do right for many things. Better to make re-usable, strong solutions to the various problems we keep encountering where possible. Can always mix high, medium, and average assurance components as a compromise where we have X goals but only Y resources. Pieces can improve over time. "Indeed, the implementation of such is becoming near free many cases due to convergence on SoA/microservices." I've yet to see one go 5 years under heavy load without downtime much less the 17 years VMS clusters did or 30+ years for some IBM mainframe installations. For security, things are more abysmal. I'll agree many solutions are popping up that can do amazing things. Robust, scalable, fairly secure, and near-zero-downtime systems don't seem to be among them in this nearly-free, mainstream market you're talking about. A rarity in the market in general. So, there's still room for both medium and high assurance software. To be clear, I'm only pushing medium because even first-timers did a better job than most do today by eliminating the problem areas & doing great detection. Similar cost, too. Only hoping for high-assurance to be selectively applied to high-impact stuff that doesn't change often. A recent example was a formal verification of SCOOP: Eiffel's race-free, easy-to-analyze, concurrency model. It found issues in the model that were fixed. Three other analysis, ranging from specific to general, eliminated deadlocks, livelocks, and performance-hit respectively. That's the kind of situation where a significant investment of resources can keep providing benefits with almost no effort. Otherwise, medium assurance aka commercial + sane extras. ""I can get another if I break it, so a clay cup trumps a grail."" Nice quote and good principle in much of current economy. Not so much my personal data or business I.P. when a byzantine failure takes out it and my hot standby. I think the comparison mainly applies to non-critical services and hardware. ;)
- contingencies 11y agoInteresting discussion, I'd be keen to try such a formalistic approach if you ever have a need for a remote critical eye. Regarding the quote, I don't think you quite took the spirit of it: the idea is that, given a replaceable component with failure expected, we can manage reasonably to achieve the same outcomes for less overall cost - ie. do not build the one true system to solve all availability and security challenges (which would be very expensive (the old "pick two" adage)), but rather build a system which makes use of the properties of components intelligently to reduce overall hassle (as measured in thought, hassle, time, cost, third party dependencies, untested code or processes) to achieve the same objectives, just as reliably/securely. I think the main reason there are less modern systems running for 15+ year timeframes is that the centralized models of yore only remain valid within traditional large organizations that are slow moving: finance, government infrastructure, military, etc. Even there, they are being dismantled, as COTS hardware + FOSS topple traditional mainframe vendors through cheap and highly scalable clustering. The paradigm has shifted: today we have components like the Linux kernel and sqlite which despite not generally running a single instance for >15 years (though they are definitely capable, and with Linux it has been done; sqlite is not old enough yet!) they do run in harsh environments with limited resources in hundreds of millions - nay - billions of instances, extremely reliably.
- nickpsecurity 11y ago"Interesting discussion, I'd be keen to try such a formalistic approach if you ever have a need for a remote critical eye." I'll try to keep it in mind. There's a few people doing stuff in public if you want to play with it. Plus lists to stuff to try. Here's a few. One person did an overview of Cleanroom and examples in Python, etc here: http://infohost.nmt.edu/~shipman/soft/clean/ http://infohost.nmt.edu/~shipman/soft/clean/ I just pulled a list of techniques that were empirically proven to increase robustness into a Pastebin: http://pastebin.com/xZ6m4T8Z http://pastebin.com/xZ6m4T8Z Note: One can pick and choose among these to find right work/benefit ratio. I'd like to see lots of experiments on mainstream software to get data on what's 80/20 rule. I'm betting on sane subsets of safe languages, code reviews, usage-driven testing, interface checks (even asserts), and fuzz testing maybe. The first four are widely available and take little work. Case study of Altran's Correct-by-Construction approach showing great metrics: https://www.commoncriteriaportal.org/iccc/7iccc/t1/t1201100.pdf https://www.commoncriteriaportal.org/iccc/7iccc/t1/t1201100.... Note: Altran does plenty of medium and high assurance work. Their method can do either. I'd rate Tokeener as Medium-High if just focusing on their part of it. Note that they got it done with 3 part-timers in less than a year with code-level proofs. Quite a leap from millions spent on teams of geniuses doing proofs in Gypsy, etc two years at a time back in the day. :) "Regarding the quote, I don't think you quite took the spirit of it: the idea is that, given a replaceable component with failure expected, we can manage reasonably to achieve the same outcomes for less overall cost " I'm not questioning the spirit so much as the implementation. " do not build the one true system to solve all availability and security challenges (which would be very expensive (the old "pick two" adage)), but rather build a system which makes use of the properties of components intelligently to reduce overall hassle" That's what I advocated. With medium assurance, it cost almost nothing extra so apply it to the components. With high assurance, if used at all, it can be used on just the most important and is by some niche players with good cost/benefit results. It's just that... "to achieve the same objectives, just as reliably/securely." ...this has rarely proven to work. There's an old rule that you can't make high robustness systems by composing low-robustness components. They experience both individual and byzantine failures. That's why we see the companies building on them experiencing downtime that shouldn't happen on top of unavoidable downtime which I don't judge. So, certainly break the problem into smaller ones and compose simple (or cheap) solutions where possible. You just have a baseline of quality you need for the components. Commonly expressed as "weakest link" or "Garbage In, Garbage Out." "I think the main reason there are less modern systems running for 15+ year timeframes is that the centralized models of yore only remain valid within traditional large organizations that are slow moving: finance, government infrastructure, military, etc." Someone else speculated the same thing. There's some truth to it. Yet, there's overlap between the kinds of services running on them and in the modern, downtime-prone infrastructures. The stuff works on robust systems of old times but fails on new ones. What's the key difference? One was designed to be robust and one is a composition of pieces of known to unknown quality. Garbage in, garbage out. "as COTS hardware + FOSS topple traditional mainframe vendors through cheap and highly scalable clustering." They're definitely cheaper and faster. That's been the case since Beowulf clusters I built back in the day. Yet, their uptimes still haven't reached what OpenVMS hit in the 80's with bullet-proof clustering. Are you truly saying hardware and admin labor is so free right now that we can just throw dozens to hundreds of servers in 3-5 locations at every businesses' problems while ignoring the architectural, interface or code quality of any of it? I'll take QNX 2 here and 2 there over a bunch of Linux boxes any day so long as budget will allow the license. I'll have predictability, performance, self-healing, damage containment, and live updates all in a few megs of well-written code w/ POSIX apps. See the difference between that and mainstream systems? Good architecture still matters. System/38 (AS/400), OpenVMS, NonStop, QNX, BeOS... all great architecture that squeezed way more out of any given piece of hardware, software, or admin effort. OpenVMS was also an example of medium assurance that was point of our discussion. Their Red/Blue iterations did 1 week development & test writing then 1 straight week of review and fixes with regression tests over weekends. Results were boxes so reliable admins sometimes forgot how to reboot it and such. Still not common on Windows and Linux boxes despite the great strides in reliability that nearly a billion dollars worth of labor bought those bad architectures. I like to poke at Linux folks that MINIX 3 eliminated most crashes in a few years with a small team while UNIX architecture took decades to mostly do the same thing. Architecture & assurance activities matter. "though they are definitely capable, and with Linux it has been done" I'd like to see evidence of that. The mainframes, AS/400's, VMS, NonStop's... these run all sorts of workloads without software repairs for years. Heavy workloads. I've investigated numerous claims of individual Linux boxes doing the same. It was almost always someone's barely-utilized, hobbyist web server, DHCP server, etc. I want to see the equivalent of a mainframe or VMS server running everything in a business 24/7 without a crash in years. Optionally that system getting lost due to never needing maintenance with people scouring the building looking for it. That was a recurring problem with pizza box servers and embedded cards running aforementioned OS's. ;) "they do run in harsh environments with limited resources in hundreds of millions - nay - billions of instances, extremely reliably." Again, a source for this. They certainly run regular workloads they're optimized for quite reliably. The cloud vendors optimizing on specific HW configurations with all kinds of disaster tolerance appear to run them reliably by hiding the failures. Yet, surveys of industry on their servers' downtime contradicts the overall claim with all kinds of downtime still in mainstream stuff. Plus, they show recent Windows servers ahead of Linux in reliability and AIX usually stayed on top. They don't include the stuff I mentioned due to maybe market share or perhaps the contrast would make the others look bad. ;) Nonetheless, adopting medium assurance methods to build our components or using components built that way is a solution. It was proven... is used... in many commercial systems that have higher quality at acceptable price and performance. Outside one-offs, the only reason it's not happening is lack of knowledge or will to do it. Mainstream devs were told about the stuff for years so they just don't give a shit. Surprisingly, Microsoft is ahead in efforts as they implemented SDL, formal methods on drivers, integrity controls, app protections, safe language use, and so on. I would love to see that level of commitment on BSD and Linux side given I use both. Least they sometimes accept fixes from academics who apply various tools in their spare time or with grant funding. ;)