9 ms·

"If they had a perfectly normalized database, no NULLing and formally verified code, this bug would not have happened." That may be. What's not specified there

• SoftTalker 11mo ago
  Why is being able to "capture the market" something we want to encourage? This leads to monopolies or oligopolies and makes possible various types of abuse that a free competitive market would normally correct. If you're going to step into the role of managing a large percentage of public internet traffic, maybe you need to be held to a different standard and set of rules than a startup trying to get a foothold among dozens or hundreds of other competitors. Something more like a public utility than a private enterprise.
  • immibis 11mo ago
    It doesn't matter what "we" "encourage". This is a natural selection process: all sorts of teams exist, and then the market decides to be captured by certain ones. We do not prescribe which attributes capture the market; we discover them.
  • xeromal 11mo ago
    I assume wanting a company to succeed is fundamental to hacker news. The world is better of with CF being around for sure
  • dzikimarian 11mo ago
    You would have to completely flip how funding works. As of now most VCs have abysmal returns, so heightening the bar is last thing on their mind.
  • xoa 11mo ago
    The three other replies you've gotten so far have given some generically applicable though still good answers, but I want to address something regard Cloudflare specifically: a major part of their entire core goal and value proposition revolves around being able to defend their customers from continuously scaling ever larger hostile attacks. This isn't merely a case of "natural selection" or what a company/VCs might desire, but that it's hard to see how under the current (depressing, shitty) state of the Internet it'd be possible to cheaply defend against terabit-plus class DDOS and the like without Cloudflare level scale in turn. And "cheaply" is in fact critical too because the whole point of resource exhaustion attacks is that they're purely economic, if it costs many times more to mitigate them then to launch and profit from them then the attackers are going to win in the end. Ideally we'd be solving this collective action problem collectively with standards amongst nations and ISPs to mitigate or eliminate botnets at the source, but we have to trundle along as best we can in the mean time right? I'm not sure there is room for a large number of players in Cloudflare's role, and they've been a pretty dang decent one so far.
• aishsh 11mo ago
  I’d be with you except that cloudflare prioritizes profit over doing a good job (layoffs, offshoring, etc). You don’t get to make excuses when you willingly reduced quality to keep your profits high.
• dfabulich 11mo ago
  That's entirely right. Products have to transition from fast-moving exploratory products to boring infrastructure. We have different goals and expectations for an ecommerce web app vs. a database, or a database vs. the software controlling an insulin pump. Having said that, at this point, Cloudflare's core DDOS-protection proxy should now be built more like an insulin pump than like a web app. This thing needs to never go down worldwide, much more than it needs to ship a new feature fast.
  • jacquesm 11mo ago
    Precisely. This is key infrastructure we're talking about not some kind of webshop.
    • simlevesque 11mo ago
      Yeah but the anti-DDOS feature needs to react to new methods all the time, it's not a static thing you build once and it works forever. An insulin pump is very different. Your human body, insulin, and physics aren't changing any time soon.
      • wathef 11mo ago
        an insulin pump is a good metaphor; insulin as a hormone has a lot of interactions and the pump itself, if not wanting to unalive its user, should (most do not) account for external variables, such as: exercise, heart rate, sickness, etc. these variables are left for the user to deal with, and in this case, is a subpar experience in managing a condition.
      • jacquesm 11mo ago
        You are simplifying the control software of an insulin point to a degree that does not match reality. I'm saying that because I actually reviewed the code of one and the amount of safety consciousness on display there was off the charts compared to what you usually encounter in typical web development. You also under-estimate the dynamic nature of the environment these pumps operate in as well as the amount of contingency planning that they embody, failure modes of each and every part in the pump were taken into consideration, and there are more such parts that you are most likely aware of. This includes material defects, defects as a result from abuse, wear & tear, parts being simply out of spec and so on. To see this as the typical firmware that ships with say a calculator or a watch is to diminish the accomplishment considerably.
• jacquesm 11mo ago
  When you're powering this large a fraction of the internet is it even an option not to work like that? You'd think that with that kind of market cap resource constraints should no longer be holding you back from doing things properly.
  • frumplestlatz 11mo ago
    I work in formal verification at a FAANG. It is so wildly more expensive than traditional development that it is simply not feasible to apply it anywhere but absolutely the most critical paths, and even then, the properties asserted by formal verification are often quite a bit less powerful than necessary to truly guarantee something useful. I want formal verification everywhere. I believe in provable correctness. I wish we could hire people capable of always writing software to that standard and maintaining those proofs alongside their work. We really can’t, though. Its a frustrating reality of being human — we know how to do it better, but nearly all of even the smartest engineers we can hire are not smart enough.
    • jacquesm 11mo ago
      Ok, let's start off with holding them to the same standards as avionics software development. The formal verification can wait.
      • frumplestlatz 11mo ago
        Agreed. I left out any commentary on `.unwrap()` from my original comment, but it’s an obvious example of something that should never have appeared in critical code.
        • echelon 11mo ago
          Rust needs to get rid of .unwrap() and its kin. They're from pre-1.0 Rust, before many of the type system features and error handling syntax sugar were added. There's no reason to use them as the language provides lots of safer alternatives. If you do want to trigger a panic, you can, but I'd also ask - why? Alternatively, and perhaps even better, Rust needs a way to mark functions that can panic for any reason other than malloc failures. Any function that then calls a panicky function needs to be similarly marked. In doing this, we can statically be certain no such methods are called if we want to be rid of the behavior. Perhaps something like: panic fn my_panicky_function() { None.unwrap(); // NB: `unwrap()` is also marked `panic` in stdlib } fn my_safe_function() { // with a certain compiler or Crates flag, this would fail to compile // as my_safe_function isn't annotated as `panic` my_panicky_function() } The ideal future would be to have code that is 100% panic free.
• PunchyHamster 11mo ago
  I'd not be surprised if root of the issue was some engineer who didn't add DB selector because in other SQL engines SELECT like that would select from currently connected database vs all of them
• ihaveajob 11mo ago
  Thank you for putting this in such clear terms. It really is a Catch-22 problem for startups. Most of the time, you can't reach scale unless you cut some corners along the way, and when you reach scale, you benefit from NOT cutting those corners.
• mosura 11mo ago
  Software people, especially coming through Rust, are falling into the old trap of believing if code is bug free it is reliable: it isn’t because there is a world of faults outside, including but not limited to the developer intentions. This inverts everything because structuring to be fault tolerant, of the right things, changes what is a good idea almost entirely.
  • ViewTrick1002 11mo ago
    Rust generally forces you to acknowledge these faults. The problem is managing them in a sane way, which for Rust in many cases simply is failing loudly. Compared to than many other languages which preferring chugging along and hoping that no downstream corruption happens.
    • gishh 11mo ago
      [flagged]
      • saati 11mo ago
        Was it a memory error or a data race? No. Rust only promises that those won't happen in safe Rust. What is embarrassing is trying to pin this on a specific programming language.
        • mikestorrent 11mo ago
          [flagged]
          • saati 11mo ago
            But it wasn't the culprit, the code could have been in anything, or could have bubbled up errors to main, and it still would have failed with for an incorrect config.
            • gishh 11mo ago
• bambax 11mo ago
  But does "formally verified code" really go in the same bag as "normalized database" and ensuring data integrity at the database level? The former is immensely complex and difficult; the other two are more like sound engineering principles?
• necovek 11mo ago
  This bug might not have, but others would. Formal verification methods still rely on humans to input the formal specification, which is where problems happen. As others point out, if they didn't really ship fast, they certainly would not have become profitable, and they would definitely not have captured the market to the extent they have. But really, if the market was more distributed, and Cloudflare commanded 5% of the web as the biggest player, any single outage would have been limited in impact. So it's also about market behaviour: yet "nobody is fired for choosing IBM" as it used to go 40 years ago.
• FloorEgg 11mo ago
  I agree with you. I would just add that I've noticed organizations tend to calcify as they get bigger and older. Kind of like trees, they start out as flexible saplings, and over time develop hard trunks and branches. The rigidity gives them stability. You're right that there's no way they could have gotten to where they are if they had prioritized data integrity and formal verification in all their practices. Now that they have so much market share, they might collapse under their own weight if their trunk isn't solid. Maybe investing in data integrity and strongly typed, functional programming that's formally verifiable is what will help them keep their market share. Cultures are hard to change and I'm not suggesting an expectation for them to change beyond what is feasible or practical. I don't lead an engineering organization like it so I'm definitely armchairing here. I just see some of the logic of the argument that them adopting some of these methods would probably benefit everyone using their services.
• yearolinuxdsktp 11mo ago
  Not to mention that perfectly normalizing a database always incurs join overhead that limits horizontal scalability. In fact, denormalization is required to achieve scale (with a trade-off). I’m not sure how formal verification would’ve prevented this issue from happening. In my experience, it’s unusual to have to specify a database name in the query. How could have formal verification covered this outcome? The recommendations don’t make sense saying that the query needed DISTINCT and LIMIT. Don’t forget that the incoming data was different (r0 and default did not return the same exact data, this is why the config files more than doubled in size), so using DISTINCT would have led to uncertain blending of data, producing neither result and hiding the double-database read altogether. Secondly, LIMIT only makes sense to use in conjunction with a failure circuit breaker (if LIMIT items is returned, fail the query). When does it make business-logic sense to LIMIT the query-in-question’s result? And do you think the authors would have known how to set the LIMIT to not exceed the configuration file consumers’ limitations? The article says: > “You can’t reliably catch that with more tests or rollouts or flags. You prevent it by construction—through analytical design.” That’s the big design up front fallacy. Of course you can catch it reliably with more tests, and limit the damage with flags and rollouts. There’s zero guarantee that the analytical design would’ve caught this up front.
  • b-man 11mo ago
    > Not to mention that perfectly normalizing a database always incurs join overhead that limits horizontal scalability. In fact, denormalization is required to achieve scale (with a trade-off). This is just not true, at least not in general. Inserting on a normalized design is usually faster, due to smaller index sizes, fewer indexes and fitting more rows per page.
• no_wizard 11mo ago
  Why is this inherently slower? There’s for example, languages or features of languages that work entirely on not allowing these things. I ask because I feel like I’m missing something
• swiftcoder 11mo ago
  > It limits, radically, the percent of engineers you can hire (to those who understand this and are willing to work this way), and it slows deployment radically. We could also invest in tooling to make this kind of thing easier. Unclear why humans need to hand-normalise the database schema - isn't this exactly the kind of thing compilers are good at?
  • b-man 11mo ago
    Normalization cannot be done by machines, because it depends on expressing the (and only the) predicate that corresponds to the business rule in question. It requires apprehending the essence of the situation, something a machine cannot do.
    • swiftcoder 11mo ago
      Sure it can. Not in a vacuum, maybe, but with some guidance from the user as to dependency relations. Ideally you have enough data baked into your schema to infer those relations.
      • b-man 11mo ago
        a fully normalized relation is one where the SQL (say) table in question represents one and only one predicate of your business rules. It is literally impossible for that to be done automatically. Someone needs to look at the resulting code and confirm that that was the case.
        • swiftcoder 11mo ago
          Or they need to write the business rules down, in some sort of expressive format that can be tested against the requirements... Say, an artificial language of some sort. One might call it "programming"
• TYPE_FASTER 11mo ago
  What I have seen work in the past is testing using a production backup as a final step prior to releasing, including applying database scripts. In this case, the permissions change would have been executed, the query would have run, and the failure would have been observed.
• Dylan16807 11mo ago
  Formally verifying code is an enormous endeavor. But a normalized database without NULL should not be a significant burden.
• femiagbabiaka 11mo ago
  database normalization and formal verification aren't on the same level of difficulty in terms of implementation, and we all could do the former from the beginning, if we choose to (nobody ever chooses to)
• uoaei 11mo ago
  > It limits, radically, the percent of engineers you can hire (to those who understand this and are willing to work this way), and it slows deployment radically. Never seen the amoral-capitalist argument to stunting progress at the expense of profit put so succinctly!
• b-man 11mo ago
  > That may be. What's not specified there is the immense, immense cost of driving a dev org on those terms I'm happy that we agree on the solution, but disagree only if it is cost worthy. About the cost, I took that into consideration when I wrote the conclusion: > FAANG-style companies are unlikely to adopt formal methods or relational rigor wholesale. But for their most critical systems, they should. It’s the only way to make failures like this impossible by design, rather than just less likely. There is an actionable plan in the article. It is possible to run teams like these. It is an economical decision of upper management to run the risk of having these outages vis-a-vis this alternative.