8 ms·

With fear of sounding like a douche-bag, I honestly believe there's A LOT of incompetence in the tech-world, which permeates all layers, security companies, AV

• commandersaki 2y ago
  I know people really dislike how Apple restricts your freedom to use their software in any way they don't intend. But this is one of the times where they shine. Apple recognised kernel extension brought all sorts of trouble for users such as instability, crashing, etc. and presented a juicy attack surface. They deprecated and eventually disallowed kernel extensions supplanting them with a system extensions framework to provide interfaces for VPN functionality, EDR agents, etc. A Crowdstrike agent couldn't panic or boot loop macOS due to a bug in the code when using this interface.
  • Avamander 2y ago
    Microsoft is also trying to make drivers and similar safer with HVCI, WDAC, ELAM and similar efforts. But given how a large part of their moat is backwards compatibility, very few of those things are the default and even then probably wouldn't have prevented this scenario.
    • nimish 2y ago
      > large part of their moat is backwards compatibility This is more of a religious belief than truth, IMO. They could strong-arm recalcitrant customers, but they don't.
      • vetinari 2y ago
        > They could strong-arm recalcitrant customers, but they don't. They really can't. When the customers have to redo their stack, they might do that in a way that doesn't need Microsoft at all.
        • nimish 2y ago
          These customers wouldn't be able to do that in time frames measured in anything but decades and/or they would risk going bankrupt attempting to switch. Microsoft has far more leverage than they choose to exert, for various reasons.
    • nullindividual 2y ago
      Microsoft has routinely changed the display driver model, breaking backward compatibility. They've also barred print drivers.
  • rightbyte 2y ago
    If the user want remote code execution (auto updates are) in kernel space, let them. Apple sell the whole hardware stack. I don't think limeting drivers would fly on Windows or Linux.
    • commandersaki 2y ago
      Pretty sure there's an exception for drivers but requires at minimum notarisation from Apple, but more likely a review as well.
      • madeofpalk 2y ago
        They just developed a new framework that allows drivers to work just in user space https://developer.apple.com/documentation/driverkit https://developer.apple.com/documentation/driverkit
        • 12_throw_away 2y ago
          Well - recognition where it's due - that actually looks pretty great. (Assuming that, contrary to prior behavior, they actually support it, and fix bugs without breaking backwards compatibility every release, and don't keep swapping it out for newer frameworks, etc etc)
    • jimbokun 2y ago
      No. Go buy a different product if you want that functionality. I'm sticking with my Apple phone so outages like this are much less likely to affect me.
    • BobaFloutist 2y ago
      Ok what if they sold it off by default but there was a physical switch that could turn it on, that required hardware access? Good compromise?
      • School-Cotton 2y ago
        That’s exactly how macOS works (except it’s not a physical switch). You can disable SIP if you have hardware access to a machine.
  • brightlancer 2y ago
    > I know people really dislike how Apple restricts your freedom to use their software in any way they don't intend. But this is one of the times where they shine. Yes, the problem here is that the system owners had too much control over their systems. No, no, that's the EXACT OPPOSITE of what happened. The problem is Crowdstrike had too much control of systems -- arguing that we should instead give that control to Apple is just swapping out who's holding the gun.
    • maksimum 2y ago
      This is good nuance to add to the conversation, thanks. I think in most cases you have to trust some group of parties. As an individual you likely don't have enough time and expertise to fully validate everything that runs on your hardware. Do you trust the OSS community, hardware vendors, OS vendors like IBM, Apple, M$, do you trust third party vendors like Crowdstrike? For me, I prefer to minimize the number of parties I have to trust, and my trust is based on historical track record. I don't mind paying and giving up functionality.
      • __MatrixMan__ 2y ago
        Even if you've trusted too many people, and been burned, we should design our systems such that you can revoke that trust after the fact and become un-burned. Having to boot into safe mode and remove the file is a pretty clumsy remediation. Better would be to boot into some kind of trust-management interface and distrust cloudstrike updates dated after July 17, then rebuild your system accordingly (this wouldn't be difficult to implement with nix). Of course you can only benefit from that approach if you trust the end user a bit more than we typically do. Physical access should always be enough to access the trust management interface, anything else is just another vector for spooky action at a distance.
      • MinusGix 2y ago
        It is some mix of priorities along the frontier, with Apple being on the significantly controlling end such that I wouldn't want to bother. Your trust should also be based on prediction, and giving a major company even more control over what your systems are allowed to do has been historically bad and only gets worse. Even if Apple is properly ethical now (I'm skeptical, I think they've found a decently sized niche and that most of their users wouldn't drop them even if they moved to significantly higher levels of telemetry, due to being a status good in part), there's little reason to give them that power in perpetuity. Removing that control when it is absued hasn't gone well in the past.
  • adrr 2y ago
    I can't run a 10year old game on my Mac but i can run a 30 year old game on my windows 11 box. Microsoft prioritizes backwards compatibility for older software,
    • mulmen 2y ago
      If you are a Crowdstrike customer you can’t run anything today.
      • adrr 2y ago
        For apple you just need to be an apple customer, they do a good job on crashing computers with their OSX updates like Sonoma. I remember my first macbook pro retina couldn’t go to sleep because it wouldn’t wake up till apple decided to release a fix for it. Good thing they don’t make server OSes.
        • p_l 2y ago
          I remember fearing every OSX update because until they switched to just shipping read-only partition images you had considerable chance of hitting a bug in Installer.app that resulted in infinite loop... (the bug existed since ~10.6 until they switched to image-based updates...)
    • branko_d 2y ago
      You can’t run a 30 year old driver in Windows, nor a 10 year old in all likelihood. Microsoft prioritizes userspace compatibility, but their driver models have changed (relatively) frequently.
    • mdavidn 2y ago
      Maybe. Have you tried? 30 year old games often did not implement delta timing, so they advance ridiculously fast on modern processors. Or the games required a memory mode not supported by modern Windows (see real mode, expanded memory, protected mode), requiring DOSBox or other emulator to run today. DOSBox runs on Mac too, incidentally.
    • 2y ago
• donatj 2y ago
  Yes. I have been working in the tech industry since the early aughts and I never seen the industry so weak on engineer lead firms. Something really happened and the industry flipped.
• williamvds 2y ago
  Far from being douchey, I think you've hit the nail on the head. No one is perfect, we're all incompetent to some extent. You've written shitty code, I've definitely written shitty code. There's little time or consideration given to going back and improving things. Unless you're lucky enough to have financial support while working on a FOSS project where writing quality software is actually prioritized. I get the appeal software developers have to start from scratch and write their own kernel, or OS, etc. And then you realize that working with modern hardware is just as messy. We all stack our own house of cards upon another. Unless we tear it all down and start again with a sane stable structure, events like this will keep happening.
• l33t7332273 2y ago
  I know 100k+ engineers (some in security) that definitely should not be described as skilled.
  • 1over137 2y ago
    Wow, you know a lot of people!
• hnthrowaway0328 2y ago
  I think you are correct on that many SWEs are incompetent. I definitely am. I wish I had the time and passion to go through a complete self-training of CS fundamentals using Open Course resources.
  • ken47 2y ago
    You do realize that knowledge of CS fundamentals is extremely unlikely to have prevented this?
• deleted 2y ago
  [deleted]
• locococo 2y ago
  I don't know that incompetence is the best way to describe the forces at play but I agree with your sentiment. There is always tension between business people and engineering. Where the engineers want things to be perfect and safe, because we need to fix the arising issues during nights and weekends. The business people are interested in getting features released, and don't always understand the risks by pushing arbitrary dates. It's a tradeoff which in healthy organizations where the two sides and leadership communicate effectively is well managed.
  • sealeck 2y ago
    > Where the engineers want things to be perfect and safe, because we need to fix the arising issues during nights and weekends. The business people are interested in getting features released, and don't always understand the risks by pushing arbitrary dates. Isn't this issue a vindication of the engineering approach to management, where you try to _not_ brick thousands of computers because you wanted to meet some internal deadline faster?
  • carapace 2y ago
    > There is always tension between business people and engineering. Really? I think this situation (and the situation with Boeing!) shows that the tension is between ultimately between responsibility and irresponsibility. I cannot be said that this is a win for short-sighted and incompetent business people? If people don't understand the risks they shouldn't be making the decisions.
    • sealeck 2y ago
      I think this is especially true in businesses where the thing you are selling is literally your ability to do good engineering. In the case of Boeing the fundamental thing customers care about is the "goodness" of the actual plane (for example the quality, the value for money, etc). In the case of Crowdstrike people wanted high quality software to protect their computers.
      • carapace 2y ago
• yard2010 2y ago
  Pardon my snarkiness but this is what you get when imbeciles MBAs and marketers run every company, not engineers.
  • sbuk 2y ago
    Applying rigorous engineering principles is not something I see developers doing often. Whether or not it's incompetence on their part, or pressure from 'imbecile MBAs and marketers', it doesn't matter. They are software developers, not engineers. Engineers in most countries have to belong to a professional body and meet specific standards before they can practice as professionals. Any asshat can call themselves a 'software engineer', the current situation being a prime example, or was this a marketing decision?
    • isaacremuant 2y ago
      You're making the title be more than it is. This won't get solved by more certification. The checkbox of having certified security is what allowed it to happen in the first place.
      • sbuk 2y ago
        No. Engineering means something. This is a software ‘engineering’ problem. If the field wants the nomenclature, then it behooves them to apply rigour to who can call themselves an engineer or architect. Blaming middle management is missing the wood for the trees. The root cause was a bad patch. That is developments fault, and no one else’s. As to why this fault could happen, well the design of Windows should be scrutinised. Again, middle management isn’t really to blame here, software architects and engineers design the infrastructure, they choose to use Windows for a variety of reasons. The point here m trying to make is blaming “MBAs and marketing” shifts blame and misses the wood for the trees. The OP is as on the holier-than-thou “engineer” trip. They are not engineers.
        • morning-coffee 2y ago
          I think engineering only means something because of culture. It all starts from the culture of collective people who define and decide what principles are to be followed and why. All the certifications and licensing that are prerequsite to becoming an engineer are outcomes of the culture that defined them. Today we have pockets of code produced by one culture linked (literally) with pockets of code produced by a completely different ones and somehow expect the final result to adhere to the most principled and disciplined culture.
• MetaWhirledPeas 2y ago
  > I honestly believe there's A LOT of incompetence in the tech-world I can understand why. An engineer with expertise in one area can be a dunce in another; the line between concerns can be blurry; and expectations continue to change. Finding the right people with the right expertise is hard.
• jimbokun 2y ago
  > it looked like the engineers Engineers can be lazy and greedy, too. But at least they should better understand the risks of cutting corners. > Have professional integrity, tests is not optional or something that can be cut, it's part of SWE. Gradual rollout, feature-toggles, fall-backs/watchdogs etc. basic tools everyone should know. In my career, my solution for this has been to just include doing things "the right way" as part of the estimate, and not give management the option to select a "cutting corners" option. The "cutting corners" option not only adds more risk, but rarely saves time anyway when you inevitably have to manually roll things back or do it over.
  • 12_throw_away 2y ago
    Sigh, I've tried this. So management reassigned to a dev who was happy to ship a simalcrum of the thing that, at best, doesn't work or, at worst, is full of security holes and gives incorrect results. And this makes management happy because something shipped! Metrics go up! And then they ask why, exactly, did the senior engineer say this would take so long? Why always so difficult?
• dclowd9901 2y ago
  We’re definitely in a moment. I’ve seen a large shift away from discipline in the field. People don’t seem to care about professionalism or “good work”. I mean back in the mid teens we had the whole “move fast and break things” motif. I think that quickly morphed into “be agile” because no one actually felt good about breaking things. We don’t really have any software engineering leaders these days. It would be nice if one stood up and said “stop being awful. Let’s be professionals and earn our money.” Like, let’s create our own oath.
  • gedy 2y ago
    > We don’t really have any software engineering leaders these days. It would be nice if one stood up and said “stop being awful. Let’s be professionals and earn our money.” I assume you realize that you don't get very far in many companies when you do that. I'm not humble-bragging, but I used to say just this over past 10-15 years even when in senior/leadership positions, and it ended up giving me a reputation of "oh, gedy is difficult", and you get sidelined by more "helpful" junior devs and managers who are willing to sling shit over the wall to please product. It's really not worth it.
    • layer8 2y ago
      It’s a matter of getting a critical mass of people who do that. In other words, changing the general culture. I’m lucky to work at a company that more or less has that culture.
      • dclowd9901 2y ago
        Yeah I’ve found this is largely cultural, and it needs to come from the top. The best orgs have a gnarly, time-wisened engineer in a VP role who somehow is also a good people person, and pushes both up and down engineering quality above all else. It’s a very very rare combination.
    • matrix87 2y ago
      If it's a mature system and management is highly risk averse, not fucking up means more than slinging shit quickly
• ThrowawayR2 2y ago
  "Listen to the engineers?" The problem is that there are no engineers, in the proper sense of the term. What there are is tons and tons of software developers who are all too happy to be lax about security and safe designs for their own convenience and fight back hard against security analysts and QA when called out on it.
• citizen_friend 2y ago
  Nobody did gradual rollout in 1992
  • neffy 2y ago
    Not entirely true. The company I worked for, major network equipment provider, had a customer user group that had self-organised to take it in turns to be the first customer to deploy major new software builds. It mostly worked well.
• altdataseller 2y ago
  I would add that a lot of ppl in this industry also just blindly follow the herd too, without any independent thinking. Oh, everyone is using Crowdstrike? I guess i have to do so too! Oh, everyone is using Kubernetes? I guess i better start migrating our services to it too! Oh, everyone is using this fancy Vercel stuff? We better use it too! Oh everyone is migrating their workloads to the cloud even tho we dont need to and it costs 5x more?!! We better do so too!!
• nyarlathotep_ 2y ago
  > I also blame us, the SWE engineers, we are waay to easily busied around by these types of people who have no clue. Have professional integrity, tests is not optional or something that can be cut, it's part of SWE. Then maybe most of what's done in the "tech-industry" isn't, in any real sense, "engineering"? I'd argue the areas where there's actual "engineering" in software are the least discussed---example being hard real-time systems for Engine Control Units/ABS systems etc. That _has_ to work, unlike the latest CRUD/React thingy that had "engineering" processes of cargo-culting whatever framework is cool now and subjective nonsense like "code smells" and whatever design pattern is "needed" for "scale" or some such crap. Perhaps actual engineering approaches could be applied to software development at large, but it wouldn't look like what most programmers do, day to day, now. How is mission-critical software designed, tested, and QA'd? Why not try those approaches?
  • bobsomers 2y ago
    Amen to that. Software Engineering as a discipline badly suffers from not incorporating well-known methods for preventing these kinds of disasters from Systems Engineering. And when I say Systems Engineering I don't mean Systems Programming, I mean real Systems Engineering: https://en.wikipedia.org/wiki/Systems_engineering https://en.wikipedia.org/wiki/Systems_engineering > How is mission-critical software designed, tested, and QA'd? Why not try those approaches? Ultimately, because it is more expensive and slower to do things correctly, though I would argue that while you lose speed initially with activities like actually thinking through your requirements and your verification and validation strategies, you end up gaining speed later when you're iterating on a correct system implementation because you have established extremely valuable guardrails that keep you focused and on the right track. At the end of the day, the real failure is in the risk estimation of the damage done when these kinds of systems fail. We foolishly think that this kind of widespread disastrous failure is less likely than it really is, or the damage won't be as bad. If we accurately quantified that risk, many more systems we build would fall under the rigor of proper engineering practices.
  • jcgrillo 2y ago
    Accountability would drive this. Engineering liability codes are a thing, trade liability codes are a thing. If you do work that isn't up to code, and harm results, you're liable. Nobody is holding us software developers accountable, so it's no wonder these things continue to happen.
• rustcleaner 2y ago
  Just wait until the PE licensing requirements come to legally charge money for code.
• ken47 2y ago
  In most companies, businesspeople without any real software dev experience control the purse strings. Such people should never run companies that sell life-or-death software. The reality is there is plenty of space in the software industry to trade off velocity against "competent" software engineering. Take Instagram as an example. No one is going to die if e.g. a bug causes someone's IG photo upload to only appear in a proper subset of the feeds where it should appear. There's a lot of incompetence by choice.
  • marcodave 2y ago
    In the civil engineering world, at least in Europe, the lead engineer would sign papers that would put him as liable if a bridge or a building structure collapses on its own. The civil engineers face literal prison time if they make a sloppy work. In the software engineering world, we have TOSs that deny any liability if the software fails. Why? It boils my blood to think that the heads of CrowdStrike would maybe get a slap on the wrist and everything will slowly continue as usual as the machines will get fixed. People died for this bug.
    • ken47 2y ago
      We have those TOS's in the software world because it would be prohibitively expensive to make all software reliable as a publicly used bridge. For those who died as a direct result of CrowdStrike, that's where the litigious nature of the US becomes a rare plus. And CrowdStrike will lose a lot of customers over this. It isn't perfect, but the market will arbitrate CrowdStrike's future in the coming months and years.
    • peheje 2y ago
      Let's think about this for a second. I agree to some extend with what you are trying to say, I just think there's a critical thing missing here in your consideration, and that is usage of the product outside its intended purpose/marketing. Civil engineers built bridges knowingly that civilians use them, and structural failure can cause deaths. The line of responsibility is clear. SW companies (like CrowdStrike (CS)) it MAY BE less straight-forward. A relevant real-world example is the use of consumer drones in military conflicts. Companies like DJI design and market their drones for civilian use, such as photography. However, these drones have been repurposed in conflict zones, like Ukraine, to carry explosives. If such a drone malfunctioned during military use, it would be unreasonable to hold DJI accountable, as this usage clearly falls outside the product's intended purpose and marketing. The liability depends on the guarantees they make. If they market it for AV used for critical infrastructure, such as healthcare (seems like they do https://www.crowdstrike.com/platform/ https://www.crowdstrike.com/platform/) - by all means, it's reasonable to hold with accountable. However, SW companies should be able to sell products and long as they're clear what the limitations are, and it needs to be clearly communicated to the customers.
• cyanydeez 2y ago
  100% what we seen in the last couple of decades is the march of normies into the techno sphere to the detriment of the prior natives. We've essentially watched digital colonialism, and it certainly peaks with Elon musk wealth and ego, attempting to buy up the digital market place of ideas.