6 ms·

Nobody could have predicted this /s Joke aside, it's been pretty obvious since the beginning that security was an afterthought for most "AI" companies, with ev

Nobody could have predicted this /s Joke aside, it's been pretty obvious since the beginning that security was an afterthought for most "AI" companies, with even MCP adding secure features after the initial release.

brookst 1y ago
How does this compare to the way security was implemented by early websites, internet protocols, or telecom systems?
- jraph 1y ago
  Early stuff was designed in a network of trusty organizations (universities, labs...). Security wasn't much a concern but it was reasonable given the setting in which it was designed. This AI stuff? No excuse, it should have been designed with security and privacy in mind given the setting in which it's born. The conditions changed. The threat model is not the same. And this is well known. Security is hard, so there's some excuse, but it is reasonable to expect basic levels.
  - brookst 1y ago
    It’s really not. AI, like every other tech advance, was largely created by enthusiasts carried away with what could be done, not by top-down design that included all best practices. It’s frustrating to security people, but the reality is that security doesn’t become a design consideration until the tech has proven utility, which means there are always insecure implementations of early tech. Does it make any sense that payphones would give free calls for blowing a whistle into them? Obvious design flaw to treat the microphone the same as the generated control tones; it would have been trivial to design more secure control tones. But nobody saw the need until the tech was deployed at scale. It should be different, sure. But that’s just saying human nature “should” be different.
    - jraph 1y ago
      The payphones giving free calls was far less avoidable, virtually cost nothing to anybody and more importantly, didn't hurt anybody / threaten users' security. I don't buy into this "enthusiasts carried away" theory; Comet is developed by a company valued at 18 billion US dollars in July 2025 [1]. We are talking about a company that seriously considers buying Google Chrome for $34.5 billion. They had the money required for 1 person to think 5 minutes and see this prompt injection from page content from arbitrary internet places coming. That's as basic as the simplest SQL injection. I actually can't even imagine how they missed this. Maybe they didn't, and decided to not give a fuck and go ahead anyway. More generally I don't believe one second that all this tech is largely created by "enthusiasts carried away", without planning and design. You don't deal with multiple billion dollars this way. I will more gladly take "planned carelessness". Unless you are describing, by "enthusiasts carried away", the people out there that want to make quick money without giving any fuck to anything. > Perplexity AI has attracted legal scrutiny over allegations of copyright infringement, unauthorized content use, and trademark issues from several major media organizations, including the BBC, Dow Jones, and The New York Times. > In August 2025, Cloudflare published research finding that Perplexity was using undeclared "stealth" web crawlers to bypass Web application firewalls and robots.txt files intended to block Perplexity crawlers. Cloudflare's CEO Matthew Prince tweeted that Perplexity acts "more like North Korean hackers" than like a reputable AI company. Perplexity publicly denied the claims, calling it a "charlatan publicity stunt". Yeah… I see I blocked PerplexityBot in my nginx config because it was hammering my server. This industry just doesn't give one shit. They respect nobody. Screw them already. Tech is not blissful and innocent, and certainly not AI. Large scale tech like this is not done by some blissful / clueless dev in their garage, clueless and disconnected from reality. And this lone clueless dev in his garage phantasm actually needs to die. We need people thoughtful of consequences of what they do on other people and on the environment, there's really nothing desirable about someone who doesn't. [1] https://en.wikipedia.org/wiki/Perplexity_AI https://en.wikipedia.org/wiki/Perplexity_AI
      - queenkjuul 1y ago
        Just me, my soldering iron, my garage, and my $4B of Nvidia H100s
- SoftTalker 1y ago
  Must we learn the same lessons over and over again? Why? Is our industry particularly stupid? Or just lazy?
  - zahlman 1y ago
    Rather: it's perpetually in a rush for business reasons, and concerned with convenience. Security generally impedes both.
  - porridgeraisin 1y ago
    The winner (financially, and DAU-wise) is not going to be the one that moves slowly because they are building a secure product. That is, you only need security when you are big enough to either have Big Business customers or big enough to be the target of lawsuits.
  - evilduck 1y ago
    Financially motivated to not prioritize security. It's hard to sell what your product specifically can't do, while your competitors are spending their time building out what they can do. Beloved products can make a whole lot of serious mistakes before the public will actually turn on them.
    - SoftTalker 1y ago
      "Our bridges don't collapse" is a selling point for an engineering firm, on something that their products don't do. We need to stop calling ourselves engineers when we act like garage tinkerers. Or, we need to actually regulate software that can have devastating failure modes such as "emptying your bank account" so that companies selling software to the public (directly or indirectly) cannot externalize the costs of their software architecture decisions. Simply prohibiting disclaimer of liability in commercial software licenses might be enough.
      - brookst 1y ago
        Call yourself whatever you choose, but the garage tinkerers will always move faster and discover new markets before the Very Serious Engineers have completed the third review of the comprehensive threat model with all stakeholders.
- add-sub-mul-div 1y ago
  1. It's novel, meaning we have time to stop it before it becomes normalized. 2. It's a whole new category of threat vectors across all known/unknown quadarants. 3. Knowing what we know now vs. then, it's egregious and not naive, contextualizing how these companies operate and treat their customers. 4. There's a whole population of sophisticated predators ready to pounce instantly, they already have the knowledge and tools unlike in the 1990s. 5. Since it's novel, we need education and attention for this specifically. Should I go on? Can we finally put to bed the thought-limiting midwit take that AI's flaws and risks aren't worth discussion because past technology has had flaws and risks?
- ModernMech 1y ago
  Very poorly, because no matter how bad it was then, at least now we know better.
- phyzome 1y ago
  The way it compares is that we've had 30 years to learn from our mistakes (and apparently some of us have failed to).

brookst 1y ago

How does this compare to the way security was implemented by early websites, internet protocols, or telecom systems?

jraph 1y ago
Early stuff was designed in a network of trusty organizations (universities, labs...). Security wasn't much a concern but it was reasonable given the setting in which it was designed. This AI stuff? No excuse, it should have been designed with security and privacy in mind given the setting in which it's born. The conditions changed. The threat model is not the same. And this is well known. Security is hard, so there's some excuse, but it is reasonable to expect basic levels.
- brookst 1y ago
  It’s really not. AI, like every other tech advance, was largely created by enthusiasts carried away with what could be done, not by top-down design that included all best practices. It’s frustrating to security people, but the reality is that security doesn’t become a design consideration until the tech has proven utility, which means there are always insecure implementations of early tech. Does it make any sense that payphones would give free calls for blowing a whistle into them? Obvious design flaw to treat the microphone the same as the generated control tones; it would have been trivial to design more secure control tones. But nobody saw the need until the tech was deployed at scale. It should be different, sure. But that’s just saying human nature “should” be different.
  - jraph 1y ago
    The payphones giving free calls was far less avoidable, virtually cost nothing to anybody and more importantly, didn't hurt anybody / threaten users' security. I don't buy into this "enthusiasts carried away" theory; Comet is developed by a company valued at 18 billion US dollars in July 2025 [1]. We are talking about a company that seriously considers buying Google Chrome for $34.5 billion. They had the money required for 1 person to think 5 minutes and see this prompt injection from page content from arbitrary internet places coming. That's as basic as the simplest SQL injection. I actually can't even imagine how they missed this. Maybe they didn't, and decided to not give a fuck and go ahead anyway. More generally I don't believe one second that all this tech is largely created by "enthusiasts carried away", without planning and design. You don't deal with multiple billion dollars this way. I will more gladly take "planned carelessness". Unless you are describing, by "enthusiasts carried away", the people out there that want to make quick money without giving any fuck to anything. > Perplexity AI has attracted legal scrutiny over allegations of copyright infringement, unauthorized content use, and trademark issues from several major media organizations, including the BBC, Dow Jones, and The New York Times. > In August 2025, Cloudflare published research finding that Perplexity was using undeclared "stealth" web crawlers to bypass Web application firewalls and robots.txt files intended to block Perplexity crawlers. Cloudflare's CEO Matthew Prince tweeted that Perplexity acts "more like North Korean hackers" than like a reputable AI company. Perplexity publicly denied the claims, calling it a "charlatan publicity stunt". Yeah… I see I blocked PerplexityBot in my nginx config because it was hammering my server. This industry just doesn't give one shit. They respect nobody. Screw them already. Tech is not blissful and innocent, and certainly not AI. Large scale tech like this is not done by some blissful / clueless dev in their garage, clueless and disconnected from reality. And this lone clueless dev in his garage phantasm actually needs to die. We need people thoughtful of consequences of what they do on other people and on the environment, there's really nothing desirable about someone who doesn't. [1] https://en.wikipedia.org/wiki/Perplexity_AI https://en.wikipedia.org/wiki/Perplexity_AI
    - queenkjuul 1y ago
      Just me, my soldering iron, my garage, and my $4B of Nvidia H100s
SoftTalker 1y ago
Must we learn the same lessons over and over again? Why? Is our industry particularly stupid? Or just lazy?
- zahlman 1y ago
  Rather: it's perpetually in a rush for business reasons, and concerned with convenience. Security generally impedes both.
- porridgeraisin 1y ago
  The winner (financially, and DAU-wise) is not going to be the one that moves slowly because they are building a secure product. That is, you only need security when you are big enough to either have Big Business customers or big enough to be the target of lawsuits.
- evilduck 1y ago
  Financially motivated to not prioritize security. It's hard to sell what your product specifically can't do, while your competitors are spending their time building out what they can do. Beloved products can make a whole lot of serious mistakes before the public will actually turn on them.
  - SoftTalker 1y ago
    "Our bridges don't collapse" is a selling point for an engineering firm, on something that their products don't do. We need to stop calling ourselves engineers when we act like garage tinkerers. Or, we need to actually regulate software that can have devastating failure modes such as "emptying your bank account" so that companies selling software to the public (directly or indirectly) cannot externalize the costs of their software architecture decisions. Simply prohibiting disclaimer of liability in commercial software licenses might be enough.
    - brookst 1y ago
      Call yourself whatever you choose, but the garage tinkerers will always move faster and discover new markets before the Very Serious Engineers have completed the third review of the comprehensive threat model with all stakeholders.
add-sub-mul-div 1y ago
1. It's novel, meaning we have time to stop it before it becomes normalized. 2. It's a whole new category of threat vectors across all known/unknown quadarants. 3. Knowing what we know now vs. then, it's egregious and not naive, contextualizing how these companies operate and treat their customers. 4. There's a whole population of sophisticated predators ready to pounce instantly, they already have the knowledge and tools unlike in the 1990s. 5. Since it's novel, we need education and attention for this specifically. Should I go on? Can we finally put to bed the thought-limiting midwit take that AI's flaws and risks aren't worth discussion because past technology has had flaws and risks?
ModernMech 1y ago
Very poorly, because no matter how bad it was then, at least now we know better.
phyzome 1y ago
The way it compares is that we've had 30 years to learn from our mistakes (and apparently some of us have failed to).