6 ms·

ChatGPT writing a blog post attacking Gemini security flaws. It's their world now, we're just watching how it plays out.

bryanrasmussen 8mo ago
How do you know that this blog post was written by ChatGPT?
- SecretDreams 8mo ago
  It's too structured and consistent. Imo. Has that AI smell to it, but I guess humans will eventually also start writing more like the AIs they learn from.
  - devsda 8mo ago
    > guess humans will eventually also start writing more like the AIs they learn from. With the AI feedback loop being so fast and tight for some tasks, the focus moves on to delivery than learning. There is no incentive, space or time for learning.
    - bpodgursky 8mo ago
      Won't be well received here, but this is the truth.
    - OakNinja 8mo ago
      For me personally, both at work and in my free time, I spend _more_ time on writing things _that matter_ since I’ve freed up time by using LLM’s for boilerplate tasks. My motto is - If it wasn’t worth writing, it won’t be worth reading. A good example of writing where I’d recommend using LLM’s is product documentation. You pass the diff, the description of the task, and the context (existing documentation) with a prompt ”Update the documentation…”. Documentation is important but it’s not prose. However, writing a comment on hacker news is.
  - Hnrobert42 8mo ago
    AI was trained on human writing.
    - SecretDreams 8mo ago
      And now humans are trained on AI writing. Like what happens to YouTube videos that go through the compression algorithm 20 times.
    - palmotea 8mo ago
      > AI was trained on human writing. AI output is not varied like real human writing. This is a very distinctive narrowing of style.
  - Dylan16807 8mo ago
    This is the first time I've seen people accuse AI text of being "too structured and consistent" compared to human text. Usually it's about specific patterns or tons of repetition or outright mistakes.
    - SecretDreams 8mo ago
      Patterns = consistent?
      - Dylan16807 8mo ago
        Patterns like heavy use of certain words or dashes or bullet points don't change how consistent the overall post is.
    - roywiggins 8mo ago
      One example of being "too structured" is that LLMs love an explicit introduction and conclusion even when one that isn't really warranted. It's always telling you what it's going to say, and what it just said.
- bpodgursky 8mo ago
  > The Core Problem > What You Should Do Right Now > Bonus: Scan with TruffleHog. > TruffleHog will verify whether discovered keys are live and have Gemini access, so you'll know exactly which keys are exposed and active, not just which ones match a regular expression. I don't know exactly, but I'm sure. The cadence, the clarity, the bolding, the italics, it's all just crisp and clean structured and actionable in a way that a meandering human would not distill it down to.
  - cyral 8mo ago
    Yup, it was actually an interesting article but there are a few telltale parts that sound like every AI spam post on /r/webdev and similar. "No warning. No confirmation dialog. No email notification." is another. The three negatives repeated is present in so many AI generated promotional posts.
    - bpodgursky 8mo ago
      I don't even have a problem with the content itself, I think frankly the smell is that it's too good. It's just fascinating in the sense that it's one LLM attacking another LLM.
  - roywiggins 8mo ago
    I've reached the point where if any blog post has a subheading with some variant of "The Problem", I assume it's been edited with an LLM, because it co-locates with other indicators so strongly.
- solid_fuel 8mo ago
  It feels generated to me too. It’s this: When you enable the Gemini API (Generative Language API) on a Google Cloud project, existing API keys in that project (including the ones sitting in public JavaScript on your website) can silently gain access to sensitive Gemini endpoints. No warning. No confirmation dialog. No email notification. Specifically, the last bit - “No warning. No confirmation dialog. No email notification.” Immediately smells like LLM generated text to me. Punchy repetition in a set of 3. If you scroll through tiktok or instagram you can see the same exact pattern in a lot of LLM generated descriptions.
  - larusso 8mo ago
    I’m not a native speaker so my level of AI recognition is already low. I find it very interesting what patters people bring up to declare it’s AI. The 3 punchline one for instance is a pattern I use while speaking. Can’t say I would write like this though.
    - solid_fuel 8mo ago
      It's not so much the grouping of 3 or way it's supposed to be punchy specifically that's the problem, that is just one example of what gives the article the "LLM Generated" feeling since whatever cheap model people are using for this kind of spam has some common ticks. I use groupings of 3 and try to make things punchy myself sometimes, especially when I'm writing something intended to sway others. I think the problem with this article is the way it feels like the perfect average of corporate writing. It's sort of like the "written by committee" feel that incredibly generic pop music often has. When I write things, I often go back and edit and reword parts. Like the brushstrokes in an oil painting, the flow of thought varies between paragraphs and even sentences. LLMs only generate things from left to right (or vice versa in RTL languages, I presume). I think that gives LLM generated text a "smooth" texture that really stands out to anyone who reads a lot.
      - nimonian 8mo ago
        I completely agree with you. There's something conspicuous about this particular use of the "group of three" device. It's trying but it's goofy and conspicuous. I think it's not human, it's 52 trillion parameters in a trenchcoat.
- jibal 8mo ago
  They don't. Many of these claims are due to illiteracy. Someone is complaining that > it's all just crisp and clean structured and actionable in a way that a meandering human would not distill it down to. but this is a security report ... people intentionally write such things carefully and crisply with multiple edits and reviews.
- raincole 8mo ago
  It's too well structured and the message is too clear. HN (and the whole internet) is allergic to proper writing. We praise human sloppiness now. No, I'm not being sarcastic. People have given up em-dash, which is an official punctuation you use in proper writing. And it's all a downhill from there.
  - palmotea 8mo ago
    > It's too well structured and the message is too clean. HN (and the whole internet) is allergic to proper writing. We praise human sloppiness now. Yes. And it's only a matter of time that the model companies start to try to train in that "human sloppiness." After all, a lot of their customers want machines that can pass for humans. > No, I'm not being sarcastic. People have given up em-dash, which is an official punctuation you use in proper writing. And it's all a downhill from there. I wouldn't be surprised if the internet language of people devolves into a weird constantly-changing mish-mash of slang and linguistic fads. Basically an arms race where people constantly innovate in order to stay distinct from the latest models. But the end result of that would be probably fragmentation, isolation, and a kind of dark ages. Different communities would have different slang, and that slang would change so fast that old text would quickly become hard to understand.
  - oasisbob 8mo ago
    Strongly disagree. The post is really poorly structured and circles the drain a few times getting to the thesis. The issues of style are annoying, but I find it much worse to wade through these 3000 word posts which are far longer than they need to be just because they're so damn cheap to compose.
- deaux 8mo ago
  The fact that according to this reply section most of HN can't tell means that predictably, all hope is lost and there's no point in writing anything by hand any more if you're in it for money/engagement. While writing this I suddenly realized that marketers and writers probably do a better job at recognizing it than developers and engineers, so maybe all hope isn't. For those who want to know the tells: overall cadence and frequency of patterns - especially infrequency of patterns - are the biggest ones. And that means that we can't actually give you the best tells, because they're more about what is absent than what is present. What's absent is a single sentence pattern that falls completely out of the LLM go-toes. Anything human written has at most a good mix of both. LLM-written text just entirely lacks it. Humans do use the LLM-preferred patterns, but not for every single sentence. But anyway, here we go. > Transparently, the initial triage was frustrating; the report was dismissed as "Intended Behavior”. But after providing concrete evidence from Google's own infrastructure, the GCP VDP team took the issue seriously. ^ Fun fact - The ";" would've originally been an em-dash but was either rewritten or a rule was included for this. > Then Gemini arrived. ^ Dramatic short sentences, a pattern with magnitudes higher LLM-frequency than human frequency, but hasn't reached the public conscious yet a la "not just X but Y". > No warning. No confirmation dialog. No email notification. ^ Another such pattern. Not just because it's three of them, but also because of the content and repetition. Humans rarely write like that because it again sounds overly dramatic. It's something you see in fiction rather than a technical writeup. In a thriller. > Retroactive Privilege Expansion. You created a Maps key three years ago and embedded it in your website's source code, exactly as Google instructed. Last month, a developer on your team enabled the Gemini API for an internal prototype. Your public Maps key is now a Gemini credential. Anyone who scrapes it can access your uploaded files, cached content, and rack up your AI bill. Nobody told you. This style of scenario writing is another one. > Nobody told you. Absolute drama queen. >The UI shows a warning about "unauthorized use," but the architectural default is wide open. Again. > The attacker never touches your infrastructure. They just scrape a key from a public webpage. Again. > These aren't just hobbyist side projects. The victims included major financial institutions, security companies, global recruiting firms, and, notably, Google itself. .. > A key that was deployed years ago for a completely benign purpose had silently gained full access to a sensitive API without any developer intervention. Surprised it hasn't gained consciousness by now. Maybe that's a future plot point. Here's a great example to train your skills on, because it's rare in that the ratio of "human : straight from LLM" increased gradually as the article goes on: https://www.wallstreetraider.com/story.html https://www.wallstreetraider.com/story.html It started at heavy human editing (or just human-written), but less and less towards the end. The author confirmed this upon pointing it out, FWIW [0]. [0] https://news.ycombinator.com/item?id=47013150 https://news.ycombinator.com/item?id=47013150
- deleted 8mo ago
  [deleted]
- oasisbob 8mo ago
  It's far longer than it needs to be because the writing process was too cheap.
raesene9 8mo ago
They may have used ChatGPT or similar to help with the prose but the technical content (as discussed elsewhere on this page) is good, so does it really matter if they did? The problem with AI slop (to me) is more that the technical content is not good or is entirely the product of the LLM. At that point, there's no point in me reading it, I can just prompt the question if I'm interested. This is original research which wasn't public before, so the value is still there and I didn't think whichever combination of a human and LLM that generated it did a bad job.

bryanrasmussen 8mo ago

How do you know that this blog post was written by ChatGPT?

SecretDreams 8mo ago
It's too structured and consistent. Imo. Has that AI smell to it, but I guess humans will eventually also start writing more like the AIs they learn from.
- devsda 8mo ago
  > guess humans will eventually also start writing more like the AIs they learn from. With the AI feedback loop being so fast and tight for some tasks, the focus moves on to delivery than learning. There is no incentive, space or time for learning.
  - bpodgursky 8mo ago
    Won't be well received here, but this is the truth.
  - OakNinja 8mo ago
    For me personally, both at work and in my free time, I spend _more_ time on writing things _that matter_ since I’ve freed up time by using LLM’s for boilerplate tasks. My motto is - If it wasn’t worth writing, it won’t be worth reading. A good example of writing where I’d recommend using LLM’s is product documentation. You pass the diff, the description of the task, and the context (existing documentation) with a prompt ”Update the documentation…”. Documentation is important but it’s not prose. However, writing a comment on hacker news is.
- Hnrobert42 8mo ago
  AI was trained on human writing.
  - SecretDreams 8mo ago
    And now humans are trained on AI writing. Like what happens to YouTube videos that go through the compression algorithm 20 times.
  - palmotea 8mo ago
    > AI was trained on human writing. AI output is not varied like real human writing. This is a very distinctive narrowing of style.
- Dylan16807 8mo ago
  This is the first time I've seen people accuse AI text of being "too structured and consistent" compared to human text. Usually it's about specific patterns or tons of repetition or outright mistakes.
  - SecretDreams 8mo ago
    Patterns = consistent?
    - Dylan16807 8mo ago
      Patterns like heavy use of certain words or dashes or bullet points don't change how consistent the overall post is.
  - roywiggins 8mo ago
    One example of being "too structured" is that LLMs love an explicit introduction and conclusion even when one that isn't really warranted. It's always telling you what it's going to say, and what it just said.
bpodgursky 8mo ago
> The Core Problem > What You Should Do Right Now > Bonus: Scan with TruffleHog. > TruffleHog will verify whether discovered keys are live and have Gemini access, so you'll know exactly which keys are exposed and active, not just which ones match a regular expression. I don't know exactly, but I'm sure. The cadence, the clarity, the bolding, the italics, it's all just crisp and clean structured and actionable in a way that a meandering human would not distill it down to.
- cyral 8mo ago
  Yup, it was actually an interesting article but there are a few telltale parts that sound like every AI spam post on /r/webdev and similar. "No warning. No confirmation dialog. No email notification." is another. The three negatives repeated is present in so many AI generated promotional posts.
  - bpodgursky 8mo ago
    I don't even have a problem with the content itself, I think frankly the smell is that it's too good. It's just fascinating in the sense that it's one LLM attacking another LLM.
- roywiggins 8mo ago
  I've reached the point where if any blog post has a subheading with some variant of "The Problem", I assume it's been edited with an LLM, because it co-locates with other indicators so strongly.
solid_fuel 8mo ago
It feels generated to me too. It’s this: When you enable the Gemini API (Generative Language API) on a Google Cloud project, existing API keys in that project (including the ones sitting in public JavaScript on your website) can silently gain access to sensitive Gemini endpoints. No warning. No confirmation dialog. No email notification. Specifically, the last bit - “No warning. No confirmation dialog. No email notification.” Immediately smells like LLM generated text to me. Punchy repetition in a set of 3. If you scroll through tiktok or instagram you can see the same exact pattern in a lot of LLM generated descriptions.
- larusso 8mo ago
  I’m not a native speaker so my level of AI recognition is already low. I find it very interesting what patters people bring up to declare it’s AI. The 3 punchline one for instance is a pattern I use while speaking. Can’t say I would write like this though.
  - solid_fuel 8mo ago
    It's not so much the grouping of 3 or way it's supposed to be punchy specifically that's the problem, that is just one example of what gives the article the "LLM Generated" feeling since whatever cheap model people are using for this kind of spam has some common ticks. I use groupings of 3 and try to make things punchy myself sometimes, especially when I'm writing something intended to sway others. I think the problem with this article is the way it feels like the perfect average of corporate writing. It's sort of like the "written by committee" feel that incredibly generic pop music often has. When I write things, I often go back and edit and reword parts. Like the brushstrokes in an oil painting, the flow of thought varies between paragraphs and even sentences. LLMs only generate things from left to right (or vice versa in RTL languages, I presume). I think that gives LLM generated text a "smooth" texture that really stands out to anyone who reads a lot.
    - nimonian 8mo ago
      I completely agree with you. There's something conspicuous about this particular use of the "group of three" device. It's trying but it's goofy and conspicuous. I think it's not human, it's 52 trillion parameters in a trenchcoat.
jibal 8mo ago
They don't. Many of these claims are due to illiteracy. Someone is complaining that > it's all just crisp and clean structured and actionable in a way that a meandering human would not distill it down to. but this is a security report ... people intentionally write such things carefully and crisply with multiple edits and reviews.
raincole 8mo ago
It's too well structured and the message is too clear. HN (and the whole internet) is allergic to proper writing. We praise human sloppiness now. No, I'm not being sarcastic. People have given up em-dash, which is an official punctuation you use in proper writing. And it's all a downhill from there.
- palmotea 8mo ago
  > It's too well structured and the message is too clean. HN (and the whole internet) is allergic to proper writing. We praise human sloppiness now. Yes. And it's only a matter of time that the model companies start to try to train in that "human sloppiness." After all, a lot of their customers want machines that can pass for humans. > No, I'm not being sarcastic. People have given up em-dash, which is an official punctuation you use in proper writing. And it's all a downhill from there. I wouldn't be surprised if the internet language of people devolves into a weird constantly-changing mish-mash of slang and linguistic fads. Basically an arms race where people constantly innovate in order to stay distinct from the latest models. But the end result of that would be probably fragmentation, isolation, and a kind of dark ages. Different communities would have different slang, and that slang would change so fast that old text would quickly become hard to understand.
- oasisbob 8mo ago
  Strongly disagree. The post is really poorly structured and circles the drain a few times getting to the thesis. The issues of style are annoying, but I find it much worse to wade through these 3000 word posts which are far longer than they need to be just because they're so damn cheap to compose.
deaux 8mo ago
The fact that according to this reply section most of HN can't tell means that predictably, all hope is lost and there's no point in writing anything by hand any more if you're in it for money/engagement. While writing this I suddenly realized that marketers and writers probably do a better job at recognizing it than developers and engineers, so maybe all hope isn't. For those who want to know the tells: overall cadence and frequency of patterns - especially infrequency of patterns - are the biggest ones. And that means that we can't actually give you the best tells, because they're more about what is absent than what is present. What's absent is a single sentence pattern that falls completely out of the LLM go-toes. Anything human written has at most a good mix of both. LLM-written text just entirely lacks it. Humans do use the LLM-preferred patterns, but not for every single sentence. But anyway, here we go. > Transparently, the initial triage was frustrating; the report was dismissed as "Intended Behavior”. But after providing concrete evidence from Google's own infrastructure, the GCP VDP team took the issue seriously. ^ Fun fact - The ";" would've originally been an em-dash but was either rewritten or a rule was included for this. > Then Gemini arrived. ^ Dramatic short sentences, a pattern with magnitudes higher LLM-frequency than human frequency, but hasn't reached the public conscious yet a la "not just X but Y". > No warning. No confirmation dialog. No email notification. ^ Another such pattern. Not just because it's three of them, but also because of the content and repetition. Humans rarely write like that because it again sounds overly dramatic. It's something you see in fiction rather than a technical writeup. In a thriller. > Retroactive Privilege Expansion. You created a Maps key three years ago and embedded it in your website's source code, exactly as Google instructed. Last month, a developer on your team enabled the Gemini API for an internal prototype. Your public Maps key is now a Gemini credential. Anyone who scrapes it can access your uploaded files, cached content, and rack up your AI bill. Nobody told you. This style of scenario writing is another one. > Nobody told you. Absolute drama queen. >The UI shows a warning about "unauthorized use," but the architectural default is wide open. Again. > The attacker never touches your infrastructure. They just scrape a key from a public webpage. Again. > These aren't just hobbyist side projects. The victims included major financial institutions, security companies, global recruiting firms, and, notably, Google itself. .. > A key that was deployed years ago for a completely benign purpose had silently gained full access to a sensitive API without any developer intervention. Surprised it hasn't gained consciousness by now. Maybe that's a future plot point. Here's a great example to train your skills on, because it's rare in that the ratio of "human : straight from LLM" increased gradually as the article goes on: https://www.wallstreetraider.com/story.html https://www.wallstreetraider.com/story.html It started at heavy human editing (or just human-written), but less and less towards the end. The author confirmed this upon pointing it out, FWIW [0]. [0] https://news.ycombinator.com/item?id=47013150 https://news.ycombinator.com/item?id=47013150
deleted 8mo ago
[deleted]
oasisbob 8mo ago
It's far longer than it needs to be because the writing process was too cheap.

raesene9 8mo ago

They may have used ChatGPT or similar to help with the prose but the technical content (as discussed elsewhere on this page) is good, so does it really matter if they did? The problem with AI slop (to me) is more that the technical content is not good or is entirely the product of the LLM. At that point, there's no point in me reading it, I can just prompt the question if I'm interested. This is original research which wasn't public before, so the value is still there and I didn't think whichever combination of a human and LLM that generated it did a bad job.