10 ms·

The commits are revealing. Look at this one: > Ask Claude to remove the "backup" encryption key. Clearly it is still important to security-review Claude's cod

The commits are revealing. Look at this one: > Ask Claude to remove the "backup" encryption key. Clearly it is still important to security-review Claude's code! > prompt: I noticed you are storing a "backup" of the encryption key as `encryptionKeyJwk`. Doesn't this backup defeat the end-to-end encryption, because the key is available in the grant record without needing any token to unwrap it? I don’t think a non-expert would even know what this means, let alone spot the issue and direct the model to fix it.

bootsmann 1y ago
There is also one quite early in the repo where the dev has to tell Claude to store only the hashes of secrets
throwaway2037 1y ago
While I think this is a cool (public) experiment by Claude, asking an LLM to write security-sensitive code seems crazy at this point. Ad absurdum: Can you imagine asking Claude to implement new functionality in OpenSSL libs!?
october8140 1y ago
It's a Jr Developer that you have to check all their code over. To some people that is useful. But you're still going to have to train Jr Developers so they can turn into Sr Developers.
- PeterStuer 1y ago
  I don't like the jr dev analogy. It neither has the same weaknesses nor the same strenghts. It's more like the genious coworker that has an overassertive ego and sometimes shows up drunk, but if you know how to work with them and see past their flaws, can be a real asset.
  - hn_throwaway_99 1y ago
    I also like your analogy, but it also explains why I find working with AI-assisted coding so mentally tiresome. It's like with some auto-driving systems - I say it like having a slightly inebriated teenager at the wheel. I can't just relax and read a book, because then I'd die. But so I have to be more mentally alert than just driving myself because everything could be going smoothly and relaxed, but at any moment the driving system could decide to drive into a tree.
- Cthulhu_ 1y ago
  I don't really agree; a junior developer, if they're curious enough, wouldn't just write insecure code, they would do self-study and find out best practices etc before writing code, including not storing plaintext passwords and the like.
  - hn_throwaway_99 1y ago
    You have clearly only ever worked with the creme de la creme of junior developers.
i5heu 1y ago
Revealing against what? If you look at the README it is completely revealed... so i would argue there is nothing to "reveal" in the first place. > I started this project on a lark, fully expecting the AI to produce terrible code for me to laugh at. And then, uh... the code actually looked pretty good. Not perfect, but I just told the AI to fix things, and it did. I was shocked. > To emphasize, this is not "vibe coded". Every line was thoroughly reviewed and cross-referenced with relevant RFCs, by security experts with previous experience with those RFCs.
- risyachka 1y ago
  If the guy knew how to properly implement oauth - did he save any time though by prompting or just tried to prove a point that if you actually already know all details of impl you can guide llm to do it? Thats the biggest issue I see. In most cases I don't use llm because DIYing it takes less time than prompting/waiting/checking every line.
  - JimDabell 1y ago
    > did he save any time though Yes: > It took me a few days to build the library with AI. > I estimate it would have taken a few weeks, maybe months to write by hand. – https://news.ycombinator.com/item?id=44160208 https://news.ycombinator.com/item?id=44160208 > or just tried to prove a point that if you actually already know all details of impl you can guide llm to do it? No: > I was an AI skeptic. I thoughts LLMs were glorified Markov chain generators that didn't actually understand code and couldn't produce anything novel. I started this project on a lark, fully expecting the AI to produce terrible code for me to laugh at. And then, uh... the code actually looked pretty good. Not perfect, but I just told the AI to fix things, and it did. I was shocked. — https://github.com/cloudflare/workers-oauth-provider/?tab=readme-ov-file#written-using-claude https://github.com/cloudflare/workers-oauth-provider/?tab=re...
    - autoexec 1y ago
      > I thoughts LLMs were glorified Markov chain generators that didn't actually understand code and couldn't produce anything novel. How novel is a OAuth provider library for cloudflare workers? I wouldn't be surprised if it'd been trained on multiple examples.
PeterStuer 1y ago
Which is exactly why AI coding assistants work with your expertise rather than replace it. Most people I see fail at AI assisted development are either non-technical people expecting the AI will solve it all, or technical people playing gotcha with the machine rather than collaborating with it.
victorbjorklund 1y ago
That is how LLM:s should be used today. An expert prompts it and checks the code. Still saves a lot of time vs typing everything from scratch. Just the other day I was working on a prototype and let claude write code for a auth flow. Everything was good until the last step where it was just sending the user id as a string with the valid token. So if you got a valid token you could just pass in any user id and become that user. Still saved me a lot of time vs doing it from scratch.
- otabdeveloper4 1y ago
  > Still saves a lot of time vs typing everything from scratch No it doesn't. Typing speed is never the bottleneck for an expert. As an offline database of Google-tier knowledge, LLM's are useful. Though current LLM tech is half-baked, we need: a) Cheap commodity hardware for running your own models locally. (And by "locally" I mean separate dedicated devices, not something that fights over your desktop's or laptop's resources.) b) Standard bulletproof ways to fine-tune models on your own data. (Inference is already there mostly with things like llama.cpp, finetuning isn't.)
  - boruto 1y ago
    I realize I procrastinate less when using LLM to write code which I know I could write.
    - kentonv 1y ago
      I've noticed this too. I remember hearing somewhere that humans have a limited capacity in terms of number of decisions made in a day, and it seems to fit here: If I'm writing the code myself, I have to make several decisions on every line of code, and that's mentally tiring, so I tend to stop and procrastinate frequently. If an LLM is handling a lot of the details, then I'm just making higher-level decisions, allowing me to make more progress. Of course this is totally speculation and theories like this tend to be wrong, but it is at least consistent with how I feel.
      - 1y ago
deleted 1y ago
[deleted]
bananapub 1y ago
this seems like a true but pointless observation? if you're producing security-sensitive code then experts need to be involved, whether that's me unwisely getting a junior to do something, or receiving a PR from my cat, or using an LLM. removing expert humans from the loop is the deeply stupid thing the Tech Elite Who Want To Crush Their Own Workforces / former-NFT fanboys keep pushing, just letting an LLM generate code for a human to review then send out for more review is really pretty boring and already very effective for simple to medium-hard things.
- hn_throwaway_99 1y ago
  I think it's a critically important observation. I thought this experience was so helpful as it gave an objective, evidence-based sample on both the pros and cons of AI-assisted coding, where so many of the loudest voices on this topic are so one-sided ("AI is useless" or "developers will be obsolete in a year"). You say "removing expert humans from the loop is the deeply stupid thing the Tech Elite Who Want To Crush Their Own Workforces / former-NFT fanboys keep pushing", but the fact is many people with the power to push AI onto their workers are going to be more receptive to actual data and evidence than developers just complaining that AI is stupid.
- toofy 1y ago
  > …removing expert humans from the loop is the deeply stupid thing the Tech Elite Who Want To Crush Their Own Workforce… this is completely expected behavior by them. departments with well paid experts will be one of the first they’ll want to cut. in every field. experts cost money. we’re a long, long, long way off from a bot that can go into random houses and fix under the sink plumbing, or diagnose and then fix an electrical socket. however, those who do most of their work on a computer, they’re pretty close to a point where they can cut these departments. in every industry in every field, those will be jobs cut first. move fast and break things.
kentonv 1y ago
Yeah I was disappointed in that one. I hate to say, though, but I have reviewed a lot of human code in my time, and I've definitely caught many humans making similar-magnitude mistakes. :/
- hn_throwaway_99 1y ago
  I just wanted to say thanks so much publishing this, and especially your comments here - I found them really helpful and insightful. I think it's interesting (though not unexpected) that many of the other commenters' comments here show what a Rorschach test this is. I think that's kind of unfortunate, because your experience clearly showed some of the benefits and limitations/pitfalls of coding like this in an objective manner. I am curious, did you find the work of reviewing Claude's output more mentally tiring/draining than writing it yourself? Like some other folks mentioned, I generally find reviewing code more mentally tiring than writing it, but I get a lot of personal satisfaction by mentoring junior developers and collaborating with my (human) colleagues (most of them anyway...) Since I don't get that feeling when reviewing AI code, I find it more draining. I'm curious how you felt reviewing this code.
  - kentonv 1y ago
    I find reviewing AI code less mentally tiring that reviewing human code. This was a surprise to me! Until I tried it, I dreaded the idea. I think it is because of the shorter feedback loop. I look at what the AI writes as it is writing it, and can ask for changes which it applies immediately. Reviewing human code typically has hours or days of round-trip time. Also with the AI code I can just take over if it's not doing the right thing. Humans don't like it when I start pushing commits directly to their PR. There's also the fact that the AI I'm prompting is, obviously, working on my priorities, whereas humans are often working on other priorities, but I can't just decline to review someone's code because it's not what I'm personally interested in at that moment. When things go well, reviewing the AI's work is less draining than writing it myself, because it's basically doing the busy work while I'm still in control of high-level direction and architecture. I like that. But things don't always go well. Sometimes the AI goes in totally the wrong direction, and I have to prompt it too many times to do what I want, in which case it's not saving me time. But again, I can always just cancel the session and start doing it myself... humans don't like it when I tell them to drop a PR and let me do it. Personally, I don't generally get excited about mentoring and collaborating. I wish I did, and I recognize it's an important part of my job which I have to do either way, but I just don't. I get excited primarily about ideas and architecture and not so much about people.
ActionHank 1y ago
But AIbros will be running around telling everyone that Claude invented OAuth for Cloudflare all on its own and then opensourced it.
jofzar 1y ago
I know I'm preaching to the masses here, but isn't this why PR are so important?

bootsmann 1y ago

There is also one quite early in the repo where the dev has to tell Claude to store only the hashes of secrets

throwaway2037 1y ago

While I think this is a cool (public) experiment by Claude, asking an LLM to write security-sensitive code seems crazy at this point. Ad absurdum: Can you imagine asking Claude to implement new functionality in OpenSSL libs!?

october8140 1y ago

It's a Jr Developer that you have to check all their code over. To some people that is useful. But you're still going to have to train Jr Developers so they can turn into Sr Developers.

PeterStuer 1y ago
I don't like the jr dev analogy. It neither has the same weaknesses nor the same strenghts. It's more like the genious coworker that has an overassertive ego and sometimes shows up drunk, but if you know how to work with them and see past their flaws, can be a real asset.
- hn_throwaway_99 1y ago
  I also like your analogy, but it also explains why I find working with AI-assisted coding so mentally tiresome. It's like with some auto-driving systems - I say it like having a slightly inebriated teenager at the wheel. I can't just relax and read a book, because then I'd die. But so I have to be more mentally alert than just driving myself because everything could be going smoothly and relaxed, but at any moment the driving system could decide to drive into a tree.
Cthulhu_ 1y ago
I don't really agree; a junior developer, if they're curious enough, wouldn't just write insecure code, they would do self-study and find out best practices etc before writing code, including not storing plaintext passwords and the like.
- hn_throwaway_99 1y ago
  You have clearly only ever worked with the creme de la creme of junior developers.

i5heu 1y ago

Revealing against what? If you look at the README it is completely revealed... so i would argue there is nothing to "reveal" in the first place. > I started this project on a lark, fully expecting the AI to produce terrible code for me to laugh at. And then, uh... the code actually looked pretty good. Not perfect, but I just told the AI to fix things, and it did. I was shocked. > To emphasize, this is not "vibe coded". Every line was thoroughly reviewed and cross-referenced with relevant RFCs, by security experts with previous experience with those RFCs.

risyachka 1y ago
If the guy knew how to properly implement oauth - did he save any time though by prompting or just tried to prove a point that if you actually already know all details of impl you can guide llm to do it? Thats the biggest issue I see. In most cases I don't use llm because DIYing it takes less time than prompting/waiting/checking every line.
- JimDabell 1y ago
  > did he save any time though Yes: > It took me a few days to build the library with AI. > I estimate it would have taken a few weeks, maybe months to write by hand. – https://news.ycombinator.com/item?id=44160208 https://news.ycombinator.com/item?id=44160208 > or just tried to prove a point that if you actually already know all details of impl you can guide llm to do it? No: > I was an AI skeptic. I thoughts LLMs were glorified Markov chain generators that didn't actually understand code and couldn't produce anything novel. I started this project on a lark, fully expecting the AI to produce terrible code for me to laugh at. And then, uh... the code actually looked pretty good. Not perfect, but I just told the AI to fix things, and it did. I was shocked. — https://github.com/cloudflare/workers-oauth-provider/?tab=readme-ov-file#written-using-claude https://github.com/cloudflare/workers-oauth-provider/?tab=re...
  - autoexec 1y ago
    > I thoughts LLMs were glorified Markov chain generators that didn't actually understand code and couldn't produce anything novel. How novel is a OAuth provider library for cloudflare workers? I wouldn't be surprised if it'd been trained on multiple examples.

PeterStuer 1y ago

Which is exactly why AI coding assistants work with your expertise rather than replace it. Most people I see fail at AI assisted development are either non-technical people expecting the AI will solve it all, or technical people playing gotcha with the machine rather than collaborating with it.

victorbjorklund 1y ago

That is how LLM:s should be used today. An expert prompts it and checks the code. Still saves a lot of time vs typing everything from scratch. Just the other day I was working on a prototype and let claude write code for a auth flow. Everything was good until the last step where it was just sending the user id as a string with the valid token. So if you got a valid token you could just pass in any user id and become that user. Still saved me a lot of time vs doing it from scratch.

otabdeveloper4 1y ago
> Still saves a lot of time vs typing everything from scratch No it doesn't. Typing speed is never the bottleneck for an expert. As an offline database of Google-tier knowledge, LLM's are useful. Though current LLM tech is half-baked, we need: a) Cheap commodity hardware for running your own models locally. (And by "locally" I mean separate dedicated devices, not something that fights over your desktop's or laptop's resources.) b) Standard bulletproof ways to fine-tune models on your own data. (Inference is already there mostly with things like llama.cpp, finetuning isn't.)
- boruto 1y ago
  I realize I procrastinate less when using LLM to write code which I know I could write.
  - kentonv 1y ago
    I've noticed this too. I remember hearing somewhere that humans have a limited capacity in terms of number of decisions made in a day, and it seems to fit here: If I'm writing the code myself, I have to make several decisions on every line of code, and that's mentally tiring, so I tend to stop and procrastinate frequently. If an LLM is handling a lot of the details, then I'm just making higher-level decisions, allowing me to make more progress. Of course this is totally speculation and theories like this tend to be wrong, but it is at least consistent with how I feel.
    - 1y ago

deleted 1y ago

[deleted]

bananapub 1y ago

this seems like a true but pointless observation? if you're producing security-sensitive code then experts need to be involved, whether that's me unwisely getting a junior to do something, or receiving a PR from my cat, or using an LLM. removing expert humans from the loop is the deeply stupid thing the Tech Elite Who Want To Crush Their Own Workforces / former-NFT fanboys keep pushing, just letting an LLM generate code for a human to review then send out for more review is really pretty boring and already very effective for simple to medium-hard things.

hn_throwaway_99 1y ago
I think it's a critically important observation. I thought this experience was so helpful as it gave an objective, evidence-based sample on both the pros and cons of AI-assisted coding, where so many of the loudest voices on this topic are so one-sided ("AI is useless" or "developers will be obsolete in a year"). You say "removing expert humans from the loop is the deeply stupid thing the Tech Elite Who Want To Crush Their Own Workforces / former-NFT fanboys keep pushing", but the fact is many people with the power to push AI onto their workers are going to be more receptive to actual data and evidence than developers just complaining that AI is stupid.
toofy 1y ago
> …removing expert humans from the loop is the deeply stupid thing the Tech Elite Who Want To Crush Their Own Workforce… this is completely expected behavior by them. departments with well paid experts will be one of the first they’ll want to cut. in every field. experts cost money. we’re a long, long, long way off from a bot that can go into random houses and fix under the sink plumbing, or diagnose and then fix an electrical socket. however, those who do most of their work on a computer, they’re pretty close to a point where they can cut these departments. in every industry in every field, those will be jobs cut first. move fast and break things.

kentonv 1y ago

Yeah I was disappointed in that one. I hate to say, though, but I have reviewed a lot of human code in my time, and I've definitely caught many humans making similar-magnitude mistakes. :/

hn_throwaway_99 1y ago
I just wanted to say thanks so much publishing this, and especially your comments here - I found them really helpful and insightful. I think it's interesting (though not unexpected) that many of the other commenters' comments here show what a Rorschach test this is. I think that's kind of unfortunate, because your experience clearly showed some of the benefits and limitations/pitfalls of coding like this in an objective manner. I am curious, did you find the work of reviewing Claude's output more mentally tiring/draining than writing it yourself? Like some other folks mentioned, I generally find reviewing code more mentally tiring than writing it, but I get a lot of personal satisfaction by mentoring junior developers and collaborating with my (human) colleagues (most of them anyway...) Since I don't get that feeling when reviewing AI code, I find it more draining. I'm curious how you felt reviewing this code.
- kentonv 1y ago
  I find reviewing AI code less mentally tiring that reviewing human code. This was a surprise to me! Until I tried it, I dreaded the idea. I think it is because of the shorter feedback loop. I look at what the AI writes as it is writing it, and can ask for changes which it applies immediately. Reviewing human code typically has hours or days of round-trip time. Also with the AI code I can just take over if it's not doing the right thing. Humans don't like it when I start pushing commits directly to their PR. There's also the fact that the AI I'm prompting is, obviously, working on my priorities, whereas humans are often working on other priorities, but I can't just decline to review someone's code because it's not what I'm personally interested in at that moment. When things go well, reviewing the AI's work is less draining than writing it myself, because it's basically doing the busy work while I'm still in control of high-level direction and architecture. I like that. But things don't always go well. Sometimes the AI goes in totally the wrong direction, and I have to prompt it too many times to do what I want, in which case it's not saving me time. But again, I can always just cancel the session and start doing it myself... humans don't like it when I tell them to drop a PR and let me do it. Personally, I don't generally get excited about mentoring and collaborating. I wish I did, and I recognize it's an important part of my job which I have to do either way, but I just don't. I get excited primarily about ideas and architecture and not so much about people.

ActionHank 1y ago

But AIbros will be running around telling everyone that Claude invented OAuth for Cloudflare all on its own and then opensourced it.

jofzar 1y ago

I know I'm preaching to the masses here, but isn't this why PR are so important?