4 ms·

Agents just eat these things. After last week's HN post about Cyphral Distich, I pointed Astra and Fable at some unsolved ciphers just to see whether some joker

Agents just eat these things. After last week's HN post about Cyphral Distich, I pointed Astra and Fable at some unsolved ciphers just to see whether some joker who knew nothing about the field could get the same results, and sure enough there's plenty of low hanging fruit. https://aaymeloglu.github.io/unsolved-ciphers/ https://aaymeloglu.github.io/unsolved-ciphers/ But I got nothing on Daniel Bordeau, who in the past week seems to have built himself a whole code breaking factory! https://dbourdeau.github.io/cyphersolver/index.html https://dbourdeau.github.io/cyphersolver/index.html

deleted 14d ago
[deleted]
93po 14d ago
Mildly interesting anecdote: when the Cyphral Distich solution popped up a few days ago, I spent about an hour with ChatGPT trying to solve it myself without looking at the proposed solution. ChatGPT opened by saying “the solution is disputed online,” and made the dispute sound fairly convincing, which struck me as odd because things like this are usually either clearly solved or clearly not. After I gave up (mostly because ChatGPT had given me incomplete information needed to solve it) I checked the source of the dispute. It was a site very similar to this one and someone had an AI agent working on the same problem, publishing dozens or hundreds of pages of notes. The agent found the solution page and concluded it was wrong because many of the 32 source passages supposedly didn’t contain enough text. I dug up the PDF of the book and found the mistake - whenever a passage continued onto the next page, the agent wasn’t including that continuation. The passages weren’t actually too short. Annoying that ChatGPT can cite sources like this without being able to properly weigh their reliability.
- appplication 14d ago
  I think this is what obstacles on the path to AGI look like now. It’s random things that would be obvious to a human but are unrepresentative in how an AI views the world and therefore it suddenly becomes seemingly incapable, despite having basically superpowers for proximal work. I don’t mean that to say AGI is here or easy or necessarily that close but it’s likely going to feel like one thing after another until one day most of these things that make you think “how could something so capable be that dumb” are largely solved.
  - 93po 13d ago
    Yes, and right now it seems we get around this problem by spawning 10k agents (that's what OpenAI did for the stokes problem) and hoping that at least one of the 10k catches this and does it right - which it very likely will.
- DenisM 14d ago
  Great story! Verifying sources is a recursive problem - where do you stop? Humans have intuitive feel for it, but agents don’t or at least not yet (I wonder if intuition is just a secondary neural net which is currently being added to the agents as we speak). Also as a human you are able to examine agents erroneous trajectory, real or imaginary, without contaminating your own. Agent have a problem with that - as soon as someone else’s thought is in the context it can lose track of provenance and veracity. Sometimes I think we need a bloom filter to retroactively assign “dirty” flag to invalidated or questionable token spans already in the context.
  - 93po 13d ago
    Even before AI I strongly believed the internet was eventually going to have to move to a web of trust model. I think World ID (formerly Worldcoin) can be a really cool part of the solution to this, though people widely criticize it without fully understanding it and make assumptions that are wrong. But I agree that trust and verification is increasingly a large problem and one of the best ways to combat that is to actually choose unique identities to trust.
  - LordDragonfang 13d ago
    > I wonder if intuition is just a secondary neural net which is currently being added to the agents as we speak Arguably, intuition is primary neural net, the only thing an LLM has without CoT, it just is spiky so humans only notice where it's below-average and just dismiss the rest as normal. Of course it's going to lag behind in some areas compared to others.
- DANmode 13d ago
  > Annoying that ChatGPT can cite sources like this without being able to properly weigh their reliability. It can’t make it past the abstract, in some cases - just like most people!
greenavocado 14d ago
We are so jaded by constant breakthroughs that 100 year old unsolved ciphers are referred to as "low hanging fruit"
- Forgeties79 14d ago
  Something being old and unsolved does not make it impressive when it’s solved. The question is “has there been any concerted effort to solve it and if so how much time/effort has gone in?” I’m sure I can make some brand new “discovery” that is completely useless, which is why it wasn’t “discovered” in the first place.
  - simonklee 14d ago
    It’s simply a lot of grunt work to do these which is why many are unsolved.
- GolfPopper 14d ago
  They are low hanging fruit. This is the sort of thing it ought to be good at, and it's not particularly surprising that it is. But it is also not what is being promoted by LLM advocates on the public stage. Investors are not putting billions into OpenAI to crack historical ciphertexts. This is supposedly a trillion dollar general purpose artificial intelligence, but still can't reliably tell me how many p's are in 'raspberry'. Cracking pre-computer era ciphers with LLMs is like me claiming I have a super-efficient hypersonic precooled hybrid air-breathing rocket engine that will revolutionize all forms of transportation, and then for a demo bragging about how nicely I can grill with it at my backyard BBQ.
  - noir_lord 14d ago
    Yesterday I had one try and use Generics in a language that doesn't have Generics and then use a type declaration that is a trivially obvious error at run time. (Qwen both times). I just asked chatgpt the run time error and this was the entire thing... > Me: is `?mixed $foobar = null` a error in PHP 8.5? > ChatGPT: No. In PHP 8.5, this is valid: `mixed $foobar = null` mixed includes null (as well as all other PHP types), so the default value is compatible. > Me: did you miss the ? > ChatGPT: Yes <SmileyEmojii> — I missed the ?. ?mixed $foobar = null is an error in PHP 8.5. If I'd invested trillions of dollars in that industry I'd be getting worried... Also it using emojii's and words describing emotions just makes my soul shudder. It's not that they don’t generate useful things, they can and then the next minute they smash into the wall repeatedly because the door is 3ft to the left. Since it's stochastic by nature you never know which the answer is going to be and they are making the same kinds of mistakes they where making a couple of years ago, I'm not sure it's actually solvable with the current approach.
simonklee 14d ago
Nice, used a similar approach for another one of these this week https://simonklee.dk/farnese-letter https://simonklee.dk/farnese-letter
deleted 14d ago
[deleted]
huflungdung 14d ago
[dead]

deleted 14d ago

[deleted]

93po 14d ago

Mildly interesting anecdote: when the Cyphral Distich solution popped up a few days ago, I spent about an hour with ChatGPT trying to solve it myself without looking at the proposed solution. ChatGPT opened by saying “the solution is disputed online,” and made the dispute sound fairly convincing, which struck me as odd because things like this are usually either clearly solved or clearly not. After I gave up (mostly because ChatGPT had given me incomplete information needed to solve it) I checked the source of the dispute. It was a site very similar to this one and someone had an AI agent working on the same problem, publishing dozens or hundreds of pages of notes. The agent found the solution page and concluded it was wrong because many of the 32 source passages supposedly didn’t contain enough text. I dug up the PDF of the book and found the mistake - whenever a passage continued onto the next page, the agent wasn’t including that continuation. The passages weren’t actually too short. Annoying that ChatGPT can cite sources like this without being able to properly weigh their reliability.

appplication 14d ago
I think this is what obstacles on the path to AGI look like now. It’s random things that would be obvious to a human but are unrepresentative in how an AI views the world and therefore it suddenly becomes seemingly incapable, despite having basically superpowers for proximal work. I don’t mean that to say AGI is here or easy or necessarily that close but it’s likely going to feel like one thing after another until one day most of these things that make you think “how could something so capable be that dumb” are largely solved.
- 93po 13d ago
  Yes, and right now it seems we get around this problem by spawning 10k agents (that's what OpenAI did for the stokes problem) and hoping that at least one of the 10k catches this and does it right - which it very likely will.
DenisM 14d ago
Great story! Verifying sources is a recursive problem - where do you stop? Humans have intuitive feel for it, but agents don’t or at least not yet (I wonder if intuition is just a secondary neural net which is currently being added to the agents as we speak). Also as a human you are able to examine agents erroneous trajectory, real or imaginary, without contaminating your own. Agent have a problem with that - as soon as someone else’s thought is in the context it can lose track of provenance and veracity. Sometimes I think we need a bloom filter to retroactively assign “dirty” flag to invalidated or questionable token spans already in the context.
- 93po 13d ago
  Even before AI I strongly believed the internet was eventually going to have to move to a web of trust model. I think World ID (formerly Worldcoin) can be a really cool part of the solution to this, though people widely criticize it without fully understanding it and make assumptions that are wrong. But I agree that trust and verification is increasingly a large problem and one of the best ways to combat that is to actually choose unique identities to trust.
- LordDragonfang 13d ago
  > I wonder if intuition is just a secondary neural net which is currently being added to the agents as we speak Arguably, intuition is primary neural net, the only thing an LLM has without CoT, it just is spiky so humans only notice where it's below-average and just dismiss the rest as normal. Of course it's going to lag behind in some areas compared to others.
DANmode 13d ago
> Annoying that ChatGPT can cite sources like this without being able to properly weigh their reliability. It can’t make it past the abstract, in some cases - just like most people!

greenavocado 14d ago

We are so jaded by constant breakthroughs that 100 year old unsolved ciphers are referred to as "low hanging fruit"

Forgeties79 14d ago
Something being old and unsolved does not make it impressive when it’s solved. The question is “has there been any concerted effort to solve it and if so how much time/effort has gone in?” I’m sure I can make some brand new “discovery” that is completely useless, which is why it wasn’t “discovered” in the first place.
- simonklee 14d ago
  It’s simply a lot of grunt work to do these which is why many are unsolved.
GolfPopper 14d ago
They are low hanging fruit. This is the sort of thing it ought to be good at, and it's not particularly surprising that it is. But it is also not what is being promoted by LLM advocates on the public stage. Investors are not putting billions into OpenAI to crack historical ciphertexts. This is supposedly a trillion dollar general purpose artificial intelligence, but still can't reliably tell me how many p's are in 'raspberry'. Cracking pre-computer era ciphers with LLMs is like me claiming I have a super-efficient hypersonic precooled hybrid air-breathing rocket engine that will revolutionize all forms of transportation, and then for a demo bragging about how nicely I can grill with it at my backyard BBQ.
- noir_lord 14d ago
  Yesterday I had one try and use Generics in a language that doesn't have Generics and then use a type declaration that is a trivially obvious error at run time. (Qwen both times). I just asked chatgpt the run time error and this was the entire thing... > Me: is `?mixed $foobar = null` a error in PHP 8.5? > ChatGPT: No. In PHP 8.5, this is valid: `mixed $foobar = null` mixed includes null (as well as all other PHP types), so the default value is compatible. > Me: did you miss the ? > ChatGPT: Yes <SmileyEmojii> — I missed the ?. ?mixed $foobar = null is an error in PHP 8.5. If I'd invested trillions of dollars in that industry I'd be getting worried... Also it using emojii's and words describing emotions just makes my soul shudder. It's not that they don’t generate useful things, they can and then the next minute they smash into the wall repeatedly because the door is 3ft to the left. Since it's stochastic by nature you never know which the answer is going to be and they are making the same kinds of mistakes they where making a couple of years ago, I'm not sure it's actually solvable with the current approach.

simonklee 14d ago

Nice, used a similar approach for another one of these this week https://simonklee.dk/farnese-letter https://simonklee.dk/farnese-letter

deleted 14d ago

[deleted]

huflungdung 14d ago

[dead]