4 ms·
Just did some preliminary testing on decrypting some ROT cyphertext which would have been viable for a human on paper. The output was pretty disappointing: lot
by evrydayhustling 2y ago
Just did some preliminary testing on decrypting some ROT cyphertext which would have been viable for a human on paper. The output was pretty disappointing: lots of "workish" steps creating letter counts, identifying common words, etc, but many steps were incorrect or not followed up on. In the end, it claimed to check its work and deliver an incorrect solution that did not satisfy the previous steps.
I'm not one to judge AI on pratfalls, and cyphers are a somewhat adversarial task. However, there was no aspect of the reasoning that seemed more advanced or consistent than previous chain-of-thought demos I've seen. So the main proof point we have is the paper, and I'm not sure how I'd go from there to being able to trust this on the kind of task it is intended for. Do others have patterns by which they get utility from chain of thought engines?
Separately, chain of thought outputs really make me long for tool use, because the LLM is often forced to simulate algorithmic outputs. It feels like a commercial chain-of-thought solution like this should have a standard library of functions it can use for 100% reliability on things like letter counts.
- charlescurt123 2y agoIt's RL so that means it's going to be great on tasks they created for training but not so much on others. Impressive but the problem with RL is that it requires knowledge of the future.
- changoplatanero 2y agoHmm, are you sure it was using the o1 model and not gpt4o? I've been using the o1 model and it does consistently well at solving rotation ciphers.
- mewpmewp2 2y agoDoes it do better than Claude, because Claude (3.5 sonnet) handled ROTs perfectly and was able to also respond in ROT.
- evrydayhustling 2y agoJust tried, no joy from Claude either: Can you decrypt the following? I don't know the cypher, but the plaintext is Spanish. YRP CFTLIR VE UVDRJZRUF JREZURU, P CF DRJ CFTLIR UV KFUF VJ HLV MVI TFJRJ TFDF JFE VE MVQ UV TFDF UVSVE JVI
- mewpmewp2 2y agoInteresting, it was able to guess it's Rot 17, but it translated it wrong, although "HAY" and some other words were correct. I've tried only in English so far though. It told me it's 17, and "HAY GENTE MU DIFERENTE LECTURA, A LO MUY GENTE DE TODO ES QUE VER COSAS COMO SON EN VEZ DE COMO DEBEN SER" although it really should be "HAY LOCURA EN DEMASIADO SANIDAD, Y LO MAS LOCURA DE TODO ES QUE VER COSAS COMO SON EN VEZ DE COMO DEBEN SER"
- evrydayhustling 2y agoClaude made similar mistakes of generating decryption that was similar to plaintext but with stuff mixed in. I suspect my version of the quote (Miguel de Cervantes) is an apocryphal translation, and there's some utility well on both models to pull it towards the real one. With that said, I did not see o1-preview get as close as you did.
- mewpmewp2 2y agoFor testing I think it's better to use uncommon sentences and also start with English first, if it can solve that, then try other languages.
- ianbutler 2y agoHAY LOCURA EN DEMASIADO SANIDAD, Y LO MAS LOCURA DE TODO ES QUE VER COSAS COMO SON EN VEZ DE COMO DEBEN SER Is that correct? I don't know anything but basic Spanish. All I did was: "The plaintext is in Spanish but I don't know anything else, solve this and explain your reasoning as you go step by step."
- evrydayhustling 2y agoo1-preview . Were you using common plaintexts by chance (e.g. proverbs) or ROT13 specifically? Mine use all the right steps but just can't string them together.
- mewpmewp2 2y agoCould you share your prompt?
- evrydayhustling 2y agoShared one above. Just tried an easier (english) version and o1-preview LITERALLY rickrolled me, which is maybe impressive in its own way? UPDATE: Claude crushed it however. o1-preview Transcript follows. ** Can you decrypt the following? I don't know the cypher. mzilj ivdrzej ufidrek ze kyv sfup reu tre ivrtkzmrkv crkvi ze czwv Certainly! The encrypted text is: ``` mzilj ivdrzej ufidrek ze kyv sfup reu tre ivrtkzmrkv crkvi ze czwv ``` When we decrypt it, we get: ``` never gonna give you up let you down and desert you ``` It appears to be a line from Rick Astley's song "Never Gonna Give You Up."
- mewpmewp2 2y agoThat's weird. Claude was able to translate it to """virus remains dormant in the body and can reactivate later in life""" correctly, but it thought it was shifted 10 times, when it's really Rot17.
- tstrimple 2y ago4o completely botched the Spanish version, but seems to have nailed this one. > Without knowing the cipher used to encrypt this message, it’s hard to decrypt it directly. However, the structure and letter patterns suggest it might be a simple substitution cipher, possibly Caesar or some variation of it. > Let me attempt a few common methods to see if I can identify the cipher. > The most frequent letters in the encrypted message are 'r', 'z', 'v', and 'e'. In English, the most common letters are typically 'E', 'T', 'A', and 'O'. This suggests the cipher could be a simple substitution or a Caesar cipher. > I'll now attempt to apply a Caesar cipher decryption by shifting the alphabet through all possible rotations to see if we get readable text. > One of the decrypted messages, with a shift of 17, reads: > *"virus remains dormant in the body and can reactivate later in life"* > It seems like the correct decryption, revealing information about a virus that can remain dormant in the body.
- J_cst 2y agoOn my machine just works with 4o https://chatgpt.com/share/66e34020-33dc-800d-8ab8-8596895844c9 https://chatgpt.com/share/66e34020-33dc-800d-8ab8-8596895844... With no drama. I'm not sure the bot answer is correct, but looks correct.
- mewpmewp2 2y agoOut of curiousity can you try the same thing with Claude. Because when I tried Claude with any sort of ROT, it had amazing performance, compared to GPT.