6 ms·

Then why is there no link to the actual write-up?

by yorwba 4mo ago

kelseyfrog 4mo ago
You can use Claude, like the author, to reproduce the result.
- justin_dash 4mo ago
  Unless if it was done by Fable!
  - kelseyfrog 4mo ago
    The 'major insight' described in the article predates Fable's release by two week four days. It would be a complicated timeline.
- atrus 4mo ago
  somehow I suspect it was a bit more involved than: Claude, please solve Linear A.
  - smsm42 4mo ago
    You also have to add "make no mistakes"!
  - fragmede 4mo ago
    A little bit more. If you ask ChatGPT to "solve linear a" it thinks you mean linear algebra. If you specify that it's the Minoan translation problem, you get a table similar to the one that we get a glimpse of in the without access to the paper, we can't say how much more work the paper has than my gist. https://gist.github.com/fragmede/bbf277d36a2398065f109484f3473684 https://gist.github.com/fragmede/bbf277d36a2398065f109484f34...
  - Kosturdistan 4mo ago
    You are correct!
- _verandaguy 4mo ago
  This isn't really a reasonable approach, is it? The original prompts aren't provided, nor is the original context; even then, you can't really treat a stochastic system like an LLM as a major component in reproducibility.
  - ben_w 4mo ago
    > even then, you can't really treat a stochastic system like an LLM as a major component in reproducibility. If you had the other things, being "stochastic" is not even remotely a show-stopper. Stochastic processes abound and are the reason the mathematics of statistics was developed in the first place, ultimately allowing us to create such things as LLMs. When all the relevant steps gets published, I absolutely expect a lot of people to (attempt to) reproduce this work even though LLMs are stochastic.
    - _verandaguy 4mo ago
      My issue with this is that it's a form of "soft" reproducibility, where it'll work for many (maybe even most!) people, but that depends on the way the original prompt was formulated (read on) and the state of the random noise in the system. On the prompt formulation; prompts with very similar formulations (in terms of both semantics, hamming distance, or both) can lead to _wildly divergent_ outputs in my experience. It's not rigourous, and when that divergence happens, it's extremely difficult (arguably impossible, by nature of the architecture of transformers) to identify why the divergence happened and where.
  - fragmede 4mo ago
    Sure it is. We're humans, not robots (well, I think I am, and I presume you are as well, but for all we know, we could be living in a simulation), so if the non-deterministic system decides to generate code that calls the variable foo one day and bar the next, as long as the code still does what's being asked of it, why do I care that the non deterministic system chose to call the variable something different when run on Tuesday? There's the computer science definition of determinism and the engineering result of "does it work", which are at odds. It's like the halting problem. We haven't solved the computer science definition of the halting problem, but give some C code with a loop that won't terminate to Claude, and it'll call that out as not halting.
GavinMcG 4mo ago
Presumably because it hasn’t yet been published?
- deleted 4mo ago
  [deleted]
m0llusk 4mo ago
It seems this is still extremely early in the process. There is an apparent finding that was shared. Evidence which would be the basis for a paper is "being reviewed by linguistics experts at Rutgers and Cambridge". So they are trying to do the right thing by talking about what they believe they have done but holding off publication and serious claims until later. The general idea that written forms can be categorized by systems built with Claude could be applied to other as yet undecipherable languages could be used by other interested investigators just with what is discussed here.
- sillysaurusx 4mo ago
  > The general idea that written forms can be categorized by systems built with Claude could be applied to other as yet undecipherable languages could be used by other interested investigators just with what is discussed here. Could you rephrase this or explain it more thoroughly? I don’t follow. What does it mean to categorize a written form by systems built with Claude?
  - tyingq 4mo ago
    The same pattern/tech is generic enough that it might be able to solve other unrelated, and so-far undecipherable, written languages.
Kosturdistan 4mo ago
The only write-up at the moment is my blog post, hopefully that changes in the coming weeks.
- Sniffnoy 4mo ago
  The blog post mentions a draft of a manuscript though. I was expecting something like a preprint. He's not willing to post that draft yet?
  - Kosturdistan 4mo ago
    I have seen and read his draft article, he's not comfortable sharing it publicly yet since it's being reviewed by experts.
    - ahknight 4mo ago
      But the Internet nerds wish to blindly judge something they know nothing about so they can feel better with the assumption that they could have done better somehow. How will they be appeased if the document they will say they have read and understood (without having done either) is not available to point at? How, I ask?!
      - qustio 4mo ago
        It's entirely reasonable to ask for the underlying research in response to a blog post hyping up an unproven claim in an area notoriously full of amateurs making the same claim that historically fail to stand up to scrutiny. Particularly when the only source is a friend of the author, posting on a blog named "AI Clambake" about "A weekly, human-powered newsletter for advertising folks who want to stay on top of the AI mayhem" and not a publication with any credibility in linguistics. None of that means it can't be true, but some basic skepticism is warranted here. Otherwise we end up in a situation like the LK99 room temperature superconductor where a lot of HN commenters were also upset at the cynical "downers" who just couldn't root for a good thing/progress.

Then why is there no link to the actual write-up?

by yorwba 4mo ago

Then why is there no link to the actual write-up?

kelseyfrog 4mo ago

You can use Claude, like the author, to reproduce the result.

justin_dash 4mo ago
Unless if it was done by Fable!
- kelseyfrog 4mo ago
  The 'major insight' described in the article predates Fable's release by two week four days. It would be a complicated timeline.
atrus 4mo ago
somehow I suspect it was a bit more involved than: Claude, please solve Linear A.
- smsm42 4mo ago
  You also have to add "make no mistakes"!
- fragmede 4mo ago
  A little bit more. If you ask ChatGPT to "solve linear a" it thinks you mean linear algebra. If you specify that it's the Minoan translation problem, you get a table similar to the one that we get a glimpse of in the without access to the paper, we can't say how much more work the paper has than my gist. https://gist.github.com/fragmede/bbf277d36a2398065f109484f3473684 https://gist.github.com/fragmede/bbf277d36a2398065f109484f34...
- Kosturdistan 4mo ago
  You are correct!
_verandaguy 4mo ago
This isn't really a reasonable approach, is it? The original prompts aren't provided, nor is the original context; even then, you can't really treat a stochastic system like an LLM as a major component in reproducibility.
- ben_w 4mo ago
  > even then, you can't really treat a stochastic system like an LLM as a major component in reproducibility. If you had the other things, being "stochastic" is not even remotely a show-stopper. Stochastic processes abound and are the reason the mathematics of statistics was developed in the first place, ultimately allowing us to create such things as LLMs. When all the relevant steps gets published, I absolutely expect a lot of people to (attempt to) reproduce this work even though LLMs are stochastic.
  - _verandaguy 4mo ago
    My issue with this is that it's a form of "soft" reproducibility, where it'll work for many (maybe even most!) people, but that depends on the way the original prompt was formulated (read on) and the state of the random noise in the system. On the prompt formulation; prompts with very similar formulations (in terms of both semantics, hamming distance, or both) can lead to _wildly divergent_ outputs in my experience. It's not rigourous, and when that divergence happens, it's extremely difficult (arguably impossible, by nature of the architecture of transformers) to identify why the divergence happened and where.
- fragmede 4mo ago
  Sure it is. We're humans, not robots (well, I think I am, and I presume you are as well, but for all we know, we could be living in a simulation), so if the non-deterministic system decides to generate code that calls the variable foo one day and bar the next, as long as the code still does what's being asked of it, why do I care that the non deterministic system chose to call the variable something different when run on Tuesday? There's the computer science definition of determinism and the engineering result of "does it work", which are at odds. It's like the halting problem. We haven't solved the computer science definition of the halting problem, but give some C code with a loop that won't terminate to Claude, and it'll call that out as not halting.

GavinMcG 4mo ago

Presumably because it hasn’t yet been published?

deleted 4mo ago
[deleted]

m0llusk 4mo ago

It seems this is still extremely early in the process. There is an apparent finding that was shared. Evidence which would be the basis for a paper is "being reviewed by linguistics experts at Rutgers and Cambridge". So they are trying to do the right thing by talking about what they believe they have done but holding off publication and serious claims until later. The general idea that written forms can be categorized by systems built with Claude could be applied to other as yet undecipherable languages could be used by other interested investigators just with what is discussed here.

sillysaurusx 4mo ago
> The general idea that written forms can be categorized by systems built with Claude could be applied to other as yet undecipherable languages could be used by other interested investigators just with what is discussed here. Could you rephrase this or explain it more thoroughly? I don’t follow. What does it mean to categorize a written form by systems built with Claude?
- tyingq 4mo ago
  The same pattern/tech is generic enough that it might be able to solve other unrelated, and so-far undecipherable, written languages.

Kosturdistan 4mo ago

The only write-up at the moment is my blog post, hopefully that changes in the coming weeks.

Sniffnoy 4mo ago
The blog post mentions a draft of a manuscript though. I was expecting something like a preprint. He's not willing to post that draft yet?
- Kosturdistan 4mo ago
  I have seen and read his draft article, he's not comfortable sharing it publicly yet since it's being reviewed by experts.
  - ahknight 4mo ago
    But the Internet nerds wish to blindly judge something they know nothing about so they can feel better with the assumption that they could have done better somehow. How will they be appeased if the document they will say they have read and understood (without having done either) is not available to point at? How, I ask?!
    - qustio 4mo ago
      It's entirely reasonable to ask for the underlying research in response to a blog post hyping up an unproven claim in an area notoriously full of amateurs making the same claim that historically fail to stand up to scrutiny. Particularly when the only source is a friend of the author, posting on a blog named "AI Clambake" about "A weekly, human-powered newsletter for advertising folks who want to stay on top of the AI mayhem" and not a publication with any credibility in linguistics. None of that means it can't be true, but some basic skepticism is warranted here. Otherwise we end up in a situation like the LK99 room temperature superconductor where a lot of HN commenters were also upset at the cynical "downers" who just couldn't root for a good thing/progress.