5 ms·

This isn't really a reasonable approach, is it? The original prompts aren't provided, nor is the original context; even then, you can't really treat a stochast

This isn't really a reasonable approach, is it? The original prompts aren't provided, nor is the original context; even then, you can't really treat a stochastic system like an LLM as a major component in reproducibility.

ben_w 4mo ago
> even then, you can't really treat a stochastic system like an LLM as a major component in reproducibility. If you had the other things, being "stochastic" is not even remotely a show-stopper. Stochastic processes abound and are the reason the mathematics of statistics was developed in the first place, ultimately allowing us to create such things as LLMs. When all the relevant steps gets published, I absolutely expect a lot of people to (attempt to) reproduce this work even though LLMs are stochastic.
- _verandaguy 4mo ago
  My issue with this is that it's a form of "soft" reproducibility, where it'll work for many (maybe even most!) people, but that depends on the way the original prompt was formulated (read on) and the state of the random noise in the system. On the prompt formulation; prompts with very similar formulations (in terms of both semantics, hamming distance, or both) can lead to _wildly divergent_ outputs in my experience. It's not rigourous, and when that divergence happens, it's extremely difficult (arguably impossible, by nature of the architecture of transformers) to identify why the divergence happened and where.
fragmede 4mo ago
Sure it is. We're humans, not robots (well, I think I am, and I presume you are as well, but for all we know, we could be living in a simulation), so if the non-deterministic system decides to generate code that calls the variable foo one day and bar the next, as long as the code still does what's being asked of it, why do I care that the non deterministic system chose to call the variable something different when run on Tuesday? There's the computer science definition of determinism and the engineering result of "does it work", which are at odds. It's like the halting problem. We haven't solved the computer science definition of the halting problem, but give some C code with a loop that won't terminate to Claude, and it'll call that out as not halting.
- _verandaguy 4mo ago
  All things aside, I think this misses the forest for the trees on the halting problem. It's not about being able to throw claude or codex at a loop and having it evaluate it for halting, it's about being able to do this for arbitrary code. Computer science rigourously defines the halting problem as not computable and undecidable. within the framework of using something akin to static analysis using any deterministic Turing machine. There's not really a question of "solving" the halting problem like there's some as-yet unknown way of generally figuring out if arbitraty code halts. Turing proposed a proof in 1937 in favour of undecidability of what we now know as the halting problem, building on ideas first articulated by Church a few years prior. Frankly, if anything, it's reasonable to say that the halting problem's been solved, just in the direction of undecidability rather than decidability. Anyway, back to LLMs; as code gets more complex, the robot will need a bigger context window, more hardware resources, and more time, all of which will be variable due to the noise inherent in the system. It'll be difficult to put a useful upper and lower bound on how much computing power and time it'll take to figure out if a program ever halts. Which is all a bit moot, frankly, in the context of halting, but useful to keep in mind in the more general context of using these things as analysis tools.
- deleted 4mo ago
  [deleted]
iwontberude 4mo ago
Actually it is because Claude did the work and being a lay person isn’t really that high of a bar.
- Kosturdistan 4mo ago
  Claude helped, but did not do the work. This was a human dude who had a very helpful assist from Claude
TeMPOraL 4mo ago
> stochastic system Every day when you lower your butt onto your chair, you trust a stochastic system enough to assume you'll rest on the chair safely and not spontaneously phase through, which would lead to rather gory and painful terminal experience. Physics at macro scale is stochastic, which is a good reminder that stochastic != uniformly random. Expected distributions matter.
- ben_w 4mo ago
  While strictly true, QM has such small standard deviations as to be irrelevant on the macro for things like bums and chairs. IMO a better example would be the stochastic nature of quality control in manufacturing.
  - TeMPOraL 4mo ago
    > QM has such small standard deviations as to be irrelevant on the macro for things like bums and chairs I was going to segue into thermodynamics as a backup example, but you made me think of something better. > IMO a better example would be the stochastic nature of quality control in manufacturing. How about, more specifically, food manufacturing? Or maybe, let's talk about cooking? Cooking is as stochastic as it gets, and we handle it fine. It could be better - the better version is called "chemical process engineering", it's what cooking looks like when you care about quality and consistency of output, and can afford the equipment and process actually necessary for it. Regular people don't (i.e. neither care, nor can afford) - we call this cooking. It's an art, not a science, and people not only do it, but love it, and tie their identities to it, and build businesses around it, and a culture that embraces all the compromises (and calls the more serious approach "unhealthy").
    - ben_w 4mo ago
      > Cooking is as stochastic as it gets, and we handle it fine. It could be better My attempts at making bread have been too stochastic, in that it hardly ever produces nice results. But yes. Eyeballing how much dried herbs to put in my dishes because I like what 2-isopropyl-5-methylphenol does for them. Usually it works, sometimes it's just a bit too Italian.
Kosturdistan 4mo ago
Claude code was used to organize the material and to run simulations. The simulations were to determine the likelihood that the text was Semitic vs Tom got lucky. Tom has assigned probabilities to each of the syllables he has proposed sound values for.
peterfirefly 4mo ago
I think I caught this guy's reddit posts on the subject. Someone was playing around with statistical analyses of a big Linear A corpus + some other corpora. There was an extremely clear signal that Linear A seemed to be much more similar to one other corpus than to the others. This was the first time I've ever heard of something that might* have been a good hint for decipherment. There's a Dutch professor emeritus (in linguistics) who claims it is Hurrian-Urartian and he's been posting youtube videos about his "decipherment" but he didn't seem too convincing to me. Claude helped write code to read and parse the corpora and to do some fairly basic statistical analysis along the lines of "which Linear A symbols most often occur together" and "if we use known Linear B sound values, which of the other corpora most often have vowel similarities with the Linear A corpus". You can write that code yourself or you can ask an LLVM to write it for you. The provenience of the code isn't important. *) He later deleted some of them, I think. What was still there on reddit a few weeks ago had dead links to a web site of his with statistical tables and I believe also code.
- petjuh 4mo ago
  Ok, but Hurro-Urartian is an agglutinative language family that is an isolate and has nothing to do with Semitic.
  - peterfirefly 4mo ago
    There are two relevant decipherment ideas: the new one that is the first one sufficiently "un-bad" to be taken seriously and the one by the Dutch emeritus. The Dutch guy thinks he can recognize some names (personal and toponyms) and that seems to be his main evidence for Linear A being Hurrian/Urartian. His theory doesn't seem very convincing (and his youtube videos are incredibly booooring).
cake-rusk 4mo ago
Isn't this like the P vs NP problems? Once you have a solution it is easy to verify?

ben_w 4mo ago

> even then, you can't really treat a stochastic system like an LLM as a major component in reproducibility. If you had the other things, being "stochastic" is not even remotely a show-stopper. Stochastic processes abound and are the reason the mathematics of statistics was developed in the first place, ultimately allowing us to create such things as LLMs. When all the relevant steps gets published, I absolutely expect a lot of people to (attempt to) reproduce this work even though LLMs are stochastic.

_verandaguy 4mo ago
My issue with this is that it's a form of "soft" reproducibility, where it'll work for many (maybe even most!) people, but that depends on the way the original prompt was formulated (read on) and the state of the random noise in the system. On the prompt formulation; prompts with very similar formulations (in terms of both semantics, hamming distance, or both) can lead to _wildly divergent_ outputs in my experience. It's not rigourous, and when that divergence happens, it's extremely difficult (arguably impossible, by nature of the architecture of transformers) to identify why the divergence happened and where.

fragmede 4mo ago

Sure it is. We're humans, not robots (well, I think I am, and I presume you are as well, but for all we know, we could be living in a simulation), so if the non-deterministic system decides to generate code that calls the variable foo one day and bar the next, as long as the code still does what's being asked of it, why do I care that the non deterministic system chose to call the variable something different when run on Tuesday? There's the computer science definition of determinism and the engineering result of "does it work", which are at odds. It's like the halting problem. We haven't solved the computer science definition of the halting problem, but give some C code with a loop that won't terminate to Claude, and it'll call that out as not halting.

_verandaguy 4mo ago
All things aside, I think this misses the forest for the trees on the halting problem. It's not about being able to throw claude or codex at a loop and having it evaluate it for halting, it's about being able to do this for arbitrary code. Computer science rigourously defines the halting problem as not computable and undecidable. within the framework of using something akin to static analysis using any deterministic Turing machine. There's not really a question of "solving" the halting problem like there's some as-yet unknown way of generally figuring out if arbitraty code halts. Turing proposed a proof in 1937 in favour of undecidability of what we now know as the halting problem, building on ideas first articulated by Church a few years prior. Frankly, if anything, it's reasonable to say that the halting problem's been solved, just in the direction of undecidability rather than decidability. Anyway, back to LLMs; as code gets more complex, the robot will need a bigger context window, more hardware resources, and more time, all of which will be variable due to the noise inherent in the system. It'll be difficult to put a useful upper and lower bound on how much computing power and time it'll take to figure out if a program ever halts. Which is all a bit moot, frankly, in the context of halting, but useful to keep in mind in the more general context of using these things as analysis tools.
deleted 4mo ago
[deleted]

iwontberude 4mo ago

Actually it is because Claude did the work and being a lay person isn’t really that high of a bar.

Kosturdistan 4mo ago
Claude helped, but did not do the work. This was a human dude who had a very helpful assist from Claude

TeMPOraL 4mo ago

> stochastic system Every day when you lower your butt onto your chair, you trust a stochastic system enough to assume you'll rest on the chair safely and not spontaneously phase through, which would lead to rather gory and painful terminal experience. Physics at macro scale is stochastic, which is a good reminder that stochastic != uniformly random. Expected distributions matter.

ben_w 4mo ago
While strictly true, QM has such small standard deviations as to be irrelevant on the macro for things like bums and chairs. IMO a better example would be the stochastic nature of quality control in manufacturing.
- TeMPOraL 4mo ago
  > QM has such small standard deviations as to be irrelevant on the macro for things like bums and chairs I was going to segue into thermodynamics as a backup example, but you made me think of something better. > IMO a better example would be the stochastic nature of quality control in manufacturing. How about, more specifically, food manufacturing? Or maybe, let's talk about cooking? Cooking is as stochastic as it gets, and we handle it fine. It could be better - the better version is called "chemical process engineering", it's what cooking looks like when you care about quality and consistency of output, and can afford the equipment and process actually necessary for it. Regular people don't (i.e. neither care, nor can afford) - we call this cooking. It's an art, not a science, and people not only do it, but love it, and tie their identities to it, and build businesses around it, and a culture that embraces all the compromises (and calls the more serious approach "unhealthy").
  - ben_w 4mo ago
    > Cooking is as stochastic as it gets, and we handle it fine. It could be better My attempts at making bread have been too stochastic, in that it hardly ever produces nice results. But yes. Eyeballing how much dried herbs to put in my dishes because I like what 2-isopropyl-5-methylphenol does for them. Usually it works, sometimes it's just a bit too Italian.

Kosturdistan 4mo ago

Claude code was used to organize the material and to run simulations. The simulations were to determine the likelihood that the text was Semitic vs Tom got lucky. Tom has assigned probabilities to each of the syllables he has proposed sound values for.

peterfirefly 4mo ago

I think I caught this guy's reddit posts on the subject. Someone was playing around with statistical analyses of a big Linear A corpus + some other corpora. There was an extremely clear signal that Linear A seemed to be much more similar to one other corpus than to the others. This was the first time I've ever heard of something that might* have been a good hint for decipherment. There's a Dutch professor emeritus (in linguistics) who claims it is Hurrian-Urartian and he's been posting youtube videos about his "decipherment" but he didn't seem too convincing to me. Claude helped write code to read and parse the corpora and to do some fairly basic statistical analysis along the lines of "which Linear A symbols most often occur together" and "if we use known Linear B sound values, which of the other corpora most often have vowel similarities with the Linear A corpus". You can write that code yourself or you can ask an LLVM to write it for you. The provenience of the code isn't important. *) He later deleted some of them, I think. What was still there on reddit a few weeks ago had dead links to a web site of his with statistical tables and I believe also code.

petjuh 4mo ago
Ok, but Hurro-Urartian is an agglutinative language family that is an isolate and has nothing to do with Semitic.
- peterfirefly 4mo ago
  There are two relevant decipherment ideas: the new one that is the first one sufficiently "un-bad" to be taken seriously and the one by the Dutch emeritus. The Dutch guy thinks he can recognize some names (personal and toponyms) and that seems to be his main evidence for Linear A being Hurrian/Urartian. His theory doesn't seem very convincing (and his youtube videos are incredibly booooring).

cake-rusk 4mo ago

Isn't this like the P vs NP problems? Once you have a solution it is easy to verify?