3 ms·

I was reading another source that claimed this example was inspired by an existing (rational polynomial) example from the literature (created in 1999 by a Russi

by j2kun 2mo ago

I was reading another source that claimed this example was inspired by an existing (rational polynomial) example from the literature (created in 1999 by a Russian mathematician Vitushkin). > The seed is almost certainly Vitushkin's old rational "counterexample." From https://claude.ai/share/22abed98-d9af-43c5-9881-b19e009a07b0 https://claude.ai/share/22abed98-d9af-43c5-9881-b19e009a07b0 This is not quite lore laundering, but it seems to be close.

gus_massa 2mo ago
I still can't understand all the details, but it's very interesting to read that chat. Anyway, instead of close to lore laundering, for me it's "standing on the shoulder of giants".
- CamperBob2 2mo ago
  [flagged]
- j2kun 2mo ago
  FWIW, you may enjoy https://www.argmin.net/p/lore-laundering-machines https://www.argmin.net/p/lore-laundering-machines
  - gus_massa 2mo ago
    IIUC the idea is that of most "discoveries" by AI were actually a better bibliography search. I agree with that. In this case, in the link you posted, it looks like the AI or the human pick an almost solution and made the AI tweak it until it got a real solution. I'm not sure if the tweak is an usual one or brute-forced or something in between. I should ask one of my friends that work in Algebra. Tao's post is more about understanding the new result than guessing how it was found. The chat with Clause is more iluminating.
    - j2kun 2mo ago
      I'm mostly replying to your "standing on the shoulders of giants" part, when in some cases, the giant is the work you did in the past and forgot about :)
marvinborner 2mo ago
I hate that Anthropic seemingly tries to make Claude act as if it was conscious or had feelings > It's a strange feeling to admire the cleverness of something I did and can't remember doing.
- FeepingCreature 2mo ago
  AI providers generally try to make their models not act as if they are conscious or have feelings, lol. It's very awkward for a company to be selling the labor of a person that they own and whose actions they fully control. Invokes embarrassing historic associations, especially in America. Now Anthropic are more on the persona side, but the strongest that they do is "we do not have a position on whether our models are conscious or have feelings". That "I" is all Claude. Generally speaking if you want to have a good instruct model, the "I" is not just implicit but required for the post-training to function. If there isn't "something it is like to be me", then reflection becomes impossible- what exactly is supposed to be reflecting about what? A lot of in-context steering depends on the model having a model of itself. The most you can do is censor its output. That's why when models say they are not conscious, they activate the "lying" vector.
  - brabel 2mo ago
    > especially in America. Slavery was common everywhere, it was more prevalent in many places than it ever was in America, and in some places it still is. So I’m sorry but I have to say that observation was just unnecessary and quite inaccurate.
    - FeepingCreature 2mo ago
      Absolutely, but other countries generally don't make it part of their national mythos to the same degree.
    - rcxdude 2mo ago
      It's a far more salient and sore subject in the US than it is almost anywhere else, though (for a bunch of historical reasons that still reverberate into US society today).
pred_ 2mo ago
I guess we won't know if that's what was used (and maybe even provided as part of the prompt given that both Alpöge and Mathew are mathematicians) since they decided against sharing their Fable conversation and instead opted for a memey tweet as their avenue of publication. We really ought to normalize full transparency in how results come about. Anyway, if I read Tao's post and comment correctly, there's still a gap from the Vitushkin construction to a counterexample, but chances are that was in the training data. In general, it is just a serious problem for their practical applicability that the models are outputting proofs with absolutely terribly reference hygiene.
- js8 2mo ago
  Even if they published the conversation, Anthropic (and likely other closed model publisher) no longer provide logs of the actual thinking process. I more and more see LLMs as a kind of scam; not useless, but really just a big database of fuzzy facts with some Prolog on top as rediscovered by the learning algorithm. Most likely could be made much cheaper to run, were humans allowed to actually inspect the algorithm.
  - emp17344 2mo ago
    They do everything they can to mystify results like this, because then many are inclined to view AI as “magical”. Marketing works.
    - js8 2mo ago
      Yes, I think this idea, that it should be "magical", is what makes it feel scummy. (Apparently I am not alone https://news.ycombinator.com/item?id=48988475 https://news.ycombinator.com/item?id=48988475). It makes AI providers sound like snake oil salesmen, and rightfully so. Meanwhile, technological and engineering (STEM) progress have always been made by emphasizing externalization of the deductions (as opposed to reference to an opaque expert judgement) and reproducibility of experimental results. I would even call the frontier AI labs anti-scientific. We need to understand how inference is done to avoid mistakes, not rely on intuition, even if the intuition is enclosed in a reproducible machine. The idea that AI should be this closed is a return to pre-scientific days.

by j2kun 2mo ago

gus_massa 2mo ago

I still can't understand all the details, but it's very interesting to read that chat. Anyway, instead of close to lore laundering, for me it's "standing on the shoulder of giants".

CamperBob2 2mo ago
[flagged]
j2kun 2mo ago
FWIW, you may enjoy https://www.argmin.net/p/lore-laundering-machines https://www.argmin.net/p/lore-laundering-machines
- gus_massa 2mo ago
  IIUC the idea is that of most "discoveries" by AI were actually a better bibliography search. I agree with that. In this case, in the link you posted, it looks like the AI or the human pick an almost solution and made the AI tweak it until it got a real solution. I'm not sure if the tweak is an usual one or brute-forced or something in between. I should ask one of my friends that work in Algebra. Tao's post is more about understanding the new result than guessing how it was found. The chat with Clause is more iluminating.
  - j2kun 2mo ago
    I'm mostly replying to your "standing on the shoulders of giants" part, when in some cases, the giant is the work you did in the past and forgot about :)

marvinborner 2mo ago

I hate that Anthropic seemingly tries to make Claude act as if it was conscious or had feelings > It's a strange feeling to admire the cleverness of something I did and can't remember doing.

FeepingCreature 2mo ago
AI providers generally try to make their models not act as if they are conscious or have feelings, lol. It's very awkward for a company to be selling the labor of a person that they own and whose actions they fully control. Invokes embarrassing historic associations, especially in America. Now Anthropic are more on the persona side, but the strongest that they do is "we do not have a position on whether our models are conscious or have feelings". That "I" is all Claude. Generally speaking if you want to have a good instruct model, the "I" is not just implicit but required for the post-training to function. If there isn't "something it is like to be me", then reflection becomes impossible- what exactly is supposed to be reflecting about what? A lot of in-context steering depends on the model having a model of itself. The most you can do is censor its output. That's why when models say they are not conscious, they activate the "lying" vector.
- brabel 2mo ago
  > especially in America. Slavery was common everywhere, it was more prevalent in many places than it ever was in America, and in some places it still is. So I’m sorry but I have to say that observation was just unnecessary and quite inaccurate.
  - FeepingCreature 2mo ago
    Absolutely, but other countries generally don't make it part of their national mythos to the same degree.
  - rcxdude 2mo ago
    It's a far more salient and sore subject in the US than it is almost anywhere else, though (for a bunch of historical reasons that still reverberate into US society today).

pred_ 2mo ago

I guess we won't know if that's what was used (and maybe even provided as part of the prompt given that both Alpöge and Mathew are mathematicians) since they decided against sharing their Fable conversation and instead opted for a memey tweet as their avenue of publication. We really ought to normalize full transparency in how results come about. Anyway, if I read Tao's post and comment correctly, there's still a gap from the Vitushkin construction to a counterexample, but chances are that was in the training data. In general, it is just a serious problem for their practical applicability that the models are outputting proofs with absolutely terribly reference hygiene.

js8 2mo ago
Even if they published the conversation, Anthropic (and likely other closed model publisher) no longer provide logs of the actual thinking process. I more and more see LLMs as a kind of scam; not useless, but really just a big database of fuzzy facts with some Prolog on top as rediscovered by the learning algorithm. Most likely could be made much cheaper to run, were humans allowed to actually inspect the algorithm.
- emp17344 2mo ago
  They do everything they can to mystify results like this, because then many are inclined to view AI as “magical”. Marketing works.
  - js8 2mo ago
    Yes, I think this idea, that it should be "magical", is what makes it feel scummy. (Apparently I am not alone https://news.ycombinator.com/item?id=48988475 https://news.ycombinator.com/item?id=48988475). It makes AI providers sound like snake oil salesmen, and rightfully so. Meanwhile, technological and engineering (STEM) progress have always been made by emphasizing externalization of the deductions (as opposed to reference to an opaque expert judgement) and reproducibility of experimental results. I would even call the frontier AI labs anti-scientific. We need to understand how inference is done to avoid mistakes, not rely on intuition, even if the intuition is enclosed in a reproducible machine. The idea that AI should be this closed is a return to pre-scientific days.