Vetch

10 ms·

31.
by Vetch 2y ago
This is an interesting proposition. Have you tested this with the best open LLMs?
32.
by Vetch 2y ago
This seems wrong. Reasoning scales all the way up to the discovery of quaternions and general relativity, often requiring divergent thinking. Reasoning has a core aspect of maintaining uncertainty for better exploration and being able to te
33.
by Vetch 2y ago
The information on the creative step which you provided to o1, was also the key step and contained almost all the difficulty. The hope is that 2025 models could eventually come up with solutions like this given enough time, but this is also
34.
by Vetch 2y ago
Excellent point. The hope is reasoning LLMs will make a difference for such problems. But it's also a great example of why the those who think being able to have the LLM iterate more will be crucial to reasoning are off base. There are
35.
by Vetch 2y ago
I've also used it to make a function tail recursive, which is useful in languages with TCO.
36.
by Vetch 2y ago
Note that this is an expressibility (upper) bound on transformers granted intermediate decoding steps. It says nothing about their learnability, and modern LLMs are not near that level of expressive capacity. The authors also introduce proj
37.
by Vetch 2y ago
!Long post warning! Tokenization is often scapegoated for many transformer limitations. I suppose it's because reading about the many limitations of the transformer architecture is harder than dumping everything on tokenization (which
38.
by Vetch 2y ago
I think it's exploring in-context. Bringing up related ideas and not getting confused by them is pivotal to these models eventually being able to contribute as productive reasoners. These traces will be immediately helpful in a real wo
39.
by Vetch 2y ago
I disagree, that is really only police-able for online services. For local apps, which will eventually include games, assistants and machine symbiosis, I expect a bring your own model approach.
40.
by Vetch 2y ago
F#'s computation expressions are closely related to Haskell's monads + do-notation combo, CEs are both more limited than Haskell's approach to monads (from a type expressibility perspective) and more expressive than pure mona
41.
by Vetch 2y ago
Based on the fact that squinting works, I applied a Gaussian blur to the image. Here's the response I got: Markdown: The provided image is a blurred text that reads "STOP THINKING IN CIRCLES." There are no other visible el
42.
by Vetch 2y ago
Claude Sonnet 3.5s are bars too high to clear. No other model comes close, with the occasional exception of o1-preview. But o1-preview is always a gamble, your rolls are limited and it will either be the best answer possible from an LLM or
43.
by Vetch 2y ago
In addition to the already mentioned https://huggingface.co/facebook/nougat-base , I also highly recommend https://huggingface.co/stepfun-ai/GOT-OCR2_0 . It might even be better.
44.
by Vetch 2y ago
It's amazing how Shannon contributed to almost everything important to understanding computers at all levels. He contributed to information theory (thus communication, error correction and compression), digital circuits (Master's
45.
by Vetch 2y ago
It does and the paper mentions some papers that investigate similar strategies in its appendix, but the fundamental precision issues do not go away.
46.
by Vetch 2y ago
That's not quite right. By numerical precision they mean numerical precision, of which quantization is one method to arrive at reduced precision. They also perform experiments where they train from scratch for float32 and float16. Reas
47.
by Vetch 2y ago
Cost per FLOP continues to drop on an exponential trend (and what bit flops do we mean?). Leaving aside more effective training methodologies and how that muddies everything by allowing superior to GPT4 perf using less training flops, it al
48.
by Vetch 2y ago
> The practice of solving problems that you describe is to ingrain/memorize those steps so you don't forget how to apply the procedure correctly Simply memorizing sequences of steps is not how mathematics learning works, otherw
49.
by Vetch 2y ago
> It will often contain a mistake...but the same is true for a human. If this were true textbooks could not work. Given a question, we don't consult random humans but experts of their field. If I have a question on algorithms, I mig
50.
by Vetch 2y ago
If you train it then it's no longer the same model. If I have f(x) = x + 1 and change it to f(x) = x + 1 + 1/1e9, it would not mean that `f` is not deterministic. The issue would be in whatever interface I was exposing the f'
51.
by Vetch 2y ago
The paper also proves that this capability, one unlikely to occur naturally, does not help for tasks where one must create sequentially dependent chains of reasoning, a limiting constraint. At least not without overturning what we believe a
52.
by Vetch 2y ago
This paper reads to me as being about fundamental limitations of Transformers and backdoor risk. The paper starts off by reviewing work which uses an encompassing theoretical model of transformers to prove they're limited to only expre
53.
by Vetch 2y ago
Do you mean Poe or Microsoft? Microsoft is Microsoft. Poe has significantly reduced the generosity of their free tiers over time.
54.
by Vetch 2y ago
They weren't misguided, they just over-focused on scaling parameters instead of appropriately scaling quality data in tandem. A 4B has limited capacity to encode knowledge and algorithmic circuits. It's also too small to learn pro
55.
by Vetch 2y ago
Poe's Sonnet is limited to 15 free messages per day. The best freely accessible LLM with a generous daily allotment (300 msgs/day) is Bing Green Precise Mode. It's at about GPT4 level.
56.
by Vetch 2y ago
This specific problem is certainly not one for all on-devices AI processing. As someone else mentioned, there are unique UX and browser constraints that come from serving large compute intensive binary blobs through the browser (that are al
57.
by Vetch 3y ago
I disagree with your characterization of Vinge's works as primarily about disasters but I agree they were all about an accelerating technological pace and its relation with intelligence. I'm fairly certain the mysterious event in
58.
by Vetch 3y ago
> Are you sure? I think "Open"AI uses the chat transcripts to help the next training run? > Fine-tuning. The learning that occurs through SGD is proven to be less flexible and generalizing than what happens via context. This
59.
by Vetch 3y ago
That doesn't seem to be the case here. Reading through the article and twitter thread, the impression I get is that between moyix and the author, a decent amount of time was spent on this. A valid criticism that could have been made is
60.
by Vetch 3y ago
This is a poor analogy, a better one would be nuclear physics. An expert in nuclear physics can develop positively impactful energy generation methods or very damaging nuclear weapons. It's not because of arcane secrets that so few nat

More ›