gugagore

8 ms·

31.
by gugagore 1y ago
Also, statefullness. Baudot has two codes used for switching into one of two modes: figures and letters. Typewriters have some statefullness, too, like "shift lock". Baudot needed to encode the actions of a type writer to control
32.
by gugagore 1y ago
"A language that doesn't affect the way you think about programming is not worth knowing." ― Alan J. Perlis
33.
by gugagore 1y ago
A search for "LLM Harry Potter" would suggest that LLMs are widely understood to be proficient at rote memorization. (I find the example of the computational geometry algorithm being a clear case of direct memorization not very co
34.
by gugagore 1y ago
You don't have to precisely represent the float in decimal. You just have to have each float have a unique decimal representation, which you can guarantee if you include enough digits: 9 for 32-bit floats, and 17 for 64-bit floats. ht
35.
by gugagore 1y ago
I think search is a fairly simple control loop. Beam search is an example of TTC in this modern era. It is a very wide term, IME, that means anything besides "one-shot through the network". I think the thing about the search formu
36.
by gugagore 1y ago
What you're describing — search + NN — is presently under the term "test-time compute". The rules / dynamics / objectives of chess ( and Go ) are trivial to encode in a search formulation. I personally don't re
37.
by gugagore 1y ago
That's not a joke. It's a perspective that is seriously held: https://existentialtype.wordpress.com/2011/03/19/dynamic-lan... (Not everyone agrees, for sure, but that's different from it being
38.
by gugagore 1y ago
I'll also point out what is most important part from your original message: > LLMs have hidden state not necessarily directly reflected in the tokens being produced, and it is possible for LLMs to output tokens in opposition to this
39.
by gugagore 1y ago
I'll say it once more: I think it is useful to distinguish between autoregressive and recurrent architectures. A clear way to make that distinction is to agree that the recurrent architecture has hidden state, while the autoregressive
40.
by gugagore 1y ago
The concept of "state" conveys two related ideas. - the sufficient amount of information to do evolution of the system. The state of a pendulum is it's position and velocity (or momentum). If you take a single picture of a pe
41.
by gugagore 1y ago
Do you appreciate a difference between an autoregressive model and a recurrent model? The "transformer" part isn't under question. It's the "hidden state" part.
42.
by gugagore 1y ago
That's not what "state" means, typically. The "state of mind" you're in affects the words you say in response to something. Intermediate activations isn't "state". The tokens that have already be
43.
by gugagore 1y ago
I'm not sure what you mean by "hidden state". If you set aside chain of thought, memories, system prompts, etc. and the interfaces that don't show them, there is no hidden state. These LLMs are almost always, to my knowl
44.
by gugagore 1y ago
Right, though the embedding is trivial, the conceptual distinction is not. In Lean, a subtype is a refinement that restricts by proof. In OOP, a subclass augments or overrides behavior. It's composition versus inheritance. The trivial
45.
by gugagore 1y ago
"What is ROS" is really the right question. Maybe this helps: https://goosetaco.notion.site/Beware-of-negative-polarity-st... . It links to https://answers.ros.org/question/12230/what-is-
46.
by gugagore 1y ago
FYI the use of "subtype" here does not, as far as I know, have much connection to the concept in class-based object oriented programming languages. https://lean-lang.org/doc/reference/latest/Basic-Ty
47.
by gugagore 1y ago
The post is from 2018, but it's very topical because of "abundance", as you point out. You might be interested in this podcast Bad Faith: Episode 478 - The Abundance Conspiracy (w/ Sandeep Vaheesan, Isabella Weber, &amp
48.
by gugagore 1y ago
In idealizations, there are both voltage sources and current do m sources.
49.
by gugagore 1y ago
Right, why is it a 4-connected grid instead of 8-connected, or any other topology, like a hex grid.
50.
by gugagore 1y ago
You don't need all the natural numbers for that, though. I think you need 0 and 1 only? I think there are two primitive sets for dependent type theory. The one with omega, and then the one with inductive types. None of them need axioms
51.
by gugagore 1y ago
I was talking to someone about inductive data types, and showed them the zero/succ definition of `Nat`, e.g. in Lean or Rocq. It was interesting because they were asking "is this all you need? What about the Peano axioms? Is there
52.
by gugagore 1y ago
Does video these days really ever exist uncompressed other than when it's in transit, like through an HDMI cable or in a frame buffer? Does anyone store uncompressed video these days? Honest question.
53.
by gugagore 1y ago
Do you have go-to multi repo automations and other tools for operating across multiple repos?
54.
by gugagore 1y ago
> I bring up Godel just to show that static typing cannot be a panacea. You are necessarily going to have false positives or false negatives or both. If you're OK with that, fine, more power to you. But don't judge me if I&#x27
55.
by gugagore 1y ago
> But I don't think of these as programming language I don’t want to get too caught up in what counts as a programming language, but you can absolutely write programs in a language like Lean, e.g. computing factorial 10. The fact th
56.
by gugagore 1y ago
I'm not sure how familiar you are with Julia, but it is very spiritually aligned with lisp. The reason for the DomainError is "type stability" and it's not stupid ideology unless you consider high-performing numerical co
57.
by gugagore 1y ago
> The intrinsic (Church) view builds meaning into the syntax itself. I shouldn’t have phrased it that way --- the distinction between syntax and semantics is awkward for me, especially when there are discussions of "static" vs
58.
by gugagore 1y ago
Ah, your example reminds me of a quirk in Julia where some methods are type stable, and others are not. (This is just an aside. I'll provide a response in another comment.) ``` julia> sqrt(-1.0) ERROR: DomainError with
59.
by gugagore 1y ago
> Those concerns seem worlds away from the kinds of things type theorists care about. It’s true there’s a tradition, voiced by Dijkstra, that emphasizes correctness at a point in time, sometimes at the expense of long-term adaptability.
60.
by gugagore 1y ago
I agree with everything you said, and just wish you'd respond to the strongest points, not the weakest ;) Specifically, the two views of typing. But my fault for linking to a mediocre blog post.

More ›