jsenn

8 ms·

61.
by jsenn 3y ago
This was also where my mind went, and it's cited in the open letter, but it's worth noting that the premise of his critique is that IIT is not pseudoscience. In fact, he praises the theory for being clear enough to be proven wro
62.
by jsenn 3y ago
This might give a decent estimate of the entropy of the file, but it is a poor approximation of the Kolmogorov complexity in general. All major compression libraries that I know of essentially work by "factoring out" statistical r
63.
by jsenn 4y ago
That makes sense. So ChatGPT has to use some cleverness to present the relevant previous text in the current prompt, but the language model is still responsible for pulling out the meaning of that context and responding appropriately. I rem
64.
by jsenn 4y ago
> that the context-awareness is just a very well done illusion Could you expand on this? The context awareness is the part that blows my mind. I would be disappointed/fascinated to find out that it’s “just” a simple set of tricks!
65.
Squeaking Labyrinths (2022)(jsenn.github.io)
1 points by jsenn 4y ago|1 comments
66.
by jsenn 4y ago
A fun post I wrote last year about an unexpected acoustic feature of those brick labyrinths you see in public parks.
67.
by jsenn 4y ago
The software is called MassMotion: https://www.oasys-software.com/solutions/pedestrian-simulati... . > Given that formation is the (density * speed * wiggleness) I assume boundary variability will have a logistic rel
68.
by jsenn 4y ago
No, there really is a neural predisposition to language--not particular sounds, but language. I looked at the Wikipedia page on feral children. Many in fact do learn language when they're rescued. The ones that don't may bark like
69.
by jsenn 4y ago
Actual paper is here: https://www.science.org/doi/10.1126/science.add8091 . Very cool result. This is the gist: > We introduce a theoretical approach that uses temporal coarse-graining akin to Einstein’s kinetic
70.
by jsenn 4y ago
It may be true that the brain can route around damage to be able to produce language using different brain structures than are typically used for that purpose, but what will never happen (as far as I know) is that the brain will spontaneous
71.
by jsenn 4y ago
> But yeah they won't produce birdsong. There's no utility to that. I guess it's a matter of environment. This is the crux of the issue. GPT-3 would happily learn birdsong instead of human language, just like it has lear
72.
by jsenn 4y ago
Sure, but the type of "language" that includes HTML and C++ is very different from the type of "language" that includes English and French. Chomsky's point is that there's something special about human brains t
73.
by jsenn 4y ago
My understanding is that they work well as arbitrary sequence predictors. For example, they can write HTML markup or C++ code just as easily as they can write English sentences. If you trained them on character sequences other than text fro
74.
by jsenn 4y ago
Interesting paper, thanks for linking. I think this is slightly tangential to the Norvig-Chomsky controversy though. What Mumford is saying is that probability and statistics is a more useful basis for modelling natural phenomena than clas
75.
by jsenn 4y ago
This is true, and interesting, but doesn’t address Chomsky’s concerns. While a LLM has structure, it’s still not as structured—or structured in the same way—as the human language faculty. This is easy to see by observing that LLMs can and d
76.
by jsenn 4y ago
This is interesting stuff--thanks for writing it out. > ...super-Turing computational power. Could you expand on this? The Wikipedia page claims that the deterministic version can be efficiently simulated by a Turing machine. The Wikiped
77.
by jsenn 4y ago
Here's a paper that shows how to augment a recurrent NN with a stack: http://arxiv.org/abs/1503.01007 . I'm not an expert, so some or all of this may be wrong, but my understanding of how it works is this. You
78.
by jsenn 4y ago
* The Art of Probability by Hamming. An opinionated, slightly quirky text on probability. Unlike the text used in my university course its explanations were clear and rigourous without being pedantic. The exercises were both interesting and
79.
by jsenn 4y ago
I know what you mean about the bad writing, but I don’t think that makes it a terrible book. Unfortunately most sci-fi has flat writing for the same reason most great literature recycles the same ideas and concepts: it’s hard to do everythi
80.
by jsenn 4y ago
If you haven't read Ted Chiang's short stories yet, do so. His first collection (Stories of Your Life and Others) was my favourite, but his more recent one (Exhalation) was also very good (the title story, Exhalation, is gorgeous
81.
by jsenn 4y ago
* Three Body Problem. I've heard of it many times, but never read it. Picked it up a month or two ago, read the whole thing in a few sessions, one of my favourite sci-fi books ever. Will read the next book in the series. * Immune: A Jo
82.
by jsenn 4y ago
You might also like Entangled Life by Merlin Sheldrake. It's focused on fungi rather than trees, but has a similar outlook. Overall it's a bit more down to earth and factual (or at least it seems that way to me, having no special