shawntan

11 ms·

61.
by shawntan 3y ago
Just as a general response to the objections to this: The goal was to use XOR and PARITY as simple tasks and formal languages that exhibit a larger problem with the architecture. The unfortunate side-effect of this is that the most straight
62.
by shawntan 3y ago
Damn I've been meaning to fix that.
63.
by shawntan 3y ago
Yeah the issue tends to be as the length increases the performance drops off. Your approach reminds me of Hattie's work: https://arxiv.org/pdf/2211.09066.pdf I do wonder if the observable drop-off in performance
64.
by shawntan 3y ago
I guess I should say the recurrence growing as a function of the input size is something people don't seem to be willing to afford, not specifically the linear dependence. Not to mention parity here is a toy problem to highlight the is
65.
by shawntan 3y ago
I'm the original author of this blogpost. I've had discussions that range from "we need to fix this" to "maybe human thought isn't Turing complete..." and I'm interested in hearing what yall think ab
66.
by shawntan 3y ago
Yea this is a good solution so long as the task that needs to be outsourced to, say a python interpreter, is explicitly known and can be prompt engineered.
67.
by shawntan 3y ago
Recurrency will help, see the empirical results in this paper: https://arxiv.org/abs/2207.02098 . LSTMs can solve the PARITY problem, generalising to longer lengths of the problem without issue. The issue with re-intro
68.
by shawntan 3y ago
There's more detailed examination of Transformers from circuit complexity in this paper: https://arxiv.org/abs/2106.16213 There are a few references also in the blog post.
69.
by shawntan 4y ago
https://twitter.com/tanshawn/status/1599442780384014346
70.
by shawntan 4y ago
I was personally quite appalled that so many people believed this was what it was doing, but I suppose since the source was a DeepMind researcher, it lends a lot of credibility. By the same reasoning, I hope people do realise that there are
71.
by shawntan 4y ago
Predicting a set of answers to some confidence interval would still result in hallucinated answers.
72.
by shawntan 4y ago
Isn't the CNN a discovery in itself? Without it, we'd be following the bitter lesson and "leveraging computation" to throw more data / compute at an MLP. Clearly someone felt that there'd be a better inductive
73.
by shawntan 4y ago
I've seen many interpretations of this article and I'm curious as to the mainstream CS reading of it. One could look at the move from linear models to non-linear models or the use of ConvNets (yes I know ViTs exist, to my knowledg
74.
by shawntan 4y ago
I'm interested to hear what these techniques are. Decreasing the generality will help, but I fail to see how that scopes the output. At best that mitigates the errors to an extent.
75.
The New XOR Problem(blog.wtf.sg)
4 points by shawntan 4y ago|2 comments
76.
by shawntan 4y ago
I think part of the issue is what level of explanation is satisfactory. We can explain how every linear transformation computes its output, but the sum of it is in many ways more than its parts. Then there are efforts that look like this on
77.
by shawntan 4y ago
Ya I guess I was comparing the difficulty of learning to "produce mostly grammatically correct sentences in most cases" to continual learning. From the 'inside' it feels like everything OP said is just the opposite.
78.
by shawntan 4y ago
Grammar is surprisingly easy to learn from unstructured data, to an extent. (source: I spent a lot of my PhD doing this kinda thing.) Continual learning seems to be a tough problem though, from what I'm seeing of my friends working on
79.
by shawntan 4y ago
How I imagine what you're thinking of is a kind of small update to an existing model by doing backpropagation training on 'new' data. There are existing problems with that, one keyword to lookup is 'catastrophic forgetti
80.
by shawntan 4y ago
Like the earlier responses said, if it is retrieving information from an external data source, that would be a much easier way to update things. REALM ( https://arxiv.org/abs/2002.08909 ) comes to mind as an example of a
81.
by shawntan 5y ago
> Malaysia wasn't hostile to an anti-colonial Singapore, as this sentence suggests. Not that Singapore in 1965 was necessarily anti-colonial. Malaysians had fought the British for independence and there was a cease fire by 1965. The
82.
by shawntan 5y ago
It's definitely not to the extent that Israel is surrounded by states hostile to it, but I think there is some grounds to claim this, depending on how far back you want to go: https://en.wikipedia.org/wiki/MacDonal
83.
by shawntan 5y ago
We consider Malaysian a nationality, and Malay an ethnicity. I can see how it's confusing since Chinese and Indian are also nationalities.
84.
by shawntan 5y ago
Love the project. nostalgia++
85.
by shawntan 5y ago
Does the MissingNo glitch work?
86.
by shawntan 12y ago
Nitpicking here, but while the authors do use a recurrent neural net (RNN), they do not use it exclusively. The system consists of a memory element, and a controller element. In their evaluation of the system, they use both a standard feed-
87.
by shawntan 12y ago
The paper discusses putting the NTM through several tasks, and tests for "overfitting" or how well it has generalised the task by giving it a slightly longer task than it has seen during training. For example, in the copy task, th
88.
by shawntan 14y ago
Hi, I'm Shawn, and I work on categorization at Semantics3. We use Naive Bayesian classifiers and other heuristics for categorization to help us with the disambiguation of products (merging multiple sources referring to the same product into