5 ms·
I tried understanding the gist of the paper, and I’m not really convinced there’s anything meaningful here. It just looks like a variation of the transformer ar
by alyxya 1y ago
I tried understanding the gist of the paper, and I’m not really convinced there’s anything meaningful here. It just looks like a variation of the transformer architecture inspired by biology, but no real innovation or demonstrated results.
> BDH is designed for interpretability. Activation vectors of BDH are sparse and positive.
This looks like the main tradeoff of this idea. Sparse and positive activations makes me think the architecture has lower capacity than standard transformers. While having an architecture be more easily interpretable is a good thing, this seems to be a significant cost to the performance and capacity when transformers use superposition to represent features in the activations spanning a larger space. Also I suspect sparse autoencoders already make transformers just as interpretable as BDH.
- jimbo808 1y agoThere isn't. The title is totally clickbait. Anything "brain-like" that fits into one single paper is bullshit.
- ljlolel 1y agoA real scientist wouldn’t use an imprecise term like “brain-like”
- dxtrous 1y agoYou can, of course, use the almost equivalent scientific-sounding Latin-derived term ("neuromorphic"), buy popcorn, and come back with it for a discussion about memristors.
- astroflection 1y agoThe actual paper's title: "The Dragon Hatchling: The Missing Link between the Transformer and Models of the Brain" Don't berate the authors for the HN submitter's carelessness.
- jimbo808 1y agoThe actual title is just as click baity
- nickpsecurity 1y agoThat last line isn't true. To be brain-like, it only needs to imitate one thing in the brain. That thing is udually tested in isolation against observed results in human brains. Then, people will combine multiple, brain-inspired components in various ways. That's standard in computational neuroscience. Our standard should simply be whether they are imitating an actual structure or technique in the brain. They usually mention that one. If they don't, it's probably a nonsense comparison to get more views or funding.
- jimbo808 1y agoI am genuinely baffled by this reply. Every single sentence you've typed is complete and utter nonsense. I'm going to bookmark this as a great example of the Dunning-Kruger effect in the wild. Just to illustrate the absurdity of your point: I could claim, using your standard, that a fresh pile of cow dung is brain-like because it imitates the warmth and moistness of a brain.
- nickpsecurity 1y agoI'll ignore the insults and rhetoric to give some examples. The brain-inspired papers have done realistic models of specific neurons, spiking, Hebbian learning, learning rates tied to neuron measurements, matched firing patterns, done temporal synchronization, hippocampus-like memory, and prediction-based synthesis for self-training. Brain-like or brain-inspired appears to mean using techniques similar to the brain. They study the brain, develop models that match its machinery, implement them, and compare observed outputs of both. That, which is computational neuroscience, deserves to be called brain-like since it duplicates hypothesized, brain techniques with brain-like results. Others take the principles or behavior of the above, figure out practical designs, and implement them. They have some attributes of the brain-like models or similar behavior but don't duplicate it. They could be called brain-inspired but we need to be careful. Folks could game the label by making things that have nothing to do with brain-like models or went very far away. I prefer the be specific about what is brain-like or brain-inspired. Otherwise, just mention the technique (eg spiking NN) to let us focus on what's actually being done.
- dang 1y ago(Submitted title was "A Brain-like LLM to replace Transformers" - a mod changed it later, in keeping with the title rule at https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html)
- busssard 1y agoThis is like at the beginning or the end of the Crypto Bubble. Publish a whitepaper for the next model architecture and hope that uninformed people with money blow it up your companys... i mean blow up the economy.... i mean blow , ahh whatever you know
- raincole 1y ago> the end of the Crypto Bubble BTC literally hit all time high this month, fyi.
- cootsnuck 1y agoWhat's your point? House prices are at all time highs too. That doesn't mean the housing bubble never happened.
- AnimalMuppet 1y agoMeans it's not the end of the crypto bubble. Unless you're going to claim that previous large drops in crypto were perhaps bubbles, but this time it's real...
- jimbo808 1y agoPrices have risen by orders of magnitude, untethered to any measurable fundamentals, then crashed, multiple times. I'm not sure what other definition of bubble you're operating with...
- AnimalMuppet 1y agoI'm not arguing that it's not a bubble. I'm conceding that cootsnuck may be claiming that this time it's not a bubble. But if that's not the claim, then I'm saying that the current value makes it's clear that it's not the end of a bubble.
- DaveZale 1y ago
- oofbey 1y agoAttention mechanisms are wonderfully interpretable as is. You can literally see which tokens each token is attending to. People don’t bother much these days. But that’s not a strong selling point.