tbalsam

19 ms·

151.
by tbalsam 3y ago
Memorization of individual examples -> generalization, I can't speak about the determinant of switching as that is (partially, to some degree) work I'm working on, and I have a personal rule not to share work in progress until
152.
by tbalsam 3y ago
I think you may be missing the extensive lines of research covering those topics. Memorization vs Generalization has been a debate before LW even existed in the public eye, and inputs that networks have unusual sensitivity to have been well
153.
by tbalsam 3y ago
For more information and the related math behind associative memories, please see Hopfield Neural Networks. While the upper bound is technically "infinity", there is a tradeoff between the amount of concepts stored and the fundame
154.
by tbalsam 3y ago
I would start with Shannon's information theory and the Wikipedia page on L2/the MDL as a decent starting point. For the first, there are a few good papers that simplify the concepts even further.
155.
by tbalsam 3y ago
To further clarify things, the reason there is no mystical 'eventual recovery from overfitting ' is because overfitting is a stable bound that is approached. Adding this false denomination to this implies a non-biphasic nature to
156.
by tbalsam 3y ago
Part of the issue here is posting a LessWrong post. There is some good in there, but much of that site is like a Flat Earth conspiracy theory for neural networks. Neural network training [edit: on a fixed point task, as is often the case {s
157.
by tbalsam 3y ago
L1 drops weights to zero, L2 biases towards Gaussianality. It's not always relearning lessons or people entirely blindly trying things either, many researchers use the underlying math to inform decisions for network optimization. If yo
158.
by tbalsam 3y ago
ML researcher here wanting to offer a clarification. L1 induces sparsity. Weight decay explicitly _does not_, as it is L2. This is a common misconception. Something a lot of people don't know is that weight decay works because when app
159.
by tbalsam 3y ago
Few people capture this spirit like the classic "Congratulations" video from Big Man Tyrone. While it has gone through the revolutions from sincere, to sarcastic in intent, to back again, I believe the inherently joyful attitude u
160.
by tbalsam 3y ago
I think this is more the societal layer of it, which is definitely important. There's a lot of wannabe "Impact of AI" people so I tend to be generally distrustful of that space due to the huckster coefficient. I think the bas
161.
by tbalsam 3y ago
This is a very impassioned post, but I'd note as a researcher who's been working in the field for a while -- LLMs achieve to a degree an approximation of the basis of the task space that generates the distribution at hand under th
162.
by tbalsam 3y ago
Am an ML engineer, was around long before LLMs. Definitely am skeptical, and I think I know where one dimension that we're missing performance is, a number of people do (in regards to steerability/controllability without losing pe
163.
by tbalsam 3y ago
I linked to the main homepage. My goal isn't to debate, but to foster conversation. Building a case in either direction is going to take more than a single patient study discussed on a healthline page or even a metastudy of studies wit
164.
by tbalsam 3y ago
You might find this link very interesting if you're in the mood to explore some of the interesting corners of this topic a bit more: https://med.virginia.edu/perceptual-studies/our-research/nea...
165.
by tbalsam 3y ago
A number is a symbol, sensations are a continuous symbol....it technical is information.
166.
by tbalsam 3y ago
There are some interesting lines of discussion around near death experiences (NDEs) where the individual at hand supposedly did not have brain activity but was conscious. I can't speak to the veracity of claims, but that and the surrou
167.
by tbalsam 3y ago
(and to add to the other comment too you can edit your comment so you don't get dinged for asking about upvotes! ;P It seems very interesting. You can generally share how important it is to you within reason and that's pretty chil
168.
by tbalsam 3y ago
If this were stackoverflow, I'd mark this as an answer. Also answers my question up above a bit as well, just from a different approach. Many thanks for clarifying here and elsewhere! :) HN is a lovely, curious, and interesting place.
169.
by tbalsam 3y ago
> I'm sorry if you interpreted it that way. is probably a good marker that it is time for me take my bow in this conversation, however, for an alternative approach I recommend SideQuark's comment on iterative random variables v
170.
by tbalsam 3y ago
Alright, I tried it in the terminal and the numbers confirmed my earlier math. .5->.25->.125 mean as N->infinity. I chained the python uniform function like the above code in the originally linked post does and averaged the results
171.
by tbalsam 3y ago
How do you mean? I'm going based on the propagation of expectations through the system. The expectation of a uniform distribution is half the bound. Unless there's some math shenanigans going on, I believe that the expected expect
172.
by tbalsam 3y ago
Same, at first at least.
173.
by tbalsam 3y ago
Gotcha, yeah, that makes sense to me now. Maybe the real article was the HN comments section that we made along the way....
174.
by tbalsam 3y ago
I'm a little confused then what the value of this demonstration is. The expectation marches by half the distance towards the outer edge each time, so it becomes a soap bubble at a rate of 1/2^n Which is nice and sorta cool but I&#
175.
by tbalsam 3y ago
What language? What are the parameters for the random function here (seed? upper range value in a uniform distribution? stddev for a random distribution?) Why? What is this demonstrating? I can grok a lot of the dimensional effects of rando
176.
by tbalsam 3y ago
GPT-2's largest model was 1.5B params, LLama-65B was similar to the largest GPT3 in benchmark performance but that model was expensive in the API, a number of the people would use the cheaper one(s) instead IIRC. So this is similar to
177.
by tbalsam 3y ago
Oh, I've been using language models before a lot (or at least some significant chunk) of HN knew the word LLM, I think. I remember when going from 6B to 13B was crazy good. We've just normalized our standards to the latest models
178.
by tbalsam 3y ago
I never thought I'd see the day when a 13B model was casually referred to in a comments section as a "toy model".
179.
by tbalsam 3y ago
Don't forget that this is eventually inherently limited by the Cramér–Rao bound! Wiki link: https://en.m.wikipedia.org/wiki/Cram%C3%A9r%E2%80%93Rao_boun...
180.
by tbalsam 3y ago
I think this is a good take that I don't need to add much to, to be honest. Maybe this doesn't add as much to the discussion as vehement agreement or disagreement might to the curious HN reader, but I do really personally apprecia

More ›