sdenton4

44 ms·

601.
by sdenton4 3y ago
This method is for training. They are using a stop-gradient to 'shield' some tokens from contributing to prediction of the immediate next token, and thus producing a stream of tokens that are only used for longer term prediction.
602.
by sdenton4 3y ago
LLMs are experts at everything except what the user is an expert in.
603.
by sdenton4 3y ago
Bombing civilians and blocking food food aid also isn't nice. Sitting in an office is much nicer.
604.
by sdenton4 3y ago
Yes... That's what protest looks like.
605.
by sdenton4 3y ago
So... Is anyone writing a JavaScript to GameBoy assembly transpiler?
606.
by sdenton4 3y ago
I met the head of one of the largest French corporations once, and was pretty shocked at the degree of elitism, hierarchy, and concern about prestigious schools... SV is bottom up - you hire smart people and let them collaborate freely. The
607.
by sdenton4 3y ago
I think it's all about the controls... When I played this in the 90's, I had a sidewinder joystick, and then used the keyboard for strafing. Once you master circle-strafing, the ai enemies don't really stand a chance.
608.
by sdenton4 3y ago
I think they just want to advertise all the patents they filed and get bought.
609.
by sdenton4 3y ago
The training slowdown is not really a problem... There's a pretty wide range of robust, good-enough values that don't slow things down much at all. As with all optimizer cruft, the 'optimal' value is going to be problem-
610.
by sdenton4 3y ago
The book 'Endurance', written from interviews with the survivors, is absolutely gripping. My partner and I read the whole thing out loud over the course of about a week - like binge watching a series.
611.
by sdenton4 3y ago
Sadly, deterministic ux code is unable to do everything. So typically we hire people to do the things computers can't do on their own. Those people have a user interface (speech and text), and a tendency to not perfectly execute your i
612.
by sdenton4 3y ago
This is actually how EfficientNet trains, using random truncation of the network during training. It does just fine... The game is that each layer needs to get as close as it can to good output, improving in the previous activation quality.
613.
by sdenton4 3y ago
The horse is a horse. They were a major part of human transportation and trade networks for a very long time. The point is that humans have millennia of experience working with and relying on animal intelligence. Animals actually provide an
614.
by sdenton4 3y ago
There's used to be a note about it on the tensorflow docs; they're keeping the bad default because it's the default, and changing it would potentially change behavior unexpectedly for lots of users.
615.
by sdenton4 3y ago
There's a particular parameter (epsilon) in Adam which is typically set to a bad default which causes instability when the gradient gets sufficiently small. It is far easier to set epsilon to 0.001 or so than muck around with learning
616.
by sdenton4 3y ago
I think there's Adam and "Nothing is obviously substantially better than Adam so why bother?"
617.
by sdenton4 3y ago
Also consider human-horse interaction. The horse was quite useful for most of human history, despite not being perfectly reliable.
618.
by sdenton4 3y ago
Lindsay: Well, did it work for those people? Tobias: No, it never does. I mean, these people somehow delude themselves into thinking it might, but ... But it might work for us.
619.
by sdenton4 3y ago
I tend to think there's an explore/exploit trade-off in model scale. Animals, including humans, have lots of extra neurons when they are young, which are then shed once the learning phase settles down. And this makes sense: thinki
620.
by sdenton4 3y ago
And yet, people build and rebuild in flood plains...
621.
by sdenton4 3y ago
On the other hand, hung lines have a tendency to fall down, get tangled in trees, and start forest fires, so the cost of above ground lines is also high...
622.
by sdenton4 3y ago
Sadly, I don't think I do; I was mostly messing around with pi's ten years ago, and don't seem to have blog entries specifically on the usb disk. The main idea, though, was to set up the os so that the boot folder was on the
623.
by sdenton4 3y ago
Forests do a lot more than hold carbon. And many of these other uses require more than young, fast growing monocultures. Quite a lot of rainfall over land originates as transpiration from forests, for example; they're a key part of the
624.
by sdenton4 3y ago
It was a lot more fun when vc was buying everyone cheap taxis and movie nights.
625.
by sdenton4 3y ago
I imagine they also have significant resources to contribute to dealing with breaches - eg, analysing past cookouts by the bad actor, designing mitigations, etc.
626.
by sdenton4 3y ago
Not sure how things are these days, but it used to be the case that power fluctuations could kill an rpi's sd card. So yeah, you could mitigate with a great power supply and a good surge protector, but in practice I found it hard to fi
627.
by sdenton4 3y ago
If only there were lots of people from not-the-US who wanted to live in the US...
628.
by sdenton4 3y ago
It's a really fun area to work in, but beware that it's very easy to underestimate the complexity. And also very easy to do things which look helpful but actually are not (eg, improving classification on xeno canto, but degrading
629.
by sdenton4 3y ago
In the area in working in (bioacoustics), embeddings from supervised learning are still consistently beating self supervised transformer embeddings. The transformers win on held out training data (in-domain) but greatly underperform on nove
630.
by sdenton4 3y ago
Look at how neural voice synthesis has evolved over the last five years. It's gone from science fiction to extremely expensive to 'can run in real time on a phone' thanks to a combination of hardware and algorithmic improveme

More ›