daxfohl

19 ms·

211.
by daxfohl 1y ago
Yeah, I have to imagine animal brains are just giant Fourier transform engines under the hood, and humans brains have just evolved to make some frequencies more precise.
212.
by daxfohl 1y ago
Reminds me of the description of Peter Scholze as he was coming up with condensed mathematics. Didn't write a thing until he had it all worked out in his head (which is how he always works). Knew if he didn't get it worked out bef
213.
by daxfohl 1y ago
Yeah, kind of. Though it seems like humans do some post-analysis on the reasoning and kind of "root cause" the path to the result. Fine tuning on the raw thought output seems like it could capture a ton of unnecessary noise. I don
214.
by daxfohl 1y ago
I'd guess it works like any form of learning a subject. The better you have internalized the fundamentals, the better you can perform at higher level tasks. Though to that end, I wonder if the model "knows" that it "unde
215.
by daxfohl 1y ago
That's kind of tangential though. The article is more about using sandboxes to allow `--dangerously-skip-permissions` mode. If you're not looking at the generated code, you're correct, sandboxing doesn't help, but neithe
216.
by daxfohl 1y ago
I've seen research that shows that starting with reasoning models, and fine-tuning to slowly remove the reasoning steps, allows you to bake the reasoning directly into the model weights in a strong sense. Here's a recent example,
217.
by daxfohl 1y ago
Yeah but IIUC they're both just representations of embeddings in a latent space, translated from one format to another. So if the image interpretation of a text embedding is full of hallucinations, it's unlikely that the other dir
218.
by daxfohl 1y ago
It seems like we're still pretty far away from that being viable, if chatgpt is any indication. Whenever it suggests "should I generate an image of that <class design, timeline, data model, etc>, it really helps visualize it
219.
by daxfohl 1y ago
True, but LLMs are already really good at that kind of thing. Even back in 2015, before transformers, here's a karpathy blog post showing how you could find specific neurons that tracked things like indent position, approx column loc
220.
by daxfohl 1y ago
I wouldn't think it would be good for coding assistants, or things where character precision is important. OTOH maybe the information implied by syntax coloring could make syntax patterns easier to recognize and internalize? Once inter
221.
by daxfohl 1y ago
If anyone wants to buy me some GPU time I'd be happy to try it out! Fair warning: my only experience in deep learning thus far was training a CNN to count dots on an image, which worked semi reliably up to 8, when the image was perfect
222.
by daxfohl 1y ago
Agreed. Basically, open some random piece of code. Try explaining it in English with perfect precision. That's what "coding" in natural language would be like. Way more verbose, open to misinterpretation, harder for engineers
223.
by daxfohl 1y ago
I find the back and forth more productive. Lots of up front requirements and docs is like waterfall. You do everything up front and don't realize until implementation time that the design is unclear, has flaws and contradictions, etc.
224.
by daxfohl 1y ago
Just wait till AI starts taking managerial roles. Instead of annual reviews it'll be hourly reviews. "You only have five background AI coding sessions in progress right now. PIP!"
225.
by daxfohl 1y ago
Most people's minds are already made up. All this does is add some confirmation bias so they can feel better about what they were already certain of. I don't think it fundamentally changes anyone's opinions.
226.
by daxfohl 1y ago
I wonder if a linear-space, constant-time model like RWKV or S4 would work better here. For audio, I wouldn't think you'd need long range context, and all-to-all mapping seems like overkill. Maybe a transformer could be running in
227.
by daxfohl 1y ago
Another interesting thing here is that the model presumably has some understanding of the passage of time. That's one thing that can be odd about chat models, in that they will respond the same no matter whether you respond a second la
228.
by daxfohl 1y ago
Definitely sounds cool. But the problem hasn't even been solved locally yet. Distributed microservices, 3rd party dependencies, async callbacks, reasonable test data, unsatisfiable validations, etc. Every company has their own hacked t
229.
by daxfohl 1y ago
I was wondering recently, is fine tuning an effective way to make this the default? If so, does fine tuning this behavior on one language have a carry-over effect on other languages (maybe even non-programming language?), or is the effect l
230.
by daxfohl 1y ago
Have you heard of any attempts to bake MCP definitions into LoRA adapters? I've been wondering if that's a viable approach, so you don't have to put them all in context, and toggling them on and off would just be a matter of
231.
by daxfohl 1y ago
I imagine it's pretty bad risk to reward ratio for most companies. Especially when just tossing some stuff into your system prompt is an option.
232.
by daxfohl 1y ago
Could you use LoRA adapters to free up your context with all the stuff that normally has to go into it? Coding standards and fuzzy preferences like "prefer short names" or "prefer functional style", reference materials,
233.
by daxfohl 1y ago
This is nothing new. v1 of something is always easy. Now in the coding assistant world, everything is v1. Let the tools go through a few deprecation cycles, weird things being added that don't map to the original intent, floods of over
234.
by daxfohl 1y ago
Maybe. At least that way it doesn't feel like some separate half-baked proprietary code editor from the 90s, which is the vibe I get from Jupyter. But a regular .py file with a couple attributes that can generate notebooks still sounds
235.
by daxfohl 1y ago
Actually I don't even think connecting AWS services to each other is a good reason in most cases. I've seen too many cases where things like this start off as a simple solution, but eventually you get a use case where some s3 upda
236.
by daxfohl 1y ago
I was just thinking about something like this a couple hours ago. Jeremy Howard (fast.ai, etc) posted a few weeks ago about his new AI class [1] and I was pumped. I went through the fast.ai course a couple years ago and had been wondering i
237.
by daxfohl 1y ago
Reading the article, I don't think it has AI. They've just made the tools in a way that AI assistants can also use them, and so fix linting errors without anyone needing to fine-tune the LLM on the syntax. That's actually p
238.
by daxfohl 1y ago
Well, I guess one can take one more step back and say this is all merely an example of "premature optimization is the root of all evil". Unless you know a-priori that you have some very hard latency requirements, start with someth
239.
by daxfohl 1y ago
So it's not really leftpad that's broken, it's that each language has a different definition of "String.length", generally corresponding to the definition of the underlying encoding, when dealing with accents, emoji
240.
by daxfohl 1y ago
I don't think "serverless is bad" is necessarily the full lesson here. The bigger lesson is when a service has dependencies, moving that service closer to the client (without also moving those dependencies) will counterintuit

More ›