llm_trw

33 ms·

541.
by llm_trw 2y ago
Yes, they also fail. I've found the original gpt4 to be the most consistent. One of these days I'll spend the couple of thousands needed to benchmark all the top models and see how they actually perform on a task which can't
542.
by llm_trw 2y ago
Someone posted a paper that I didn't know about, but goes through pretty much all the work I did in the space: https://news.ycombinator.com/item?id=42095999 It's missing the colourful commentary that I'd usua
543.
by llm_trw 2y ago
Thanks for the citation, I did my work in the area around 2014 and never looked back. That's a very good summary of the state of the field as I remember it.
544.
by llm_trw 2y ago
If a model can't inately reason over 5 steps in a simple task but produces a flawless 500 step proof you either have divine intervention or memorisation.
545.
by llm_trw 2y ago
It's my job to build these tools. I'm well aware of their strengths and shortcomings.
546.
by llm_trw 2y ago
The closest explanation to how chain of through works is suppressing the probability of a termination token. People have found that even letting llms generate gibberish tokens produces better final outputs. Which isn't a surprise when
547.
by llm_trw 2y ago
These benchmarks are entirely pointless. The people making them are specialists attempting to apply their skills to areas unrelated to LLM performance, a bit like a sprinter making a training regimen for a fighter jet. What matters is the d
548.
by llm_trw 2y ago
This is a big question that needs a research paper worth of explanation. Feel free to email me if you care enough to have a more in-depth discussion.
549.
by llm_trw 2y ago
I got it from section 2.2 > The number of model parameters is slightly higher in the BitLinear setting, as we both have 1.58-bit weights as well as the 16-bit shadow weights. However, this fact does not change the number of trainable&#x2
550.
by llm_trw 2y ago
There's more to it than that. If you could train models faster, you’d be able to build larger, more powerful models that outperform the competition. The fact that Llama 3 is significantly over trained than what was considered ideal eve
551.
by llm_trw 2y ago
Diffusion requires a lot more computation to get results compared to transformers. Naively when I'm running a transformer locally I get about 30% GPU utilization, when I'm running a diffusion model I'm getting 100%. This mean
552.
by llm_trw 2y ago
None of those papers actually use quantized training, they are all about quantized inference. Which is rather unfortunate as it means that the difference between what you can train locally and what you can run locally is growing ever larger
553.
by llm_trw 2y ago
The police do both. So are they wrongest?
554.
by llm_trw 2y ago
I'm alive because my father did the same. How much value is your life worth?
555.
by llm_trw 2y ago
Computer programmer. You just need to have been in a country that's had a civil war to realise how useful a pile of cash is for getting out quickly.
556.
by llm_trw 2y ago
About a year ago I used a 70b llama 2 fine tune for financial data to play a game of jeopardy to generate training data for a Bert model used in an rag. It worked better than state of the art at the time. This may seem frivolous but it&#x27
557.
by llm_trw 2y ago
In the most general sense yes. However you can encode whatever you want on top of a trs, just like how you can encode any computer language in lambda calculus because both are Turing complete models of computation.
558.
by llm_trw 2y ago
Papers only work because they know exactly what the view portal is and can design the layout relative to that. Unless you have an a3 sized screen this will not work very well online.
559.
by llm_trw 2y ago
Yes, in term rewriting systems the only thing that matters is the lexical structure of the "program" you're running on top of the transform rules. A simple example of running a TRS is a term of a BNF grammar, a less simple on
560.
by llm_trw 2y ago
The cpi is not the price of groceries, that's literally its definition.
561.
by llm_trw 2y ago
This is vastly more pattern matching than term rewriting. In a term rewriting system you have no types for a start: https://en.wikipedia.org/wiki/Rewriting
562.
by llm_trw 2y ago
It's a shame the standard texts are all 20 years old or more than way too heavy mathematically. A little book for term rewriting would be a great new addition.
563.
by llm_trw 2y ago
There are no primaries in Australia.
564.
by llm_trw 2y ago
> I honestly do not believe your grocery bill has tripled. So much the worse for you.
565.
by llm_trw 2y ago
I'm doing great. That doesn't mean I don't notice my grocery bill is three times what it was in 2019 after being pretty much the same from 2009 till then. It's kind of annoying having people tell me this doesn't imp
566.
by llm_trw 2y ago
What's missing is that men in South Korea are expected to spend 2 to 3 years of their life in the army. This was a reasonable tradeoff when women would spend as much time being pregnant - I risk my life to protect you, you risk yours t
567.
by llm_trw 2y ago
>when elections were decided relatively quickly, https://en.wikipedia.org/wiki/Bush_v._Gore I wonder how people who seem to care so much about politics don't spend any time reading about politics beyond rage ba
568.
by llm_trw 2y ago
I've found llm generated code to work amazingly well for visualisation since you can just look and check you're getting the right results.
569.
by llm_trw 2y ago
https://en.wikipedia.org/wiki/M-expression It was never implemented. It was M-expressions which users were to interact with, and S-expressions were what the machine would use.
570.
by llm_trw 2y ago
I imagine it has more to do with the smoking and drinking rates than sugar. I very clearly remember an aunt of mine losing her mind in the 90s about me drinking coke, while smoking when she was pregnant.

More ›