gdiamos

8 ms·

91.
by gdiamos 1y ago
Users often found it hard to know exactly where the boundaries are. This is a reason why general purpose models shine. You don’t have to carefully characterize a task and put guard rails around it.
92.
by gdiamos 1y ago
llm.finetune(data) is a leaky abstraction Read Andrej’s blog that I linked earlier in the thread if you want to understand why.
93.
by gdiamos 1y ago
I think you are saying to go after the very high end of the market. That’s fair, one market segment of this is sometimes called sovereign compute. Another common model that I have seen is to become the deepmind for one very large and import
94.
by gdiamos 1y ago
Fair didn’t seem to matter to users who just wanted to build solutions with reasonable time and budget
95.
by gdiamos 1y ago
We used the best models available and went from the Pythia/gpt2 to Deepseek generations. One annoying part was switching to new and better models that came out literally every week. I don’t think it substantially changes anything. If a
96.
by gdiamos 1y ago
How can you hire enough people to scale that while making the economics work? Why would they join you rather than founding their own company?
97.
by gdiamos 1y ago
Try doing a head to head comparison using all LLM tricks available including prompt engineering, rag, reasoning, inference time compute, multiple agents, tools, etc Then try the same thing using fine tuning. See which one wins. In ML class
98.
by gdiamos 1y ago
They will hire anyone who can produce a model better than GPT5, which is the bar for fine tuning Otherwise, you should just use gpt5 Preparing a few thousands training examples and pressing fine tune can improve the base LLM in a few situat
99.
by gdiamos 1y ago
Just coming out of founding one of the first LLM fine tuning startups - Lamini - I disagree Our thesis was that fine tuning would be easier than deep learning for users to adopt because it was starting from a very capable base LLM rather th
100.
by gdiamos 1y ago
Recursion is so popular in computing that this term “recursive language model” is heavily overloaded It was even before the rise of LLMs The authors may want to consider a more specific name
101.
by gdiamos 1y ago
I wonder if this depends on what is inside the domain specific data. I’m happy to see ML papers on hacker news.
102.
by gdiamos 1y ago
They should make the challenge to translate all of Linux and then maintain it
103.
by gdiamos 1y ago
LLM app telemetry is important, but I don’t think we have seen the right metrics yet. Nothing has convinced me that they are more useful than modern app telemetry I don’t think tool calls or prompts or rag hits are it That’s like saying tha
104.
by gdiamos 1y ago
Some EA IP I remember Battlefield Mass effect Dragon age Madden The sims Titanfall
105.
by gdiamos 1y ago
It forces us to confront a question. How much investment and prioritization in scaling laws is justified?
106.
by gdiamos 1y ago
How can the community tell if models overfit to these benchmarks?
107.
by gdiamos 1y ago
LLMs have finite entropy (it is related to their training loss) and training typically doesn’t store the residuals. Some compression methods use LLMs internally and also store the residuals, making them lossless.
108.
by gdiamos 1y ago
There are some fun early theoretical ML papers on this topic. They prove that it is possible to fully clone a brain based on this method. I think one could theoretically estimate how many queries you would need to make to do it. The worst c
109.
by gdiamos 1y ago
It would be an expensive experiment to perform. I wonder how you could do it more efficiently?
110.
by gdiamos 1y ago
I think it would be interesting to deflate out to a huge dataset and see where this happens. Certainly it will occur as the generated data exceeds the original, eg after 1-10T tokens. I think you could also do this faster by moving down the
111.
by gdiamos 1y ago
What open source code do you use to pull synthetic data from LLMs?
112.
LLM-Deflate: Extracting LLMs into Datasets(scalarlm.com)
77 points by gdiamos 1y ago|39 comments
113.
by gdiamos 1y ago
best news i've heard in days
114.
by gdiamos 1y ago
It looks like it was not included in this video. Here is a link to a previous video - https://www.youtube.com/watch?v=Q7EuXNw0DaA&list=PL240A9CB42... I think it is better if he explains it
115.
by gdiamos 1y ago
I had to rewatch his point about stage fright at the end several times before I finally got it
116.
by gdiamos 1y ago
ptxas is a bit of a misnomer - it actually wraps the entire NVIDIA driver backend compiler PTX isn’t the assembly language, it is a virtual ISA, so you need a full backend compiler with 10s to 100s of passes to get to machine code
117.
by gdiamos 1y ago
Their idea is that capacity of even 4096-wide vectors limits their performance. Sparse models like BM25 have a huge dimension and thus don’t suffer from this limit, but they don’t capture semantics and can’t follow instructions. It seems li
118.
by gdiamos 1y ago
It's so good it is worth revisiting often
119.
by gdiamos 1y ago
Arm makes a specification and standard (the ARM ISA). Apple licenses that and develops their own chip, which is then manufactured by TSMC. So I guess if Intel dies the US will still have a few good CPU design firms, but no manufacturing Als
120.
by gdiamos 1y ago
Scaling laws enabled an investment in capital and GPU R&D to deliver 10,000x faster training. That took the wold from autocomplete to Claude and GPT. Another 10,000x would do it again, but who has that kind of money or R&D breakthro

More ›