om8

3 ms·

1.
by om8 9d ago
Could've been better if GGUF implemented QTIP format. GGUF representation is a major limitation for llama.cpp quantization performance
2.
by om8 10d ago
> If you’re just using a code book to reconstruct a f16 model the only savings you can get are in sending it over the wire. That’s why you need to use efficient gemm kernels like FLUTE for inference. They are ~as good as what you can do
3.
by om8 10d ago
If you want sub-2 bit llm, get one that’s already trained in higher precision, and compress it with something like YAQA/QTIP with finetuning or PV-tuning + AQLM/HIGGS
4.
by om8 10d ago
Ternary quantization does not make any sense. Vector quantization and trellis based methods are better in this region for PTQ.
5.
by om8 2mo ago
Why int4? There are a lot of superior 4 bit formats like nf4 from bitsandbytes.
6.
by om8 2mo ago
> So how can I regain my ability to focus? Getting diagnosed with ADHD and taking meds did the trick for me. Not sure how relevant this is, since inability to focus was always an issue for me.
7.
by om8 2mo ago
This project needs webgpu -- I did it on cpu about a year ago. My demo uses 2 bit quantization to run llama3 models on any device with enough ram. https://galqiwi.github.io/aqlm-rs/
8.
by om8 3mo ago
> manufacturing companies making the flimsiest, cheapest, plastic crap to save 1/3 of a cent on every mop they produce. Designed to work for the least amount of time before needing replaced We live in a world with such companies, an
9.
by om8 3mo ago
Oh, yes, you are right about watts. My bad
10.
by om8 3mo ago
> volt-amperes are technically the same unit as watts volt-amperes are joules
11.
by om8 3mo ago
Cool way to think about GWh/year: 1 GWh/year = (10 ** 9) / 24 / 365.25 / (10 ** 6) MW = 0.11 MW 70 GWh/year = 8 MW 1755 GWh/year = 200 MW 252 GWh/year = 29 MW
12.
by om8 5mo ago
cargo/uv/go have lock files though
13.
by om8 5mo ago
https://docs.vllm.ai/en/v0.20.0/api/vllm/model_executor/laye... `vllm.model_executor.layers.quantization.turboquant` > The technique implemented here consists of the scalar case of the HIGGS quan
14.
by om8 6mo ago
These are very different media types with very different goals.
15.
by om8 8mo ago
Is there a way to disable it? Sometimes I value agent not having knowledge that it needs to cut corners
16.
by om8 9mo ago
https://steveklabnik.com/writing/does-unsafe-undermine-rusts...
17.
by om8 1y ago
... Yet.
18.
by om8 1y ago
FYI you can run just `uvx pdm`
19.
by om8 1y ago
Also got it, found this thread by googling "ycombiinator"
20.
by om8 1y ago
Also got it, found this thread by googling "ycombiinator"
21.
by om8 1y ago
Have a similar project. Also written in rust, runs in a browser using web assembly In-browser demo: https://galqiwi.github.io/aqlm-rs Source code: https://github.com/galqiwi/demo-aqlm-rs
22.
by om8 1y ago
https://web.archive.org/web/20250914222638/http://ewaste.fka...
23.
by om8 1y ago
Oh, ok. Thanks, I agree
24.
by om8 1y ago
Why? They acquired books, that’s what they do
25.
by om8 1y ago
Good rationalism includes empiricism though
26.
by om8 1y ago
llama.cpp is a mess and ollama is right to move on from it
27.
by om8 1y ago
Of course it is. GPT-5 is one of the most anticipated things in AI right now. To live up to the hype, it needs to be a reasoning model.
28.
by om8 1y ago
It’s unfortunate that llama.cpp’s code is a mess. It’s impossible to make any meaningful contributions to it.
29.
by om8 1y ago
Sure, but integrated graphics usually lacks vram for LLM inference.
30.
by om8 1y ago
To have a gpu inference, you need a gpu. I have a demo that runs 8B llama on any computer with 4 gigs of ram https://galqiwi.github.io/aqlm-rs/about.html

More ›