jychang

19 ms·

241.
by jychang 1y ago
> long-time Assahi Linux leader Well, it's a shit edited article, but at least it's not AI shit
242.
by jychang 1y ago
That sounds like not the psychologist’s fault. Don’t blame the doctor if their patient makes a sprained ankle their identity. If someone smokes a lot of cigarettes and gets lung cancer, we don’t blame the doctor for the lung cancer. This s
243.
by jychang 1y ago
Well llama.cpp requires minimum CUDA 11 from 2020, or if you need CUDA C++17 support, then CUDA 12 from 2022. It'll compile fine against latest CUDA 12.8 or 12.9 etc, but there's zero need to pack whatever the latest CUDA version
244.
by jychang 1y ago
> here's never a size where the model cannot recognize faces at all True > then you add just a single neuron that recognizes them perfectly Not true. Don't think in terms of neurons, think in terms of features. A feature can
245.
by jychang 1y ago
That’d break a lot of GPU setups
246.
by jychang 1y ago
Why not? It takes models of a certain size to contain xyz neuron/feature. https://www.youtube.com/watch?v=AgkfIQ4IGaM That's not a mirage, it's clearly capability that a smaller model cannot demonstrate. A mo
247.
by jychang 1y ago
? llamacpp> ls -l \*llama\* -rwxr-xr-x 1 root root 2505480 Aug 7 05:06 libllama.so -rwxr-xr-x 1 root root 5092024 Aug 7 05:23 llama-server That's a terrible excuse, Llama.cpp is just 7.5 megabytes. You can easily
248.
by jychang 1y ago
That's a dumb answer from them. What's wrong with using an older well-tested build of llama.cpp, instead of reinventing the wheel? Like every linux distro ever who's ever ran into this issue? Red Hat doesn't ship the lat
249.
by jychang 1y ago
For inference, Nvidia/AMD/Intel/Apple are all generally on the same tier now. There's a post on github of a madman who got llama.cpp generating tokens for an AI model that's running on an Intel Arc, Nvidia 3090, and
250.
by jychang 1y ago
No, that's not true. Prompt processing just needs attention tensors in VRAM, the MLP weights aren't needed for the heavy calculations that a GPU speeds up. (After attention, you only need to pass the activations from GPU to system
251.
by jychang 1y ago
They're the same price. Both 395 machines (with 128GB of RAM) are $1999.
252.
by jychang 1y ago
You're most likely bottlenecked by memory bandwidth for a LLM. The AMD AI MAX 395+ gives you 256GB/sec. The M4 gives you 120GB/s, and the M4 Pro gives you 273GB/s. The M4 Max: 410GB/s (14‑core CPU/32‑core GPU)
253.
by jychang 1y ago
AMD Ryzen AI Max 395+ as the APU (CPU + GPU), and its corresponding soldered LPDDR5X RAM, which has a super fast memory bandwidth of 256GB/sec. So the CPU and RAM are soldered on the motherboard. You can buy just the motherboard from F
254.
by jychang 1y ago
To be fair, I’m pretty sure the code here was written before modern AI was a thing, back when dinosaurs roamed the earth.
255.
by jychang 1y ago
... I mean, yes? The entire point of local AI is so you can feed your enterprise code into it, that you don't want offloaded to somewhere else. That's the exact perfect use case for Docker, versus something heavier weight like a V
256.
by jychang 1y ago
This is annoyingly Apple-only though. Even though my main dev machine is a Macbook, this would be a LOT more useful if it was a Docker container. I'd still take a Docker container over an Apple container, because even though docker is
257.
by jychang 1y ago
Any way to install this via just a container? Similar to a `docker compose up -d` that a lot of projects offer. Just download the docker-compose.yml file into a folder, run the command, and you're running. If you want to delete everyth
258.
by jychang 1y ago
Honestly, this is probably the easiest and most accurate answer. India being a british colony for a long time means that the top 1% of Indians had a lot more exposure to English than the top 1% of Chinese people. From there, statistics come
259.
by jychang 1y ago
They implement more openai-compatible endpoints than ollama at least
260.
by jychang 1y ago
That’s just llama-swap and llama.cpp
261.
by jychang 1y ago
Who cares what Hacker News wants? You’re not obliged to participate in discussion. Am I supposed to spend money on Amazon.com when I visit the website just because Amazon wants me to?
262.
by jychang 1y ago
I fail to see how prejudice against waste is a problem. Prejudice is a bad thing- for things that people can't change, like their race or age. Prejudice against people making bad or wasteful decisions is a good thing.
263.
by jychang 1y ago
https://www.google.com/maps/place/50°40'00.0"N+156°07'00.0"E... That port right next to the water has probably disappeared.
264.
by jychang 1y ago
Yeah, this is why wikipedia is not a good resource and nobody should use it. Also why google is not a good resource, anybody can make a website. You should only trust going into a library and reading stuff from microfilm. That's the on
265.
by jychang 1y ago
I doubt that, unless you're willing to pay USA workers Taiwan salaries.
266.
by jychang 1y ago
Inspired me to write this, since it seems like most people don't understand how fast models run: https://unexcitedneurons.substack.com/p/how-to-calculate-hom...
267.
by jychang 1y ago
How many channels of DDR4 ram? What speed is it running at? DDR4-3200? The (approximate) equation for milliseconds per token, is: Time for token generation = (number of params active in the model)*(quantization size in bits)/8 bits*[(p
268.
by jychang 1y ago
What cluster do you have to do the quantizing? I'm guessing you're not using a single machine with a 3090 in your garage.
269.
by jychang 1y ago
ChatGPT is wrong. Here's Deepseek R1 running off of RAM at 8tok/sec: https://www.youtube.com/watch?v=wKZHoGlllu4
270.
by jychang 1y ago
In the current era of MoE models, the system RAM memory bandwidth determines your speed more than the GPU does.

More ›