johndough

8 ms·

151.
by johndough 9mo ago
You can run it on consumer grade hardware right now, but it will be rather slow. NVMe SSDs these days have a read speed of 7 GB/s (EDIT: or even faster than that! Thank you @hedgehog for the update), so it will give you one token rough
152.
by johndough 9mo ago
Maybe not as far-fetched as one might think. Linus about the Tux mascot: > But this wasn't to be just any penguin. Above all, Linus wanted one that looked happy, as if it had just polished off a pitcher of beer and then had th
153.
by johndough 9mo ago
> (it doesn't hallucinate on this) But how do we know that you did not hallucinate the claim that ChatGPT does not hallucinate its version number? We could try to exfiltrate the system prompt which probably contains the model name,
154.
by johndough 9mo ago
The entire "Supreme Skills!" series is amazing. Highly recommend!
155.
by johndough 9mo ago
The model absolutely can be run at home. There even is a big community around running large models locally: https://www.reddit.com/r/LocalLLaMA/ The cheapest way is to stream it from a fast SSD, but it will be qui
156.
by johndough 9mo ago
"... except for the problem of too many levels of indirection."
157.
by johndough 9mo ago
Which categories do algorithmic optimizations fall under? For example: Strassen algorithm for matrix multiplication https://en.wikipedia.org/wiki/Strassen_algorithm FFT convolution https://dsp.stackexchange.
158.
by johndough 9mo ago
Those are just examples. The real question is whether the ban produces "legal effects concerning him or her or similarly significantly affects him or her". Maybe someone with legal expertise could weight in here?
159.
by johndough 9mo ago
And even if WebGPU is enabled, the implementation might still be broken or inefficient in various ways. For example, Firefox uses some ridiculous polling-based approach [1] to check for completion, which disqualifies the implementation for
160.
by johndough 9mo ago
I do not believe that LLMs fear punishment like human employees do.
161.
by johndough 9mo ago
> A malicious AI could just add arbitrary code to your Vagrantfile > [...] > Claude could decide to fix/improve something by adding a commit hook. You can fix this by confining Claude to a subdirectory (with
162.
by johndough 9mo ago
> restricting the search space to syntactically valid programs (how do you even randomly generate that?) By using a grammar. Here is an example on how to only generate valid JSON with llama.cpp: https://github.com/ggml-or
163.
by johndough 9mo ago
What do you think the "true cost" is?
164.
by johndough 9mo ago
I had already removed three parameters which were no longer needed, but I hadn't yet heard that the other two had also become superfluous. Thank you for the update! llama.cpp sure develops quickly.
165.
by johndough 9mo ago
I've been running it with llama-server from llama.cpp (compiled for CUDA backend, but there are also prebuilt binaries and instructions for other backends in the README) using the Q4_K_M quant from ngxson on Lubuntu with an RTX 3090:
166.
by johndough 9mo ago
From the top of the linked article: > NVIDIA is also developing its own models, including NeMo, Retro-48B, InstructRetro, and Megatron. These are trained using their own hardware and with help from large text libraries, much like o
167.
by johndough 9mo ago
(>= 6 months of 1 gig of data per day) You can parse JSON at several GB/s: https://github.com/simdjson/simdjson And you could scale that by one or two orders of magnitude with thread-based parallelism
168.
by johndough 9mo ago
To add to that, llama.cpp supports GBNF grammars, which allows generation of JSON, or even programming languages: https://github.com/ggml-org/llama.cpp/blob/master/grammars/R...
169.
by johndough 9mo ago
Determinism of LLMs has often been discussed on HN, for example here: https://news.ycombinator.com/item?id=37006224 https://news.ycombinator.com/item?id=45200925 The TL;DR is that LLMs are often not determi
170.
by johndough 9mo ago
That extension might allow tracking. From their Chrome add-on page: When it's needed for the website to work properly, it will automatically accept the cookie policy for you (sometimes it will accept all and sometimes only necess
171.
by johndough 10mo ago
All the naysayer here have clearly no idea. Your large matrix multiplication implementation is quite impressive! I have set up a benchmark loop and let GPT-5.1-Codex-Max experiment for a bit (not 5.2/Opus/Gemini, because they are
172.
by johndough 10mo ago
In 2021, only the top 8 % of games sold 10k copies or more, so if you were among them, you were quite successful. Source: https://app.sensortower.com/vgi/insights/article/video-game-... In addition, a large f
173.
by johndough 1y ago
If the MPC tools come first in the conversation, it should be technically possible to cache the activations, so you do not have to recompute them each time.
174.
by johndough 1y ago
Generative models can combine different concepts from the training data. For example, the training data might contain a single image of a new missile launcher at a military parade. The model can then generate an image of that missile launch
175.
by johndough 2y ago
Use it or lose it. With the invention of the calculator, students lost the ability to do arithmetic. Now, with LLMs, they lose the ability to think. This is not conjecture by the way. As a TA, I have observed that half of the undergraduate
176.
by johndough 2y ago
I have the skills to write efficient CUDA kernels, but $2/hr is 10% of my salary, so no way I'm renting any H100s. The electricity price for my computer is already painful enough as is. I am sure there are many eastern European de
177.
by johndough 2y ago
Not to belittle Justine's achievements, but the role of the most important software developer probably goes to the maintainer of some hugely important infrastructure project that we barely know about. https://xkcd.com&#x2F
178.
by johndough 2y ago
Promoting HN posts can earn a lot of money I'd wager. The audience tends to be wealthy. Edit: A quick search suggests that one HN upvote costs about 9 cents. I wonder why we don't see even more bots.
179.
by johndough 2y ago
Note that the 'Jump Flood Algorithm' is O(N log N) where N is the number of pixels. There is a better O(N) algorithm which can be parallelized over the number of rows/columns of an image: https://news.ycombinator.c
180.
by johndough 2y ago
Two more heuristics: 1. The figures are not vectorized (text in figures can not be selected). All it takes is to replace "png" in `plt.savefig("figure.png")` with "pdf", so this is a very easy fix. Yet the auth

More ›