AaronFriel

7 ms·

31.
by AaronFriel 2y ago
Yeah, that's his channel.
32.
Tell HN: John Friel my father, internet pioneer and creator of QModem, has died
1092 points by AaronFriel 2y ago|98 comments
33.
by AaronFriel 2y ago
Sure, but I think it's the sort of "self-dealing" that's the problem. Suppose it were possible for a wholesaler in Ireland to purchase a product in bulk at around 1/3 of MSRP. Market equilibrium would drive the pric
34.
by AaronFriel 2y ago
Is there pricing available on any of these vendors? Open source models are very exciting for self hosting, but the per-token hosted inference pricing hasn't been competitive with OpenAI and Anthropic, at least for a given tier of quali
35.
by AaronFriel 2y ago
If previous quantization results hold up, fp8 will have nearly identical performance while using 405GiB for weights, but the KV cache size will still be significant. Too bad, too, I don't think my PC will fit 20 4090s (480GiB).
36.
by AaronFriel 2y ago
Blanket accusations like this are racist. Genital mutilation is an abhorrent practice. While the impact is uniquely horrible for women, it is definitely "throwing stones in glass houses" to act as though it's a strange cultur
37.
by AaronFriel 2y ago
A feature I've wanted for ages, for every OS package manager (Windows, apt, yum, apk, etc.), every language's package manager (npm, pypi, etc.), and so on is to update but filter out anything less than one day, one week, or one mo
38.
by AaronFriel 2y ago
This is incorrect, the documentation in the article involves configuring an L7 load balancer to route a path on the same domain as the origin to Google Tag Manager. This means even `SameSite=strict`, `Secure`, `HttpOnly` cookies will be s
39.
by AaronFriel 2y ago
> Override the Host header to be equal to GTM-123456.fps.goog. Allow all cookies and query strings to be forwarded. Did a security team review this? This leaks session cookies for your domain to Google in a way GTM did not previously cap
40.
by AaronFriel 2y ago
> Imagine you are writing performance sensitive code. You want to get a substring from a string, one that is not going to live outside your hot loop. In standard C you can just reference a part of a string with a pointer offset. If you w
41.
by AaronFriel 2y ago
In autoregressive models we can "feed forward" the model by injecting additional tokens. Computing the KV cache entries for those tokens (called"prefill"), then resuming decoding. If we can do this quickly, and on the sa
42.
by AaronFriel 2y ago
Because the model has been trained to do what you tell it to do? That's what instruction pretraining/fine-tuning is.
43.
by AaronFriel 2y ago
I'm impressed by the latency using a request response. It looks this uses speech detection locally using Silero voice activity detector model using the ONNX web runtime, collects audio, then performs a POST. It doesn't look like t
44.
by AaronFriel 2y ago
A good rule of thumb is that models can be quantized to 6 to 8 bits per weight without significantly degrading quality. This is convenient for the math: 70GB plus some overhead for the attention matrices (ongoing requests). This overhead de
45.
by AaronFriel 2y ago
Oil prices went negative due to an exogenous, rare event shocking demand. No one expected that would recur. (And indeed, many people did try to figure out how to store oil in one-off vessels, short term leases for storage, etc., though I do
46.
by AaronFriel 2y ago
These costs don't line up with my own experiments using vLLM on EKS for hosting small to medium sized models. For small (under 10B parameters) models on g5 instances, with prefix caching and an agent style workload with only 1 or a sma
47.
by AaronFriel 2y ago
This is a vague recollection, but I seem to recall Meta/Facebook engineers on HN having said they have a tool that allows engineers to author SQL or ORM-like queries on the frontend and close to where the data is used, but a compiler o
48.
by AaronFriel 2y ago
The PagedAttention paper is a good starting point as it represents the first major open source inference engine that had "pretty good" batch performance for transformers. https://arxiv.org/pdf/2309.06180
49.
by AaronFriel 2y ago
The attention mechanism is vastly more efficient to train when it can attend to larger, more meaningful tokens. For inference servers, a significant amount of memory goes into the KV cache, and as you note, to build up the embedding through
50.
by AaronFriel 2y ago
"Only" 10k machines producing a combined 100 billion transactions per second is pretty hard to imagine, least of all that would all be producing transactions that are part of the same namespace. Virtually all UUIDs are meaningless
51.
by AaronFriel 2y ago
True, as universally unique identifiers, 128 (less a few) bits is not enough. You're talking about humanity generating 505 exabytes per year of just UUIDs. That won't happen any time soon.
52.
by AaronFriel 2y ago
It has only been a little over one year since GPT-4 was announced, and it was at the time the largest and most expensive model ever trained. It might still be. Perhaps it's worth taking a beat and looking at the incredible progress in
53.
by AaronFriel 2y ago
Care to share your math? My understanding of the birthday paradox is that it is astoundingly unlikely.
54.
by AaronFriel 2y ago
Very cool project, breaking the assumption that the nozzle must lay down material while moving on the same plane as the bed is a great innovation. One nit though about the website: I found it very distracting that the images kept cycling be
55.
by AaronFriel 2y ago
Pulumi is open source, and committed to Apache 2.0! We liken our model to git and GitHub, building the best service for the Pulumi engine and related tools. https://www.pulumi.com/blog/pulumi-hearts-opensource/
56.
by AaronFriel 2y ago
For runtimes with tooling like .NET has, it's a niche use case. At $dayjob, we have to support non-current compilers/SDKs for five different languages and runtimes where our tool will invoke (shell out, often) the CLIs for those t
57.
by AaronFriel 3y ago
I've seen this story every few years, perhaps this is just how they Munich negotiates its contract with Microsoft.
58.
by AaronFriel 3y ago
I'd strongly recommend against this default - it's a major blocker for using the Haskell library with web APIs as it transforms JSON RPC into into readily available denial of service attacks. 8 billion digits (~100 bits?) is far m
59.
by AaronFriel 3y ago
30gb+, yeah. You can't get by streaming the model's parameters: NVMe isn't fast enough. Consumer GPUs and Apple Silicon processors boast memory bandwidths in the hundreds of gigabytes per second. To a first order approximatio
60.
by AaronFriel 3y ago
It indeed is. An attention mechanism's key and value matrices grow linearly with context length. With PagedAttention[1], we could imagine an external service providing context. The hard part is the how, of course. We can't load ou

More ›