mmoskal

7 ms·

31.
by mmoskal 2y ago
The vec pattern has some advantages though, in particular you can often get away with 32 bit indices (instead of 64 bit pointers), making it a little more cache-friendly. I did it for regex ASTs that are supposed to be hash-consed, so never
32.
by mmoskal 2y ago
I wonder how can it be Apache if it's based on Llama?
33.
TensorRT-LLM runtime now open-source(github.com)
4 points by mmoskal 2y ago|1 comments
34.
by mmoskal 2y ago
Previously, the "Executor" runtime was shipped as binary blobs. This is the bit that schedules requests and manages KV cache (similar to vLLM or SGLang server).
35.
by mmoskal 2y ago
This comes from ML (as in SML or OCaml), where 'a reads "alpha" and is a type parameter.
36.
by mmoskal 2y ago
I think universally light when using a projector! But maybe nowadays it's remote or big LCDs...
37.
by mmoskal 2y ago
My favorites are Nvidia Triton (server) and Openai Triton (compiler)
38.
by mmoskal 2y ago
Their tech report says one inference deployment is around 400 GPUs...
39.
by mmoskal 2y ago
> ACM, the Association for Computing Machinery, is the world's largest educational and scientific society, uniting computing educators, researchers and professionals to inspire dialogue, share resources and address the field's
40.
by mmoskal 2y ago
You're most likely running a destilled version. The full model is ~700GB.
41.
by mmoskal 2y ago
You essentially have to run a byte regular expression that enforces valid UTF8. When you take into account exclusion for surrogate pairs and overlongs, you end up with about 14 states in the corresponding automaton. This is one thing among
42.
by mmoskal 2y ago
llama2, llama3, gpt3, and gpt4 tokenizers are all trained on UTF8 bytes and all include invalid (partial) UTF8 tokens. For llama3 it's only 256 tokens, one for each byte, but for the others it's more interesting (eg., 1361 tokens
43.
by mmoskal 2y ago
Tokens are in fact sequences of bytes not characters. For example, llama3 tokenizer (128k tokens) includes 1361 tokens that are invalid UTF8 (or rather they are partial UTF8). Models will generally only produce valid UTF8 (that is when by
44.
by mmoskal 2y ago
Luckily in the US the govt can do no such things due to the 1st amendment, so it only takes a relevant billionaire to get a model with different political views.
45.
by mmoskal 2y ago
CCP requires models to follow "socialist values". https://www.cnbc.com/amp/2024/07/18/chinese-regulators-begin...
46.
by mmoskal 2y ago
737-800 burns about 3t of fuel per hour. NYC-SFO is about 6h, so 18t of fuel. Jet fuel energy density is 43MJ/kg, so 774000 MJ per flight, which is 215 MWh. Assuming the 60 GWh figure is true (seems widely cited on the internets), it c
47.
by mmoskal 2y ago
I assume this comes from the 60GWh figure, which does translate to about 200 flights (assuming energy density of gasoline; in actual CO2 emissions it was probably less since likely cleaner energy was used than for running planes).
48.
by mmoskal 2y ago
I saw somewhere it's 0.5T/s of memory bandwidth. So you can read the whole memory (128G) 4 times per second. So if you fill it up with model you get 4t/s give or take. For comparison H100 can read its memory 40 times per seco
49.
by mmoskal 2y ago
Models are trained on Markdown, JSON and various programming languages, so either one of these should work. However, in this case, you're best of giving the model sentences one by one to avoid it being confused. If you structure the pr
50.
by mmoskal 2y ago
It depends how fine-tuned the model is to JSON output. Also, you need to tell the model the schema. If you don't you will get more weird tokenization issues. For example, if the schema expects a JSON key "foobarbaz" and the c
51.
by mmoskal 2y ago
If you want avoid startup cost, llguidance [0] has no compilation phase and by far the fullest JSON support [1] of any library. I did a PoC llama.cpp integration [2] though our focus is mostly server-side [3]. [0] https://github.
52.
by mmoskal 2y ago
It seems ollama only supports JSON Schema. Interestingly, JSON Schema has much less of this problem than say CSV - when the model is forced to produce `{"first_key":` it will generally understand it's supposed to continue in
53.
by mmoskal 2y ago
> The academic literature has recently caught on to this approach as well and we want to highlight the excellent papers on Sarathi (link 1, link 2) and DeepSpeed-FastGen (link). The academic literature calls the technique “chunked prefil
54.
by mmoskal 2y ago
I imagine there is and order of magnitude of difference between how much you can translate in software, with large memory and significant time budget to work with, compared to microcode.
55.
by mmoskal 2y ago
Abstract WTF-8 (Wobbly Transformation Format − 8-bit) is a superset of UTF-8 that encodes surrogate code points if they are not in a pair.
56.
by mmoskal 2y ago
2Tflops was #500 in 2006. Still a supercomputer. Crazy. https://www.top500.org/statistics/perfdevel/
57.
by mmoskal 2y ago
Back in the day I did an almost-Typescript (though much closer than assembly script) to embedded ARM compiler. Some of the techniques may be useful. https://www.microsoft.com/en-us/research/uploads/prod/2
58.
by mmoskal 2y ago
B came from BCPL which came from CPL which came from ALGOL 60. The main thing about ALGOL was structured programming as opposed to goto statements. Compared to that begin/end vs braces is a very minor issue.
59.
by mmoskal 2y ago
The most typical reason argmax (temp 0) is non-deterministic is that your request is running batched with other people requests. The number and size of these affects the matrix sizes and thus tiling decisions. Then you get different floatin
60.
by mmoskal 2y ago
They actually run 2^30-way tournament (they derive an equivalent form that doesn't requires 2B operations). You do not need to run the LLM, it only depends on the tokenizer.

More ›