mmoskal

8 ms·

61.
by mmoskal 2y ago
My "favorite" are these pepper grinders, milk frothers etc that have a usb c plug but only work with usb a to c cables since they do not include the two resistors that are needed to enable 900mA on usb c to c cable. And if anyone
62.
by mmoskal 2y ago
In other words the only way we know of how significant complexity arises in the universe is (ultimately) through evolution.
63.
Show HN: LLGTRT: TensorRT-LLM+Rust server w/ OpenAI-compat and Structured Output(github.com)
6 points by mmoskal 2y ago|0 comments
64.
by mmoskal 2y ago
With a typical transformer and a GPU the batch size that saturates the compute is at least hundreds. Otherwise (including typical size of 1 for local inference) you're memory bound.
65.
by mmoskal 2y ago
Right some of them do. I just recall the cheapest STM32G0s not having them (though they did have OTP so you could burn it yourself).
66.
by mmoskal 2y ago
All except for the cheapest microcontrollers (everything above half dollar) have serial numbers (unique identifiers) burned into one time programmable memory.
67.
by mmoskal 2y ago
Tokens are often sub-word, all the way down to bytes (which are implicitly understood as UTF8 but models will sometimes generate invalid UTF8...).
68.
by mmoskal 2y ago
Even the Germans didn’t make it work…
69.
by mmoskal 2y ago
Don't get me started on Midwest in the US...
70.
by mmoskal 2y ago
I think the GP post meant that EU only wants to regulate tech if it's American.
71.
by mmoskal 2y ago
You can do tensor parallelism 8 ways (8 KV heads). You can also do pipeline parallelism (there is 126 layers). Either way would work. A million tokens is possible just very slow. Also, 405b has 8 KV heads of 128 size (hidden_size/num_a
72.
by mmoskal 2y ago
Decode speed is generally memory bandwidth bound. Prefill is typically arithmetic bound. This is the reason for mixed batches (both decode and prefill) - it let's you saturate both memory and arithmetic. Chunked prefill is for minimizi
73.
by mmoskal 2y ago
IIUC this the cpu in an iPad. The pro/max versions would be more appropriate to compare against the Zen when they are released.
74.
by mmoskal 2y ago
Also M33 vs M0 and higher clock - should be 2-3x faster. And maybe you can do something with the second core. Im super impressed you got this to work! The memories of the SAMD clocks... Shudders.
75.
by mmoskal 2y ago
It's software emulated though. The actual speed is equivalent to around 1MHz. Intrestingly the tiny Cortex M0+ is has about 4x the coremark per MHz of 386 (and twice 486). See https://www.eembc.org/coremark/scores.
76.
by mmoskal 2y ago
Literally The Diamond Age.
77.
by mmoskal 2y ago
Interestingly in a Javascript or similar runtime most of text that hits the caches where the size actually matters is still ASCII even in far east because of identifiers. Utf8 for the win!
78.
by mmoskal 2y ago
macOS has futexes it just doesn't talk about them... https://crates.io/crates/ulock-sys
79.
by mmoskal 2y ago
It's a different kind of memory chip that also does some computation. See https://en.m.wikipedia.org/wiki/In-memory_processing
80.
by mmoskal 2y ago
Typical speed difference between Python and native is around two orders of magnitude (of course for code that actually does computation in Python, not just call numpy etc).
81.
by mmoskal 2y ago
In Java you can't have a class implement both Iterator<Foo> and Iterator<Bar> (type erasure; or maybe my Java is out of date...). Anyway if you could, the for loop would not know which to use. In Rust you could have both It
82.
by mmoskal 2y ago
The way I heard it described is that you call the library while the framework calls you.
83.
by mmoskal 2y ago
I successfully used Beagle USB 12 and 480 https://www.totalphase.com/products/beagle-usb480/ . Saleae https://www.saleae.com also has some USB functions but not as extensive. It's much more versat
84.
by mmoskal 2y ago
Which alternatives exactly? Mainstream languages like Go and Swift use a GC and don't allow the same level of control over memory layout. There are probably some reasons Rust is only non-C language to get into Linux kernel.
85.
by mmoskal 2y ago
Right... and there are probably some communication overheads over NVLink that would not be present on single laptop. So a few months maybe :)
86.
by mmoskal 2y ago
H100 SXM is 2000 TFLOPS at FP16. Multiply by 8. M3 Max is 28 TFLOPS at FP16. Based on FLOPS alone, it would be more like a year or two.
87.
by mmoskal 2y ago
Paper link https://arxiv.org/abs/2308.04838
88.
by mmoskal 2y ago
You would be probably better off just running esbuild or similar - it will happily strip types from any version of TS.
89.
by mmoskal 2y ago
I think most bundlers can handle TS nowadays so you don't actually need tsc.
90.
by mmoskal 2y ago
Everything that is experienced except for awareness itself.

More ›