mmoskal

11 ms·

121.
by mmoskal 3y ago
There are two parts to AICI - one is that it uses Wasm and the other is the specific set of operations on the LLM that it exposes (which is why we call it "interface"). Having a common interface (provided it is adopted) lets you r
122.
by mmoskal 3y ago
It's definitely very exciting direction, which we have not explored at all!
123.
by mmoskal 3y ago
Each of these can be implemented on top of AICI. AICI lets you run an arbitrary (Wasm) program to determine which tokens are allowed next - this is strictly more expressible than CFGs. In addition to constraining output, AICI also lets you
124.
by mmoskal 3y ago
We believe Guidance can run on top of AICI (we're working on efficient Earley parser for that [0], together with local Guidance folks). AICI is generally lower level (though our sample controllers are at similar level to Guidance). [0]
125.
by mmoskal 3y ago
What you describe there is a great example of a custom controller - could be implemented on top of pyctrl, jsctrl, or just natively in Rust.
126.
by mmoskal 3y ago
You generally have to tell the LLM what you want it to generate and then enforce it, i.e, the LLM has to be somewhat aligned with the constraints. Otherwise, for example if you ask for JSON it will keep generating (legal) white-space or whe
127.
by mmoskal 3y ago
It's only the controller that runs in Wasm, not the inference. The pyctrl is a just a sample controller, you can write a controller that takes any kind of grammar (eg., a yacc grammar [0] - the python code in that example is only used
128.
Show HN: Prompts as WASM Programs(github.com)
209 points by mmoskal 3y ago|35 comments
129.
by mmoskal 3y ago
First hit on google [0] - no idea if it's true but seems plausible... (half a year ago sama said 100m) [0] https://explodingtopics.com/blog/chatgpt-users
130.
by mmoskal 3y ago
The bitnet paper was showing worse results than fp16 transformer with the same parameter count. The shocking result in the 1.58b paper (same group) is no quality loss compared to fp16.
131.
by mmoskal 3y ago
This kind of low level strategies can be tried on many models using AICI, including the substring one https://github.com/microsoft/aici/blob/main/controllers/aici...
132.
by mmoskal 3y ago
Training GPT-4 is estimated at 50GWh [0]. Driving a pickup truck consumes about 30MWh per year in gasoline (13500 miles/year, 15 mpg, 33kWh/gallon). So, training GPT-4 is equivalent to driving 1000-2000 trucks for a year, or in ot
133.
by mmoskal 3y ago
It's 23 rupees to a dollar at PPP. The exchange rate is 82, so a dollar goes 3.5 times further in India not 23 :)
134.
by mmoskal 3y ago
FWIW a recent The Economist report [0] lists data center usage at 1-2% of global electricity consumption (which is 1/3 of energy IIRC) and AI at 8% of that in 2023 projected to grow to 15-20% by 2028. Given how careful the hyper scaler
135.
by mmoskal 3y ago
I'm a lead dev on this. Welcome comments/questions!
136.
by mmoskal 3y ago
I learned something today :) Gravity on surface of this planet will be 3(times heavier than earth)/(1.55(radius)^2), so 1.25g, however escape velocity also depends on planet radius so we get sqrt(1.55 * 1.25) = 1.4 earths escape veloci
137.
by mmoskal 3y ago
Hmm... about 15 seconds per floor (judging from Wikipedia video), times 118 floors = half hour to get to the top! I guess still faster than taking the stairs...
138.
by mmoskal 3y ago
All the nvidia/torch stuff is "only" to make it run two orders of magnitude faster or so. For the actual principles behind LLMS, 500 lines seems quite accurate. How long would it take to explain what an LLMs is doing to someo
139.
by mmoskal 3y ago
There is also anyhow::ensure!() (which is more of a requires() but anyhow...)
140.
by mmoskal 3y ago
GPU has 10x the memory bandwidth of the CPU though, which becomes relevant for the LLMs where you essentially have to read the whole memory (if you're batching optimally, that is using all the memory either for weights or for KV cache)
141.
by mmoskal 3y ago
For the game angle https://arcade.makecode.com may be more of a fit. You can even build a cabinet. Disclaimer: worked on both.
142.
by mmoskal 3y ago
If the water is moving at 10m/s and you can spot 0.1m difference in the photo then it would need to be within 10ms of each other.
143.
by mmoskal 3y ago
It updates token logits (probabilities) after every token before sampling. I don't think this is very common yet.
144.
by mmoskal 3y ago
I agree (and would add explicit self argument, the fact that all your imports are exported and more) I think the point here is to reuse the Python ecosystem especially in ML.
145.
by mmoskal 3y ago
I've been using it for a while and love it but intend to give the open source Organic maps a try. In particular sharing a way point in Mapy.cz seem to require an account with a Czech web portal. The reason Mapy.cz has all the trails is
146.
by mmoskal 3y ago
OTOH, AFAIK when running a model locally Guidance does something really similar to what OP is doing.
147.
by mmoskal 3y ago
There is at least 6 orders of magnitude difference in computation cost of computing a token (pass through a multi-B model) and doing a single step of a program. Even if your validation is really naive, it's hard to beat 6 orders of mag
148.
by mmoskal 3y ago
I bet they have a deal with uBlox so that people prototyping with this board will use uBlox for final mass produced design. In general you use module instead of chip to save on certification. However for ESP32 the logical choice would be i
149.
by mmoskal 3y ago
Well, this M4 is most likely much slower than Rp2040 (2.5x difference in clock speed plus dual core). At the same clock the difference between M0+ and M4 isn't too dramatic (unless you use FPU).
150.
by mmoskal 3y ago
NRF52 line is Bluetooth LE not WiFi.

More ›