cztomsik

7 ms·

31.
by cztomsik 1y ago
I have no idea how it works actually (in google) but I wouldn't be surprised if it was just post-training because recently RWKV people did something similar: They replaced the whole attention mechanism with WKV (forward-only linear att
32.
by cztomsik 2y ago
More tokens -> more compute involved. Attention-based models work by attending every token with each other, so more tokens means not only having more time to "think" but also being able to think "better". That is also
33.
by cztomsik 2y ago
Thank you, so many great features. My personal favorite is .declLiteral() syntax, which makes webserver route-configuration very easy and readable! If you are ever in temptation to do a builder for internal DSL, .declLiteral() might be a be
34.
by cztomsik 2y ago
I suspect those so-called, "zero-cost abstractions".
35.
by cztomsik 2y ago
Rust is generally safer than Zig, but unsafe Rust is/was (mid)designed in such way that it's almost guaranteed to blow in your face eventually. I don't know if mutable noalias has been already enabled for good, but even if
36.
by cztomsik 2y ago
Nope, it's quite obvious why distillation works. If you just predict next token, then the only information you can use to compute the loss is THE expected token. Whereas if you distill, you can also use (typically few) logits from the
37.
by cztomsik 2y ago
That's what I meant to say. There are 2 parts, and both need to work correctly, otherwise you're wasting power.
38.
by cztomsik 2y ago
+1, no idea. But maybe to add a little bit of context, "damage tracking" means for example, that if there is any ongoing animation (like a spinner), then only a small part of the screen will be re-rendered (with proper vertex-time
39.
by cztomsik 2y ago
esbuild is a bit unfair example because Go is really great fit for this kind of job, it does a lot of I/O, it allocates a lot, and then it just dies. evanw even tried to use Rust at first but it was slower than Go version. IMHO, if JS
40.
by cztomsik 2y ago
IIRC there's `zig reduce` builtin in Zig :) https://github.com/ziglang/zig/blob/master/lib/compiler/redu...
41.
by cztomsik 2y ago
You pay for a car. This is free and OSS.
42.
by cztomsik 2y ago
explicit about branching and allocations, not so for types. we've recently got .decl() syntax, which is even more implicit than .{}
43.
by cztomsik 2y ago
See https://news.ycombinator.com/item?id=41712239
44.
by cztomsik 2y ago
I think around the BLOOM models (2022) it was found out that if you train english-only, the model performs worse than if you have even little mixture of other languages. Also, there were other papers (one epoch is all you need) where it was
45.
by cztomsik 2y ago
Exactly, they should be sued for this. I think the only people who are defending Apple are the ones who did not read the ToS carefully.
46.
Llama as a System(github.com)
7 points by cztomsik 2y ago|0 comments
47.
by cztomsik 2y ago
If you mean that LM-head is just inverted embedding matrix then this was already done in GPT-2. Unfortunately, the only thing I found out about this is that bigger models benefit from separate layer. But this was only mentioned somewhere in
48.
by cztomsik 2y ago
can you comment on what those 500 lines actually were? react+redux?
49.
by cztomsik 2y ago
How is that specific to React? And who would use webapp technology for a website?
50.
by cztomsik 2y ago
FYI there's also std.mem.zeroInit(T), or std.mem.zeroes(T) for C-style zero-init
51.
by cztomsik 2y ago
I don't like where this is going. There's nothing wrong with htmx except that aggressive marketing. SPA frameworks are overused but we got there because React was pushed everywhere exactly like htmx is being pushed right now. SPAs
52.
by cztomsik 3y ago
the cool thing with sqlite is that you can compile your whole app into a single binary, so you don't need docker for example (or it's trivial to dockerize it afterwards if you really insist)
53.
by cztomsik 3y ago
can you elaborate? I was living under impression that what sqlite has, is exactly what covering index is...
54.
by cztomsik 3y ago
column reordering is simple to fix with this migration script https://david.rothlis.net/declarative-schema-migration-for-s... if you are using Zig (and like to live on the bleeding edge), you can also just use my library wh
55.
Tokamak: A web framework for Zig (motivation, reasoning)(tomsik.cz)
3 points by cztomsik 3y ago|0 comments
56.
by cztomsik 3y ago
Why? From what I read those architectures have many similarities (and same weaknesses)
57.
by cztomsik 3y ago
> Importantly, these recurrent and convolutional forms, which I like to call “RNN mode” and “CNN mode,” are mathematically equivalent. This allows S4 to shape-shift depending on what you need it to do, with no difference in its outputs.
58.
by cztomsik 3y ago
There is also thorvg by Samsung (or the authors work in there and they are also using it, not 100% sure but it's production ready) https://github.com/thorvg/thorvg#dependencies AFAIK they have experimental GPU bac
59.
by cztomsik 3y ago
DPO is not RLHF.
60.
by cztomsik 3y ago
Thank you for your kind words :) I have so many ideas how to make it even better but the time has not been my friend lately :-D

More ›