anonymoushn

9 ms·

61.
by anonymoushn 1y ago
please post about my pet issue instead of whatever the comment section is nominally about.
62.
by anonymoushn 1y ago
Well, when it's this fast already, there may not be much point in vectorizing it. For integer parsing, this is a pretty cheap operation, so I wonder if it is worthwhile to try to avoid doing it many times. For double parsing, it is exp
63.
by anonymoushn 1y ago
Isn't this incorrect? If the user doesn't specify what to do with almost all of the special tokens, you still must detect them so you can raise an error.
64.
by anonymoushn 1y ago
Accounts with high karma have a "flag" link in the header of comments and submissions for indicating that it may not be the right kind of content for the site. Human moderators actively monitor the site, so your comment about this
65.
by anonymoushn 1y ago
Yes, I understand that the designers prefer the Allocator situation and that Reader and Writer being anytype was downstream of the difficulty of using async readers and writers otherwise. So the intention was always to go with the design th
66.
by anonymoushn 1y ago
I think this design is a regression from the previous design, in which you could use compile time introspection to check whether things are actually async (calling convention) or not. Additionally, I don't necessarily want to delegate
67.
by anonymoushn 1y ago
The closed weights models from OpenAI already do these things though
68.
by anonymoushn 1y ago
For now, at work, it's just some parts with AVX-512, some parts with AVX-512 that we can't really use, so we should use AVX2, and some parts with NEON and SVE. So the implementations for SSE basically are a courtesy to outside use
69.
by anonymoushn 1y ago
It's really not. As an example, for string processing tasks (including codecs which various server software spends a significant percentage of its runtime on), NEON includes a deinterleaving load into 4 registers and byte-wise shuffles
70.
by anonymoushn 1y ago
Sure. Another view is that an LLM is an immutable function from document-prefixes to next-token distributions.
71.
by anonymoushn 1y ago
I see. You're right. I was either badly mistaken or only thinking about small k.
72.
by anonymoushn 1y ago
It's about the immutability of the network at runtime. But I really don't think this is a big deal. General-purpose computers are immutable after they are manufactured, but can exhibit a variety of useful behaviors when supplied w
73.
by anonymoushn 1y ago
I think within the framework of "almost-orthogonal axes" you can still create a vector that has the desired mix of projections onto any combination of these axes?
74.
by anonymoushn 1y ago
There are various papers about this, maybe most prominently Byte-Latent Transformer.
75.
by anonymoushn 1y ago
I mean, this depends on your sampler. With temp=1 and sampling from the raw output distribution, setting aside numerics issues, these models output nonzero probability of every token at each position
76.
by anonymoushn 1y ago
yes, in typical architectures for models dealing with text, the output is a token from the same vocabulary as the input.
77.
by anonymoushn 1y ago
Well, to be blunt, the company does not care about this, so it does not get done.
78.
by anonymoushn 1y ago
The big recent change is that jemalloc no longer has any of its previous long-term maintainers. But it is receiving more attention from Facebook than it has in a long time, and I am somewhat optimistic that after some recent drama where som
79.
by anonymoushn 1y ago
I think that like actually using Rust and encountering situations where one must define a new error enum that exists only to wrap the values returned by two different fallible functions belonging to different libraries all the time was suff
80.
by anonymoushn 1y ago
In SuperBPE, a fixed number of tokens are learned, and then the constraints of pretokenization are removed entirely, and then the remainder of the target vocab size is learned. In Boundless BPE, no schedule must be chosen, because there is
81.
by anonymoushn 1y ago
I think that the ideal number of initial n-grams would be large enough that adding additional initial n-grams has no effect on the output, because I expect to not be very good at tuning two different knobs.
82.
by anonymoushn 1y ago
No, the segmentation algorithm you have implemented has runtime O(N*L^2). In particular, if you want to do a hash table lookup using a string key, that takes time proportional to the length of the string, not constant time.
83.
by anonymoushn 1y ago
I actually wonder if he could just claim a win by calculating validation set BPB for both equally-sized vocabs instead of targeting the same level of perplexity as in the speedrun finish line lol
84.
by anonymoushn 1y ago
Cool, I can have a look. By the way, I've filed a PR against the pathpiece repository correcting an error in the docs. I have read this paper before, and now that I have looked more closely I am a bit disappointed that the precise meth
85.
by anonymoushn 1y ago
If you try to reproduce various open datasets like fineweb by scraping the pages again, you can't, because a lot of the pages no longer exist. That's why you would prefer to store them instead of losing the content forever. It&#x2
86.
by anonymoushn 1y ago
Well, he says he achieved some downstream win on the same size, but it didn't translate into a win in perplexity, so he tried to do something else. Like I said, it's unfortunate.
87.
by anonymoushn 1y ago
https://x.com/alexjc/status/1881410039639863622 I agree that it would be more useful to compare vocabs of identical size.
88.
by anonymoushn 1y ago
You can use modded-nanogpt for testing this sort of change. If you don't want to do anything really weird, you can just train whatever tokenizer you like on the training set, then retokenize the input, then run the existing training co
89.
by anonymoushn 1y ago
They have not.
90.
by anonymoushn 1y ago
If the library has two vocabulary learners, only one of which does the described thing, then isn't it unambiguous which implementation within the library the question refers to? And wouldn't it be ambiguous to instead say "ho

More ›