belladoreai

6 ms·

31.
by belladoreai 3y ago
> Honestly it's pretty trivial. Literally no artist anywhere has ever received an actual payment from an actual customer with this method (I'm not counting the artist themselves making a test payment). No, if Gumroad bans you,
32.
The pranksters behind Goody-2(wired.com)
20 points by belladoreai 3y ago|4 comments
33.
by belladoreai 3y ago
It's a joke website. Sure it's a fully closed and proprietary joke website, but it's still a joke website.
34.
Goody-2, the world's most responsible AI model(goody2.ai)
435 points by belladoreai 3y ago|180 comments
35.
by belladoreai 3y ago
I created Allie. I don't want to link anything here that might violate HN rules, but you can find me in Discord where my username is belladore.
36.
by belladoreai 3y ago
That's quite a question coming from a self-proclaimed non-programmer (are you sure you're not one?) I didn't use a Trie. The main data structures used are Linked List and Priority Queue. The tokenization begins by transformin
37.
by belladoreai 3y ago
These are both great questions. > I take it if you do greedily go from left to right, you end up with different tokenization and the model doesn't work properly because that isn't what it was trained with? The way that the LLaM
38.
by belladoreai 3y ago
I think it would take quite a bit of work for someone to grab that BPE implementation and make it work for LLaMA. Less work than rewriting the whole tokenizer from scratch, for sure, but a non trivial amount of work anyway.
39.
by belladoreai 3y ago
> if I am ok with slightly suboptimal efficiency (keeping slightly less tokens than the model supports) ... multiply the length of the result of splitting on spaces by 1.55 for OpenAI and 1.7 for Claude This sounds reasonable to me. You
40.
by belladoreai 3y ago
I'll answer with an example. Consider the input string " grabbed". If we wanted to map this string to tokens by greedily going from left to right and choosing tokens from the vocabulary with the strategy of minimizing the num
41.
by belladoreai 3y ago
Hi HN! I was looking for a tokenizer that would accurately(!) count tokens in browser, and I couldn't find one. So I thought "how hard can it be", and here we are 2 weeks later...
42.
Show HN: LLaMA tokenizer that runs in browser(github.com)
115 points by belladoreai 3y ago|23 comments
43.
by belladoreai 3y ago
Published now: https://github.com/belladoreai/llama-tokenizer-js
44.
by belladoreai 3y ago
That's what I thought when I started working on this, but it turns out the answer is no! This approach - "minimal mapping of character sequences to numbers" - can be described as a greedy algorithm. Using this approach will o
45.
by belladoreai 3y ago
Sure. Check back to this comment after the weekend. I will post it here.
46.
by belladoreai 3y ago
TikToken doesn't provide a tokenizer that's compatible with LLaMA.
47.
by belladoreai 3y ago
Tangentially related to what you ask: LLaMA tokenizer has fallback to byte-level tokens.
48.
by belladoreai 3y ago
You are almost correct, though it doesn't happen at character level, it happens at byte level. Most characters are in LLaMA tokenizer's vocabulary, but all characters aren't. So if you use a character that was uncommon in the
49.
by belladoreai 3y ago
> The token lists are just text files that consist of the characters base64 encoded followed by the numeric ID. If you want to explore the list you can just download them and decode them yourself. If you want to look at mappings for indi