4 ms·

I'm very curious as to how this works in the backend. I realize it uses Bluesky's firehose to get the posts, but I'm more curious on how it's checking whether a

I'm very curious as to how this works in the backend. I realize it uses Bluesky's firehose to get the posts, but I'm more curious on how it's checking whether a post contains any of the available words. Any guesses?

bangaladore 1y ago
Maybe I'm being naive, but with only ~275k words to check against, this doesn't seem like a particularly hard problem. Ingest post, split by words, check each word via some db, hashmap, etc... and update metadata.
- somebehemoth 1y ago
  I think the cool part is watching words go brrr.
gpm 1y ago
Probably just a big hashtable mapping word -> the number of times it's been seen, and another hashset of all the words it hasn't seen. When a post comes in you hash all the words in it and look them up in the hashtable, increment it, and if the old value was 0 remove it from the hash set. 250k words at a generous 100 bytes per word is only 25MB of memory...
f311a 1y ago
You can probably fit all words under 10-15MB of memory, but memory optimisations are not even needed for 250k words... Trie data structures are memory-efficient for storing such dictionaries (2-4x better than hashmaps). Although not as fast as hashmaps for retrieving items. You can hash the top 1k of the most common words and check the rest using a trie. The most CPU-intensive task here is text tokenizing, but there are a ton of optimized options developed by orgs that work on LLMs.
stwrzn 1y ago
I very much hope that the backend uses one of the bluesky jetstream endpoints. When you only subscribe to new posts, it provides a stream of around 20mbit/s last time I checked, while the firehose was ~200mbit/s.
- avibagla1 1y ago
  yes it does!
avibagla1 1y ago
Hey! this is my site - it's not all that complex, i'm just using a sqlite db with two tables - one for stats, the other for all the words that's just word | count | first use | last use | post. I... did not expect this to be so popular
- gumboshoes 1y ago
  What is your source dictionary to compare to? Seems kind of small. Also, how are you handling inflected forms?
  - avibagla1 1y ago
    https://github.com/words/an-array-of-english-words https://github.com/words/an-array-of-english-words using this, a combo of "covered enough" for the bit and easy to use also, since i'm tracking every word (technically a better name for this project would be The Bluesky Corpus) all inflected forms are different words, which aligns with my thinking
- blendo 1y ago
  What are the table sizes? And what ingress bandwidth do you have?
  - avibagla1 1y ago
    DB is currently 58mb (damn lol) Ingress is actually pretty manageable, ~900kbps

bangaladore 1y ago

Maybe I'm being naive, but with only ~275k words to check against, this doesn't seem like a particularly hard problem. Ingest post, split by words, check each word via some db, hashmap, etc... and update metadata.

somebehemoth 1y ago
I think the cool part is watching words go brrr.

gpm 1y ago

Probably just a big hashtable mapping word -> the number of times it's been seen, and another hashset of all the words it hasn't seen. When a post comes in you hash all the words in it and look them up in the hashtable, increment it, and if the old value was 0 remove it from the hash set. 250k words at a generous 100 bytes per word is only 25MB of memory...

f311a 1y ago

You can probably fit all words under 10-15MB of memory, but memory optimisations are not even needed for 250k words... Trie data structures are memory-efficient for storing such dictionaries (2-4x better than hashmaps). Although not as fast as hashmaps for retrieving items. You can hash the top 1k of the most common words and check the rest using a trie. The most CPU-intensive task here is text tokenizing, but there are a ton of optimized options developed by orgs that work on LLMs.

stwrzn 1y ago

I very much hope that the backend uses one of the bluesky jetstream endpoints. When you only subscribe to new posts, it provides a stream of around 20mbit/s last time I checked, while the firehose was ~200mbit/s.

avibagla1 1y ago
yes it does!

avibagla1 1y ago

Hey! this is my site - it's not all that complex, i'm just using a sqlite db with two tables - one for stats, the other for all the words that's just word | count | first use | last use | post. I... did not expect this to be so popular

gumboshoes 1y ago
What is your source dictionary to compare to? Seems kind of small. Also, how are you handling inflected forms?
- avibagla1 1y ago
  https://github.com/words/an-array-of-english-words https://github.com/words/an-array-of-english-words using this, a combo of "covered enough" for the bit and easy to use also, since i'm tracking every word (technically a better name for this project would be The Bluesky Corpus) all inflected forms are different words, which aligns with my thinking
blendo 1y ago
What are the table sizes? And what ingress bandwidth do you have?
- avibagla1 1y ago
  DB is currently 58mb (damn lol) Ingress is actually pretty manageable, ~900kbps