5 ms·
The Bluesky Dictionary
- 75345d4c 1y agoI just saw it indexed "eluvium," but the post was referring to a band with that same name
- Kye 1y agoGeologySky will get to it soon enough.
- k7sune 1y agoThanks to this I just learned about alluvium, eluvium, illuvium, and colluvium.
- atlgator 1y agoI checked out the author's other projects and this is common issue. For example, he has a "lean checker" for bluesky that claims it is right-leaning simply because of all the people saying "That's right," "He was right," etc. None of the supposed right-leaning posts were actually conservative in nature. They just used to word right to mean correct.
- avibagla1 1y agoone, thank you for checking my website. two, that is the joke, 100% - at the time people kept talking about how "left leaning" bsky was and that idea came to mind
- OneDeuxTriSeiGo 1y agolmao that's fantastic
- neaden 1y agoIs this not working or am I missing something, it just shows as seeing 0 words for me. Firefox on a PC.
- accrual 1y agoYou may need to allow scripts from the domain avibagla.com, it shows 0 when the scripts are blocked.
- zem 1y agough, it ought to be building the results on the server and serving up static pages.
- rafram 1y agoBut it updates live...
- forgotmypw17 1y agoIt could do both...
- dymk 1y agothen go build it…
- Noumenon72 1y agoAll the scripts are ERR_SSL_PROTOCOL_ERROR for me in Chrome, I'm assuming because of a corporate firewall.
- 71bw 1y agoIndeed, all I get in Firefox are CORS issues
- SirFatty 1y ago
- GalaxyNova 1y agofascinating! I think it's really cool that this is possible, and at the same time kine of sad that the norm is slowly moving towards more locked-down APIs.
- timeon 1y ago> slowly moving towards Depends what we accept as norm.
- wantlotsofcurry 1y agoI'm very curious as to how this works in the backend. I realize it uses Bluesky's firehose to get the posts, but I'm more curious on how it's checking whether a post contains any of the available words. Any guesses?
- bangaladore 1y agoMaybe I'm being naive, but with only ~275k words to check against, this doesn't seem like a particularly hard problem. Ingest post, split by words, check each word via some db, hashmap, etc... and update metadata.
- somebehemoth 1y agoI think the cool part is watching words go brrr.
- gpm 1y agoProbably just a big hashtable mapping word -> the number of times it's been seen, and another hashset of all the words it hasn't seen. When a post comes in you hash all the words in it and look them up in the hashtable, increment it, and if the old value was 0 remove it from the hash set. 250k words at a generous 100 bytes per word is only 25MB of memory...
- f311a 1y agoYou can probably fit all words under 10-15MB of memory, but memory optimisations are not even needed for 250k words... Trie data structures are memory-efficient for storing such dictionaries (2-4x better than hashmaps). Although not as fast as hashmaps for retrieving items. You can hash the top 1k of the most common words and check the rest using a trie. The most CPU-intensive task here is text tokenizing, but there are a ton of optimized options developed by orgs that work on LLMs.
- stwrzn 1y agoI very much hope that the backend uses one of the bluesky jetstream endpoints. When you only subscribe to new posts, it provides a stream of around 20mbit/s last time I checked, while the firehose was ~200mbit/s.
- spullara 1y agoI did this against a pretty large tweet archive and got hits on about 125k of the words in the unix dictionary.
- pona-a 1y agoFor a moment I thought it would be an AT-Proto based Urban Dictionary clone.
- evbogue 1y agoThis
- tough 1y agoWords We Haven't Seen - Search unseen words made me chuckle
- crm9125 1y agoI've found content for all of my future skeets.
- refreeze654 1y agoI've wondered how blueksy affords the bandwidth to let anyone stream the full firehose.
- deleted 1y ago[deleted]
- dgacmu 1y agoNot an answer to your question, but I suspect most people don't -- my bot (a pi searcher bot, of course) just runs on Jetstream, which is pretty lightweight and heavily compressed. (The website in question uses jetstream also.)
- psionides 1y agoFrom what they say, it is a lot, but it's generally on the order of a few hundreds of connections total at the moment
- deleted 1y ago[deleted]
- blintz 1y agoI'm surprised at how normal some of the unseen words are. I expected them to all be archaic or niche, but many are pretty reasonable: 'congregant', 'definer', 'stereoscope'.
- gkoberger 1y agoFor what it's worth, there's 1.7bn posts on Bluesky according to this: https://bsky.jazco.dev/stats https://bsky.jazco.dev/stats The dictionary site has only checked 4,920,000 posts, which is 0.28% of all messages.
- SomewhatLikely 1y agoIt's likely that the commenter has read less than 5 million posts worth of text though. So perhaps this still points to a lack of diversity in content.
- wolfram74 1y agoYou got me wondering. Supposing the average post is 10 words, and a typical page of text is 250 words, that would only be ~50 pages of text a day over the last 10 years. Which I don't think I manage, but over 20 years I am probably in that window.
- IshKebab 1y agoIt now claims to have checked 11 million posts but only seen "the" 16 thousand times. I'm not sure its numbers are entirely reliable.
- awinter-py 1y agodentel, exclaustrations, gryding, datolite, frabbing?
- Kye 1y agoI can't keep up with all these new Pokemon.
- fudged71 1y agoI'm just surprised that there's revolt when Bluesky posts are used for LLMs, but regular NLP is fine for some reason.
- max_ 1y agoThis website is so pretty!
- runflyswim 1y agothank you!! design support and advice from my good friend vedantswarup.com
- anywhichway 1y agoI noticed one of the cited bluesky posts was all in French, so one might argue that technically it didn't find the English word "mouch", but rather a different French word that happens to be spelled the same. But trying to sort that out seems unrealistically challenging. "Mouch" is only in the dictionary as an alternative spelling to mooch, so probably a pretty rare word to see in English.
- steinuil 1y agoBluesky lets you select the language your post is written in before posting it and it is attached as metadata to the skeet. I guess the backend for this only searches posts in English, but it's possible the dataset is not 100% accurate due to some users forgetting to switch language before posting.
- croes 1y agoSo now someone is simply posting a dictionary
- xnorswap 1y agoSomeone just got a double-combo: > We just visited wheal Martyn museum in Cornwall, nice scones and a waterwheel, they also have a lot of gutters, sluices and pipes and a bit of a fixation about China Clay. More importantly they appear to be unattached at the moment Both "wheal" (kind of cheating, that should be Wheal and is a place name) and "sluices" were new to the dictionary.