6 ms·
Consuming the Bluesky firehose for less than $2.50/mo
- brrrrrm 2y agoyour single word firehose demo is awesome. I can see a lot of creative people doing really cool things with this
- caesil 2y agoWow this is neat! Stark contrast to the stupid locked-down new API rules twitter rolled out post-Elon.
- ks2048 2y agoThis is cool and a big reason to root for BlueSky over X, all politics aside. I wonder if they (or some third party) offers zip files of dumps of various kinds? It seems that would go easier on thier servers.
- kylehotchkiss 2y agoIt was fun playing with Twitters firehose way back when that was accessible to the common developer. Excited to see the idea make a comeback!
- tonetegeatinst 2y agoI sense this becoming training data for some AI/LLM in the near future.
- BobbyJo 2y agoTBF, a walk down the street where cctv cameras are present is likely already training data for some AI.
- bogwog 2y agoPossibly even training the model that will go into the police automaton that will kill you some day.
- cyberax 2y agoThe future is already here! https://www.reuters.com/world/europe/moscow-uses-facial-recognition-cameras-track-down-draft-evaders-tass-2023-04-18/ https://www.reuters.com/world/europe/moscow-uses-facial-reco...
- hieu 2y agoOnly two outcomes: either everyone can access it or only few big companies can. While not ideal, the former is much better.
- llm_trw 2y agoIt really isn't when you realize everyone includes people who sell home burglary as a service as scale.
- mewse-hn 2y agoI think researchers want vintage data thats not contaminated with llm vomit
- NewJazz 2y agoYou mean uncurated 100 character blurbs of plausible sounding words isn't quick diddle making sense to the time of the hour?
- paxys 2y agoBetter that everyone have access to the data rather than just a couple large corporations.
- evenw 2y agoThe bsky firehose is really fun and feels like a misty-eyed throwback to 2005. I used it last weekend to throw together this very silly little thing: https://bigmood.blue/ https://bigmood.blue/
- lelandfe 2y agoI've got the perfect soundtrack for it (flashing lights warning) https://www.youtube.com/watch?v=wNtxgxYY7sI&t=66s https://www.youtube.com/watch?v=wNtxgxYY7sI&t=66s
- evenw 2y agoThank you! Perfect.
- esperent 2y agoIs there any way to make it flash less? I'm not epileptic, and I appreciate that you put a warning first, but it's still highly unpleasant after around 2 seconds. If I could look at it for a bit long it would be a cool experiment.
- h0l0cube 2y ago+1 Would be good if it just tiled the images, and then wrapped around after the screen is filled
- cyberlimerence 2y agoI like the concept, but it is flashing way too fast. Maybe do a random selection every 1 sec, rather than a straight fire hose ?
- zamadatix 2y agoVery nifty, I like it quite a bit. I wonder if instead of black+repeating images (assumption here is that's the display and not an artifact of the firehose, the same unique image often repeats a lot in the same short instance) it'd be cleaner to have some sort of exponential backoff queue (the closer it is to empty the slower it empties like 0.5s/numInQueue, last element in queue will wait no less than 0.5s but also wait for the next element to queue). This would make the flow a bit more consistent, never blank, and never repeating. Also if images kept their aspect ratio.
- monksy 2y agoOoo we're back in the old good days of twitter again! The time where you could build against it and do things. I wrote a contest winning bot back at one tie: https://theexceptioncatcher.com/blog/2015/11/how-i-got-my-bot-banned-by-twitter/ https://theexceptioncatcher.com/blog/2015/11/how-i-got-my-bo...
- hobs 2y agoI dont understand why I would build on bluesky when it seems like it would have the same medium term issues that the twitter api had anyway, but enjoy it while it lasts I guess.
- marviel 2y agosay more? I thought the protocol was designed to be able to handle decentralization
- koolala 2y agoSo a firehose isn't different than re-hosting Bluesky and it encourages re-hosting?
- Zamiel_Snawley 2y agoThere is still major centralization. Essentially, the relays crawl the network and provide an aggregated data stream, their firehose. Independent relays can exist, which is better than the fully centralized options. There is no financial incentive to bear the cost of running a public relay.
- marviel 2y agoThank you for the explanation! Are there efforts to build a similar type of network which rewards running a public relay? I'm not sure how, for instance, Mastodon solves this problem
- donatj 2y agoI'm still irate Twitter broke all my automaton and then had the gal to try to charge me $100 a month!? Looks like it's up to $200 now. That's as much as my car loan... No normal individual can justify that.
- bradgessler 2y agoTwo reasons to like BlueSky: 1. Your username can be your website. I'm @bradgessler.com. 2. Copy and pasting images into a message works on iOS (this has been broken on X/Twitter for a long time). There's more reasons to like it, but these two feel pretty great. I put a Ruby hacker starter pack together if you're looking for a way in: https://go.bsky.app/HXB2cPh https://go.bsky.app/HXB2cPh There's tons of other ones and communities as well.
- LeoPanthera 2y ago> Your username can be your website. I'm @bradgessler.com. But more importantly, you have to verify that you own the domain. I can't sign up as google.com
- numpad0 2y agoWas it on Nostr that someone realized "s3.amazonaws.com/.well-known" was a valid unused bucket name and did that?
- sathyabhat 2y agoNope, bluesky itself https://chaos.social/@jonty/110307532009155432 https://chaos.social/@jonty/110307532009155432 https://news.ycombinator.com/item?id=35820815 https://news.ycombinator.com/item?id=35820815
- Vaslo 2y agoIt’s going to go the way of Threads and Mastadon. Lukewarm results at best, but glad you are enjoying it.
- bradgessler 2y agoPerhaps, but I haven’t seen this level of momentum and consensus behind a text-based social network since early Twitter. Time will tell—ultimately it’s a self-fulfilling prophecy. If people “believe” and sign up, it will happen.
- Onavo 2y agoWhat language are you using for the backend server? 256MB is pretty tight unless you are using something like Go.
- Retr0id 2y agoI was consuming the full authenticated firehose (not jetstream) with tens of megabytes of RAM with python+sqlite (unfortunately I ran out of bandwidth on my cheapo VPS, I've yet to refactor to use jetstream instead)
- bananadreep 2y ago[flagged]
- foundart 2y agoGreat opening line! > It's fun to play with data[citation needed]
- jeffbee 2y agoI mean, the punch line of this post is that the "firehose" of even a successful social network is trivial to handle. Even Twitter at its pinnacle of popularity barely breached ~10000 posts/second intraday peaks. If you only want to observe and transiently store such traffic you barely even need a computer.
- fcantournet 2y agoThe firehose of peak twitter was 80MB/s of compressed json IIRC. That more around x00k posts/s
- smeeger 2y agodecentralized social media is the achilles heel of democrats. they just keep throwing themselves at it and it never works and never will work. while democrats work feverishly on this concept, X continues to grow daily. i would put money on this if polymarket hadnt just been the victim of a politically motivated FBI raid…
- 101008 2y agoI have an important account on Twitter where I publish manually posts, thoughts, links, announcement, etc. I tried to look for a way to crosspost to Threads and Bluesky automatically, but I couldn't find anything. Any suggestions?
- anonzzzies 2y agoI think that's another issue; there is no viable way to migrate social media data as if it were possible it would be abused mostly.
- morjom 2y agoThere are some apps (OpenVibe and Yup) that let you post to multiple places.
- 101008 2y agoI found Yup but I don't want to change the place where I originally post. I want to keep posting on Twitter using the web interface, and then automatically replicate that in Bluesky and Threads. I think it may not be possible due to the lack of Threads API.
- rc_kas 2y agoSheez man you are a wizard. This is highly skilled craftsmanship right here. Amazing skill.