8 ms·
Source code for the X recommendation algorithm
- mxstbr 1y agohttps://x.com/XEng/status/1965226798460887127 https://x.com/XEng/status/1965226798460887127
- numpad0 1y agoInteresting. So the numbers/fractions of "for you feed isn't working" complaints, and specifically that complaint, is above some threshold?
- davidw 1y agoDoes it include the bit about white South Africans?
- kennyadam 1y agoThat was Grok, not the X algorithm.
- cubefox 1y agoAnd the Grok prompts are also open source: https://github.com/xai-org/grok-prompts https://github.com/xai-org/grok-prompts
- jsheard 1y agoWell, some version of them are open source, which may or may not be what is actually running in prod. AFAIK the patch which made it obsessively bring up white South Africans was never published, and this algorithm repo went over two years between updates so it obviously wasn't tracking prod.
- cubefox 1y ago> which may or may not be what is actually running in prod. This is the case for every open source software ever. > AFAIK the patch which made it obsessively bring up white South Africans was never published Or there never was a specific patch for that purpose, contrary to what you are assuming. > this algorithm repo went over two years between updates so it obviously wasn't tracking prod. You are mixing up something. The Grok prompt repo is a different repo from the recommendation algorithm, and the former has been updated regularly.
- cma 1y agoDid they ever open source in there the stuff that was making Grok search for Musk's opinion before giving an opinion on world news?
- luma 1y agoNope, nor did that repo have the system prompts that brought is MechaHitler, nor the time earlier this year where it started injecting Trump into every completion. The Grok repo is a smokescreen for deniability (just not particularly plausible).
- cubefox 1y ago> Nope, nor did that repo have the system prompts that brought is MechaHitler False, you made that up. https://decrypt.co/329365/bye-bye-mechahitler-elon-musk-xai-quietly-fixed-grok-deleting-line-code?amp=1 https://decrypt.co/329365/bye-bye-mechahitler-elon-musk-xai-... > The Grok repo is a smokescreen for deniability (just not particularly plausible). Completely unfounded conspiracy theory.
- cma 1y agoI would be surprised if that line in the prompt caused it without the other thing they did just before mecha Hitler: Elon created a Twitter thread asking users to submit divisive politically incorrect facts for grok training. It was full of Holocaust denial and white supremecy stuff. https://x.com/elonmusk/status/1936493967320953090 https://x.com/elonmusk/status/1936493967320953090 Most likely they rolled back finetuning on that thread at the same time they adjusted the prompt.
- openquery 1y agoI've always wondered - how can I as a non X engineer be sure that the code on GH is actually deployed on their servers?
- h1fra 1y agoyou can't, and it's 100% sure it's not this code running in prod
- jjordan 1y ago100% huh? That's a bold statement with no supporting evidence.
- Pxtl 1y agoSounds like the right tone when discussing a Musk project.
- viraptor 1y agoAlready posted above: https://github.com/twitter/the-algorithm/blob/c54bec0d4e029fe34926ef3258a86ccacc0d0182/trust_and_safety_models/abusive/abusive_model.py#L18 https://github.com/twitter/the-algorithm/blob/c54bec0d4e029f... It's redacted.
- jibal 1y agoClaiming that there's no supporting evidence is a bold (and obviously false) claim when the code is 2 years old and heavily redacted.
- random3 1y agoI don’t think that’s the point of open sourcing things, in general
- openquery 1y agoI agree in general it isn't. But in this case Musk claimed that was the point of open-sourcing the algorithm. Transparency on what they are or are not suppressing.
- jsheard 1y agoRIP author_is_elon, we hardly knew ye. https://github.com/twitter/the-algorithm/issues/236 https://github.com/twitter/the-algorithm/issues/236
- echelon 1y agoRepublican, Democrat, and Elon. Wow.
- deleted 1y ago[deleted]
- ivape 1y agoIs this real? We accept that the algorithm may link you abstractly with other people, but I didn’t think they were literally labeling on this level. If you just say “we look for what’s similar and leave it at that”, then there’s much less liability. This is political targeting. This guy was one of the biggest political donors, how can this fly?
- burnte 1y agoYes, he really had twitter change their code to push his tweets more.
- jsheard 1y agoThey seem to have dialed the overt Elon boosting down now but it's still conspicuously aligned with his priorities. I just made a fresh burner account to see what the algorithm is primed to push by default nowadays, and about 80% of the feed is anti-immigration ragebait.
- mrguyorama 1y agoI think that's just an accurate and mostly genuine indicator of who is left on twitter nowadays. The people left on twitter earnestly believe that it is better now that you can shout racial slurs at people, buy your way to the top of any chain, get literally paid for ragebait, and genuinely think this repo is meaningful. It's a massive self selection bias.
- Scene_Cast2 1y agoJust like with their last release, they only released the architecture and not the weights. It may be useful for analyzing the system if you're a competitor (but from my last dive into it, it seemed like a strict subset of fancier, industry-leading rec systems), or perhaps getting into rec / retrieval systems as a newcomer. However, this gives roughly zero insight into how Twitter's feed behaves.
- amelius 1y agoThere might be some value if someone can show that the feed mis-behaves for some selection of weights.
- Scene_Cast2 1y agoNope. Every single system like that will misbehave if given a bad set of weights, or even a random set of weights. I'd go as far as saying that even with "good" weights, it's likely to have some sigma of misbehavior.
- gyanchawdhary 1y agoFor all its flaws .. it’s still a step up from how Parag and co used to run twitter
- snapcaster 1y agoWhy? I've never been a twitter user
- gyanchawdhary 1y agoPost Musk Twitter is amazing. It lets you see how stories, opinions that you support or don’t are attacked from all sides and Community noted / @grok fact checked … a lot of UX changes too .. pre Musk, the moderation / banning was biased and arbitrary (who is watching the watchers?) .. my personal fav was to see the special tick removed from journalists ..
- cubefox 1y agoThis is laudable. But the great thing about Twitter is that you don't have to use the algorithmic "For You" feed at all. You can just use the "Following" feed, which is purely chronological, and doesn't contain any recommended content. This isn't possible on Facebook, which makes it unusable for me.
- Pxtl 1y agoThat ignores the way Twitter sorts replies, which always use an algorithmic weight-based listing.
- 833 1y agoNot if you sort by recency or likes.
- deleted 1y ago[deleted]
- thomasm6m6 1y agoOn iOS at least, there is a “Sort replies > latest” option which is strictly chronological
- bprew 1y agoFacebook has a "friends only" feed (Friends button, next to Home), but it's not chronological.
- SilverElfin 1y agoI don’t use it - does it remember the setting? My recollection is that Facebook would make you switch to that chronological feed manually each time you load the page.
- cubefox 1y agoAs I said, Facebook doesn't have a chronological feed. Twitter remembers the feed setting.
- barrenko 1y agoJust ping Nikita, he'll tell you the current algo.
- qiine 1y agohope it doesn't involve goat again
- barbazoo 1y ago[flagged]
- ivape 1y agoElon’s father was not keen on him leaving South Africa from what I recall. There was an element of daddy doesn’t believe in me. He left against his fathers wishes. https://people.com/human-interest/elon-musk-errol-musk-relationship/ https://people.com/human-interest/elon-musk-errol-musk-relat... ’“He was such a terrible human being,” Elon, 46, told the magazine. “You have no idea.”’ Daddy issues out the ass. This is actually the simplest answer, because people that pursue external validation (in Elon’s case it is very extreme, nothing is ever enough) are a genuine product of child abuse (it’s obvious emotional abuse is a major factor here). I’m not absolving Elon, just trying to understand the initial state. Edit: When Elon finally came home from the hospital, his father berated him. "I had to stand for an hour as he yelled at me and called me an idiot and told me that I was just worthless," Elon recalls. Kimbal, who had to watch the tirade, says it was the worst memory of his life. "My father just lost it, went ballistic, as he often did. He had zero compassion." https://www.cbsnews.com/news/book-excerpt-elon-musk-by-walter-isaacson/ https://www.cbsnews.com/news/book-excerpt-elon-musk-by-walte... Remember, this is a guy that showed up with a toilet to the Twitter offices and fired everyone, got on everyone’s case to sleep in the offices. He is a literal product of child abuse, and while he may not be continuing the cycle with his kids (which is hard to say because he disowned his Trans kid), he is absolutely unleashing the cycle of abuse onto others wherever he goes.
- barbazoo 1y agoSucks, dads, please show your children that you love them unconditionally.
- rurp 1y agoHe didn't just disown the Trans kid, he's disowned most of his kids. He literally has the mom's sign a contract that says the kid will have no legal claim as his child.
- simianwords 1y ago
- anonym29 1y agoIt's so disappointing to see that Twitter has only released the source code of their algorithm while all of their competitors have released both algorithms and weights.
- wetpaws 1y ago[dead]
- vips7L 1y agoAlways good to see some Scala in the wild. :)
- halfmatthalfcat 1y agoThere are dozens of us!
- OG_BME 1y agoI tried going through the latest diff, but there is so much boilerplate that I was nt able to find any real insights through skimming. Has anyone found anything useful? Interesting needle-in-a-haystack problem for LLMs to try as well.
- thewisenerd 1y agosidenote: when do you think they're going to coax GitHub to transfer the `x` username? https://github.com/x/ https://github.com/x/
- mrtksn 1y agoI want a social media where I can ssh into the servers with limited privileges, enough to see what's going on but not cause harm.
- nativeit 1y agoI want a social media where all of that is true of its owners.
- swaptr 1y agoNot sure if this is the right place to ask, but why does Bluesky feel so much faster to load and interact with compared to X? On the surface, both have similar interfaces and equally rich content, yet Bluesky consistently feels snappier and more responsive, even though it’s the newer platform.
- palmfacehn 1y agoCould be from lower usage.
- pavel_lishin 1y agoNo idea, but Twitter is functionally un-usable if you're not logged in.
- recursivecaveat 1y agoNewer is generally faster, hasn't had time to accumulate cludges and dead ends from years of evolution. The bigger factor though I would imagine is not having 100 tons of analytics tracking everything.
- cropcirclbureau 1y agoIiirc, Twitter uses some mongrel version of React Native on the web. That's why you get the 3 seconds long loading thingie whenever you open a new tab.
- swaptr 1y agoOh in that case bsky is basically the same stack, they happen to have a react-native-web app with Expo. Maybe its the dozens of analytics that is being processed every minute.
- DustinBrett 1y agoHaving a fraction the activity probably helps.
- HelloUsername 1y agoPrevious discussions: 25-apr-2022 https://news.ycombinator.com/item?id=31160546 https://news.ycombinator.com/item?id=31160546 380 comments 31-mar-2023 https://news.ycombinator.com/item?id=35391433 https://news.ycombinator.com/item?id=35391433 1185 comments
- TheAceOfHearts 1y agoI browsed through it a bit and these are some details that raised questions or which I found interesting: There's multiple mentions of slop, for example: SlopsAuthorScoreFeature in HomeTweetTypePredicates. That means everyone gets a slop score between 0 and 1, which makes me wish that it was openly visible and that people with a high slop score would get a little piggy emoji next to their name. There's a CLIENT_TWEET_TAKE_SCREENSHOT action, which is likely used to keep track of when a (mobile, presumably) client takes a screenshot. I hadn't considered this before, but for a social media app where posts are often shared externally through screenshots, keeping track of this can give you another engagement metric. They have two types of NSFW filters: isNsfw and isSoftNsfw, but I couldn't figure out the distinction. Other metadata types include: isGore, isViolent, isSpam, isLowQuality, isOcr. In ContentFeatureAdapter there's a getTweetLengthType function which shows the range for each tweet type. This is used to set TWEET_LENGTH_TYPE elsewhere. I wonder if it would help your virality to switch up your tweet lengths to regularly put out tweets which hit every length category, or if it doesn't significantly affect your potential reach. There's a hardcoded list of top-level Grok topics [0]. Just mildly interesting to see what they consider to be top-level categories. Anime has achieved a significant cultural victory by getting separated into its own major category. The timeout values for different service request types varied a lot across the application, which makes me curious about how they settled on those numbers. This is a question I've pondered in the past but haven't gotten around to researching deeply. [0] https://github.com/twitter/the-algorithm/blob/c54bec0d4e029fe34926ef3258a86ccacc0d0182/home-mixer/server/src/main/scala/com/twitter/home_mixer/model/GrokTopics.scala https://github.com/twitter/the-algorithm/blob/c54bec0d4e029f...
- saagarjha 1y agoI assume soft NSFW is non-hardcore content
- lambdaone 1y agoThis is essentially useless without the training set or the weights. It's open-source theatre.
- keepamovin 1y agoI think Elon said he would release the weights. In a video somewhere. That's what he meant - when the next major version lands, they release the previous one? People's choices can change, maybe the economic/geopolitical reality of AI race has been impressed upon him, but I think that's what he said.
- TheAceOfHearts 1y agoThis post is about the Twitter algorithm. He originally said it would be open source, but he just did a source dump 2 years ago. Now they did a new source dump with updated but heavily redacted code for the For You feed. As for his claims about opening up Grok: Elon said that they would publish the n-1 weights for Grok. However, he dragged his feet and only recently released the weights for Grok 2. So now we're up to Grok 4 but he has yet to release the weights for Grok 3 despite his claims. I think the problem with Elon is that he doesn't fully hold himself accountable for his words. If he decided that it was no longer economically viable to share Grok's weights then he should post an update about that. You cannot expect to win the goodwill of claiming to support open source and then continuously drag your feet while refusing to communicate your intentions clearly.
- guluarte 1y agoso basically pay to play