35 ms·
twitter/the-algorithm
- deleted 4y ago[deleted]
- mrkramer 4y agoImagine having something like this for Google's and YouTube's algorithms; $100bn+ SEO industry would go bankrupt or at least they would pivot to some sort of advising but there wouldn't be the mayhem that we have today.
- bigfudge 4y agoScientific knowledge is written down and freely available but I still don’t understand most of it. I think a public algorithm would increase SEO business if anything because it would get more effective once the bullshit was debunked.
- sjtindell 4y agoResults would also become an absolute cesspool. Say what you want about how they are now, but if the people gaming it could see the exact rules, it would become completely useless.
- LugarOS 4y agoIt's empty.
- enahs-sf 4y agoDid it get taken down already? I think open-sourcing the algo would materially change the value of the deal.
- baisq 4y agoTaken down by whom? It was posted on the official github account of twitter.
- lhoff 4y agoTwitters main value is its user base (especially including most journalists, "stars", politicans, companies,...) I would say the algo itself is worthless, but my estimate would be nowhere near even 1B$.
- xsmasher 4y agoIf open-sourcing the algo allows bots to game the system then it has value to the bot wranglers; if the feed becomes full of garbage that gradually destroys Twitter's value.
- woah 4y agoAny worth that "the algorithm" has is not in it's implementation (probably pretty standard ML prediction systems), but in the training data and weights. In isolation from the data and the rest of the code it is probably pretty useless as anything other than an example of real world ML use.
- heffer 4y agoIt's a performance.
- deleted 4y ago[deleted]
- transitivebs 4y agoIs this supposed to be a joke? It's clearly an empty repo. Either this is a mistake, or this is a really, really misguided attempt at a joke from Twitter.
- dmonitor 4y agoIt was created 20 minutes ago. Maybe a WIP?
- transitivebs 4y agoFor a company of Twitter's scale and resources, any public repos are supposed to go through legal to clear (this is how things work at every FAANG I've worked at). So if it was a WIP, it'd be a private repo until it's ready to release publicly.
- nightski 4y agoWell, all FAANG companies are public.
- transitivebs 4y agoFor anyone wanting a non-empty version, check out my article on how Twitter's algorithmic feed works from last week https://transitivebullsh.it/oss-twitter-algorithm-part-1 https://transitivebullsh.it/oss-twitter-algorithm-part-1
- aeyes 4y agoHow you _think_ it works from 1000 miles above.
- koboll 4y agoThe tweet announcing it was captioned "watch this space": https://twitter.com/willnorris/status/1518694675909013504 https://twitter.com/willnorris/status/1518694675909013504 Which seems like a promise they intend to actually open source something there.
- Irishsteve 4y agoMany people have commented that it is empty. However what they do not realize is that there has never actually been an algorithm and that is why it is empty.
- bberenberg 4y agoI feel like this is a line from The Program (https://www.programaudioseries.com/ https://www.programaudioseries.com/)
- bastardoperator 4y agoThis is most likely the correct answer. I doubt there is a single piece of code or algorithm that controls all of twitter. From my limited perspective it's just a catch phrase to simplify what is likely tens of thousands of lines of code.
- andrewmcwatters 4y agoWell, that's an algorithm. If you have to draw a diagram showing how edge cache nodes affect a user's results based on whether or not they signed in from New York or Phoenix, or if the downtime of a cluster affects the relevance rankings of tweets for a photography enthusiast, then guess what... That's an algorithm.
- bastardoperator 4y agoI don't disagree, I might even argue that twitter is comprised of many algorithms. We can argue all software is algorithms, or we can just call it software.
- andrewmcwatters 4y agoThat just moves the terminology around and doesn't answer people's questions. Everyone would simply start asking, "what software or lines of code directly affect my timeline?" The "timeline algorithm" is a perfectly normal way to describe what people ask about.
- holtkam2 4y agoI don't get it ¯\_(ツ)_/¯
- CincinnatiMan 4y agoMakes me wonder how Twitter employees internally are handling the news. If they are celebrating or commiserating?
- mostlysimilar 4y agoTwitter Locks Down Product Changes After Agreeing to Musk Bid - https://finance.yahoo.com/news/twitter-locks-down-product-changes-195751959.html https://finance.yahoo.com/news/twitter-locks-down-product-ch...
- brink 4y agoLooks like a wise move to keep potential destructive protests at bay.
- daenz 4y agoYep. I imagine Twitter has a employee base with a very strong activist streak. I'm sure some of them wouldn't mind martyring themselves to make a virtuous statement.
- dominotw 4y agoonly because the job market is so good.
- systemvoltage 4y agoSubstack made it quite clear: https://twitter.com/lulumeservey/status/1511376638487019524?lang=en https://twitter.com/lulumeservey/status/1511376638487019524?... Personally, I think this is great. Leave your activism at home and don't bring that to work.
- daenz 4y agoI don't think it's very good to saboteurs.
- 4y ago
- ArtWomb 4y agoIt was all in your head ;)
- nighthawk454 4y agoSeems weird to start as a non-private repo until there's some content. Also bit of an unusual name. Can't tell if this is internal trolling or the future
- extheat 4y agoThey probably have their own internal Git hosting service. Pushing to the public Github repo could be done at a later point, but the Git repo hasn't actually been created yet here, just the project on Github.
- pavon 4y agoThe Twitter board has unanimously approved Musk's purchase, pending approval by stockholders, and Musk has stated that once he owns the company they will open source the "algorithms" for transparency. So not a troll, but yes it is odd to put up an empty repo, and announce the repo before there is anything in it.
- madeofpalk 4y agoSeems like a perfect troll/joke, especially given everything.
- oxplot 4y agoTrolling your new boss is a sure way to get your ass fired. EDIT: in case you're not aware, Musk has stated that one of the first things he'll be doing after taking Twitter private is to open source its algorithm.
- madeofpalk 4y agoFiring someone for making an innocent joke doesn’t seem very “free speech absolutist”!
- oxplot 4y ago
- mrintellectual 4y agoThis seems to be a practical joke by a Twitter engineer as opposed to an actual release.
- hunterb123 4y agoOr a commit is being prepared? It's only 20 minutes old.
- mrintellectual 4y agoIt just seems unlikely that the algorithm would be open-sourced right after a deal for Twitter is agreed upon (but before it actually goes through). I've never seen a buyout of this scale done by an individual, but I imagine the SEC and several other parties will need to be involved. At the minimum, I would make a private Github repo first, add all relevant commits, and then make it public once there's actually content.
- hunterb123 4y agoSure it's fast but it seems the deal is done. A practical joke by an employee seems more unlikely than an official release. The engineer from the OSS team at Twitter linked to it and said "watch this space" https://twitter.com/willnorris/status/1518694675909013504 https://twitter.com/willnorris/status/1518694675909013504 Sure I guess it could be that engineer gearing up for a joke, but I think it's more likely that it's a real release.
- kaladin-jasnah 4y agohttps://twitter.com/elonmusk/status/1518677066325053441 https://twitter.com/elonmusk/status/1518677066325053441 states that Elon Musk is to looking "make the algorithms open source."
- johntb86 4y agoIt seems more like a symbolic gesture.
- somishere 4y ago
- yabones 4y agoThere is something vaguely threatening about this.
- johns 4y agoWill Norris who works on OSS at Twitter posted this[0]: "watch this space https://github.com/twitter/the-algorithm https://github.com/twitter/the-algorithm" [0]: https://twitter.com/willnorris/status/1518694675909013504 https://twitter.com/willnorris/status/1518694675909013504
- bsimpson 4y agoOh hey - I didn't know he'd left Google. He was a big part of open source there too.
- willnorris 4y agosame team. different company.
- lupire 4y agodid the repo ever exist or was he joking, or teasing?
- FR10 4y agoI just checked and its gone. But when it did exist it was just an empty repo no README even.
- EMIRELADERO 4y agoKind of unrealistic but I hope Twitter now open-sources not only the algorithm but also the Rails monolith itself. Would be kind of interesting to see how everything is done
- influx 4y agoThe rails monolith is long gone.
- EMIRELADERO 4y agoIn that case the same applies for the microservices
- kaladin-jasnah 4y agoIsn't Twitter written in Scala now?
- kaladin-jasnah 4y agoThat gives even more reason to open-source it, right? If Twitter isn't using that codebase anymore.
- pddpro 4y agoDoes it remind anyone of Po and the Dragon scroll from Kung Fu Panda?
- zelon88 4y agoif ($has_blue_checkmark) show_post_to($everyone);
- rzarate 4y agoUgh, php...
- rvz 4y agoPerhaps Twitter will be the new Mozilla if it decides to open-source 'everything' then. Maybe that is where it is going.
- jdrc 4y agoI know the algorithm i use, it ends with ORDER BY date DESC.
- onychomys 4y agoSame here, and it blows my mind that we're in such a minority.
- hotpotamus 4y agoIs that actually possible on Twitter? (I have an account but don't use it.) I do believe them that when a user follows several 100+ other users, it would probably be impossible to read every tweet chronologically.
- ironmagma 4y agoImpossible to read, maybe. Impossible to display, definitely not.
- madeofpalk 4y agoYes. It is. It's been an option since... forever? Many many years at least. https://i.imgur.com/JM6saa2.png https://i.imgur.com/JM6saa2.png For about a week they made a change that prevent that chronological timeline from being the default, but they reasonably quickly rolled that back. https://www.theverge.com/2022/3/14/22977782/twitter-default-home-algorithmic-timeline-feed-reverse-chronological https://www.theverge.com/2022/3/14/22977782/twitter-default-...
- themanmaran 4y agoSort by recent is an available setting. Whether you can keep up with the volume is up to how many people you choose to follow.
- dilap 4y agoi always use chronological, atm following very few people, but in the past following too many to keep up w/. even when following too many to read everything, i preferred chrono because it would yield a coherent slice of what was happening. an unbiased sample. twitter is basically a medium for conversation. imagine there's a large party. would you rather listen to an out-of-order "most important" set parts of the conversation, or just a slice of conversation from a particular time? well, actually, both can be interesting, but generally the slice is more coherent. :-)
- g105b 4y agoCan someone explain this to me? All I can see from this link is an empty GitHub repository. Not sure what I'm missing here.
- unethical_ban 4y agoThere are elements of their algo that I think should be openly defined, and perhaps there should be some regulatory branch that reports to Congress that has full access. However, obfuscation is often necessary to countering bad actors.
- readbeard 4y agoOn the other hand, wouldn't open sourcing the algorithm help accelerate the identification of possible exploits?
- unethical_ban 4y ago"algorithm" here isn't some fancy, hard to debug code. It is the business logic of weighting tweets and how recommendations are made. There is great opportunity to abuse this by Twitter, yes. There is also a lot of money to be made. But in defense of some of that being secret, is the fact that any publicly known ruleset (with no hidden exceptions) _will_ be exploited by bad actors. Imagine if search engines told spam sites exactly why their site dropped in page rankings.
- LegitShady 4y ago>perhaps there should be some regulatory branch that reports to Congress I think only if you offer twitter users the level of first amendment protection they'd expect with a government body. Otherwise reporting to congress would be an a bold faced circumvention the first amendment. Twitter is a privately held company with no need to report to congress.
- Synaesthesia 4y agoSo nobody is being shadowbanned or suppressed?
- dimgl 4y agoLast I heard, Twitter has internal tooling that allows moderators to shadowban or suppress.
- nickysielicki 4y agoSurely you guys don’t think that twitters sorting algorithm is already factored out into its own repo. Of course it’s empty. That doesn’t mean it’s a joke, I see it as a show of goodwill — that there are a handful of people inside Twitter that are excited for transparency and for a revenue model that isn’t entirely based on ads, that are excited to get to work on this right away.
- deleted 4y ago[deleted]
- TrapLord_Rhodo 4y agoMusks first order of business?
- LegitShady 4y agoclean house
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- threeseed 4y agoAnyone who actually uses Twitter already knows the algorithm: * Chronological - reverse sort by date * Home - for all of the followed topics, recommended topics, retweets and tweets in the past day determine the estimated level of engagement, include the highest and reverse sort by date. This is likely to be a fairly basic ML model. It will be uncontroversial, technically unsophisticated and of no practical use to anyone - users, developers or researchers. This is not going to be PageRank where some genuine new insight was discovered.
- tomcam 4y agoYou forgot the gigantic amount of human intervention required to unperson people who tweet against Twitter‘s political interests
- root_axis 4y agoWhat does "unperson" mean?
- LegitShady 4y agoit comes from Orwell's 1984 - someone who has been removed from all records/books/photographs and killed. I don't think the kill part applies here.
- bigfudge 4y agoIt’s lazy shorthand for “belatedly apply their policies against hate speech to prominent politicians”.
- tomcam 4y agoLike PewDiePie, Azealia Banks, Babylon Bee, and Carpe Donktum? By the way, in the USA there is no legal support for the term hate speech. It is covered under the umbrella of the first amendment.
- 4y ago
- standyro 4y agoI tried to make a pull request already, haha. error forking repo: HTTP 403: The repository exists, but it contains no Git content. Empty repositories cannot be forked. (https://api.github.com/repos/twitter/the-algorithm/forks https://api.github.com/repos/twitter/the-algorithm/forks) My thoughts: - Explicit rules for temporary and permanent bans - Edit button - More fun and thoughtful conversations like HN - Less thought bubble Brooklyn based reporters, less VC and side grind hustle snake oil, maybe more comedians and memes?
- semitones 4y agoCare to explain "thought bubble Brooklyn based reporters"? Did you choose Brooklyn for the alliteration? Or is there something about my home I should know about?
- JohnWhigham 4y agoLarge amount of blue checkmark journalists live there, simple as that.
- phtrivier 4y agoBut just imagine if those journalists had to live without twitter ? They would have to report things from "the real world" ! I've been there. It's awfully complicated, and sometimes people disagree with you and you have to talk. Insane.
- Jcowell 4y agoI’m confused , doesn’t this already happen on Twitter? Isn’t one of the popular Christian’s on Twitter that if you say the “wrong thing” a mob will come for you ?
- standyro 4y agoIt was just a joke. I have many friends in Brooklyn. I just really dislike the culture of Twitter addicted journalists who see the world through a myopic lens of whatever is trending on Twitter is important to cover, and there’s quite a lot of them in New York.
- xena 4y agoI can't believe they missed the chance to make it a rick roll. Such a wasted opportunity.
- paxys 4y agoI'm going to guess some engineers at Twitter with Github org permissions are having fun with the "release the algorithm" discussion.
- Sirened 4y agohaha that's what I figured, someone decided to quit if Musk acquired Twitter and figured they'd just leave one last practical joke
- Barrin92 4y agowhatever will show up in this repo, I hope people realize that depending on what data you put into some algorithm you can get whatever output you want, and twitter is never going to (and neither can or should they) publish everyone's personal information and interaction on the site. So I'm not sure what the ultimate point of this exercise is other than producing faux-transparency.
- bigfudge 4y agoI still think there could be lots of interest, even without the data. In fact some of the most interesting parts would be the algorithm in the broadest sense — how tech interacts with company policy and SOP. For one small example, what aspects of moderation/banning happen automatically without any further human intervention?
- shrimpx 4y agoI don't understand the concept of open-sourcing "the algorithm". First of all, "the algorithm" is probably hundreds of thousands of lines of code, including all the tedious boilerplate like cache policies and multi-AZ logic. And second of all, doesn't the algorithm include machine learning components, which are trained on terabytes of data? That data will likely be impossible to open source. And open sourcing the neural nets without the training data is mostly meaningless from a transparency perspective?
- WORMS_EAT_WORMS 4y agoProbably right but I think trying to be transparent is better than not trying at all. Gesture carries weight to the users too. Not sure any big company has tried this before. I could be wrong, but either way looking forward to it / FWIW hope it catches on.
- threeseed 4y agoYou can open source the model code. And developers will be able to train a model using it on a subset of Twitter data. Just that the quality of the outcome won't be the same as having the full set of Twitter data.
- mrkramer 4y agoEver heard of Pseudocode which happens to be human-readable? If they are really going do it they will release source code for programmers and in general computer scientists to analyse and on top of that they will release pseudocode which non-techincal people can somewhat understand.
- ProAm 4y agoOr is it six cats in a basement with a laser pointer and a mouse? The point of releasing it is to let people know exactly why they see the tweets they do in the order they do. I hope Elon just goes back to time base ordering of tweets.
- LordDragonfang 4y agoI personally know a FAANG employee whose full time job is building tools to try and help understand why the company's recommendation algorithm picks the things it does (and more specifically, predict how changes to the algo will affect that). Even the people who build these systems barely know what the algorithm is going to do, much less why. It will be a herculean task to try and convey that to an average user.
- asd88 4y ago#drama?
- axg11 4y agoI've worked on very large scale recommendation systems at a FAANG. If Twitter's system resembles anything like ours, the concept of publishing or open sourcing "the algorithm" doesn't make sense. Even if we were to open source all associated code and publish all related documents it would be very difficult to make sense of the entire system. That is precisely why companies such as Twitter A/B test the hell out of everything. What most people think of as "the algorithm" is a complex system that receives many inputs (maybe hundreds) and has dependencies on many other internal Twitter services. Tweets likely pass through multiple filtering steps as well as scoring before you ever see them. Each of these steps is highly contextual, depending on: location, past tweets, verification status, etc. You can attempt to predict the effect of a certain change, but you never know the actual outcome until you test it. I think what will ultimately happen is that _some_ details will be published. Elon will parade that around as a victory for free speech as Twitter is now more "open". In reality, nothing of value will be gained as "the algorithm" isn't a simple function.
- Railsify 4y agoI get what you're saying. I'd imagine this repo will be closer to pseudo code than "complete" code for all systems that a tweet flows through. For example, the system that flags/remove picture of hotdogs would likely be represented as "if image contains hotdog: weight: -500"
- zeckalpha 4y agoThat could be a starting point for a DSL that realizes the algorithm.
- hama_industries 4y agoThis is likely in response to upcoming EU legislation on algorithm transparency[0]. It's not useful, but they'll need to do it eventually. [0]: https://mashable.com/article/eu-digital-services-act-big-tech?amp https://mashable.com/article/eu-digital-services-act-big-tec...
- mhh__ 4y agoOf course it doesn't make sense. I think it's just a dog whistle to the people who believe google have a guy in a room somewhere turning the "conservative search results" lever down a notch during elections.
- qgin 4y agoI have literally no idea how a "twitter algorithm" could be published on github. Maybe I've been doing recommender systems wrong.
- solenoidalslide 4y agoYou can publish data flows and models to github in addition to source code components.
- drnonsense42 4y agoApples are red. The sky is blue. Twitter shadowbans and tinkers with who sees who. I wonder what the old guard will do with the codebase over the next few months.
- arthurcolle 4y agois this performance art?
- minroot 4y agoWhy do we want to know the "algorithm"?
- bee_rider 4y agoClearly it will contain a straightforward bias against our pet interest, confirming a grand conspiracy and validating our paranoia.
- qudat 4y agoI’ve spent the better part of a decade writing open source projects for few to see. An empty repo gets hundreds of stars immediately. It’s all a popularity contest.
- 4e530344963049 4y agoNice, making it much easier to game!
- rickreynoldssf 4y agoWait until Musk finds out its a bunch of gnarly PHP 5.4 code much of which is a black box everyone is afraid to touch.
- Sirened 4y agoThe Timeline is actually just a SQL expression with 500 sub-queries
- a-dub 4y agoit's probably just a ripoff of pagerank with a separate spam filtering and banning system along with an army of contractors manually fixing it up. if twitter is a game, sinking $43bn into it is kinda like winning or losing the grand final boss level. (unclear which) wish elon would get back to facilitating the building of useful things. we still don't have a great clean energy generation story.
- Traster 4y agoAt the time of posting, Will Norris (the open source lead at twitter, admin of their github account presumably) posted this. It has 44 retweets, 193 likes, 17 quote tweets, on github it has 1.6k stars. That seems... bizarre to me?
- enw 4y agoNope. People are just excited that the Twitter cesspool might finally improve.
- ______-_-______ 4y agoThat's more stars than I'll ever have on any of my repos. Maybe I made a mistake not joining a faang
- bpodgursky 4y agoI'm very technical and I think it would still be valuable to have a list of all the things that weight into the timeline view, even without the models or underlying data. Like, there's no public admission right now of whether "shadow banning" or "ghost banning" is even officially a thing! Some transparency seems unquestionably more powerful than none, and we can work from there.
- newbamboo 4y agoThe government, at federal, state and local levels, all rely on Twitter to conduct official taxpayer funded work. Taxpayer funded work should not happen on proprietary systems that operate with zero oversight or public transparency. Elon polled Twitter users about this and the response was overwhelmingly in favor of open source and transparency. Everyone on Twitter got a vote. If you oppose transparency, as many now are, you lose your credibility. So it’s another one of Elon’s people hacks, and look at all the morons falling for it.
- NaturalPhallacy 4y agoNot "the algorithm", but you can check if twitter is silently suppressing your account here: https://taishin-miyamoto.com/ShadowBan/ https://taishin-miyamoto.com/ShadowBan/
- oxplot 4y agoMusk has repeatedly talked about "open sourcing" twitter's algorithm. Given Musk is (understandably) super impatient, this repo may be his first move. I expect this to start with bunch of readme and other high level docs and evolve into details and eventually code.
- 12ian34 4y agoSeems like the most reasonable take. This move feels in-character for Musk.
- hazb 4y ago"The algorithm" could mean a lot of things. Whatever it means, it probably spans hundreds or even thousands of services. That doesn't mean it cannot be made open-source. I imagine they'd probably start with documentation and white-papers that communicate "here's how we intend for it to work". It's seriously unlikely anyone in Twitter knows actually works how any non-trivial algorithm in the company works. To figure THAT out, they could decide to do a company-wide documentation and instrumentation push like they probably would've had to do for GDPR anyway, which is painful and boring and going to take a very long time. Failing that, they could just say 'the algorithm as it stands is no longer fit for purpose, given part of its core requirement has become that it needs to be transparent and publishable, and presumably legible. We need to make a new one. Publish the core algorithm. We probably won't deploy it in that exact state, it's going to span multi-services and so on, you obviously don't get the data we used to train the models, but we will work backwards from it and here's an open mechanism to measure how true-to-form it actually is'
- sakopov 4y agoI agree that there is no such thing as "the algorithm." It is Twitter in its entirety. And with that I have a wild question. Can Musk make Twitter fully open-source on GitHub?
- edouard-harris 4y agoAssuming Twitter is serious about publishing their feed algorithm [1], it's possible they're merely anticipating the EU's upcoming Digital Services Act which was finalized over the weekend. Among other things, the Act will compel large platforms to "make the working of their recommender algorithms (used for sorting content on the News Feed or suggesting TV shows on Netflix) transparent to users." [2] Twitter's EU user base is probably [3] above the 45 million threshold that triggers the strictest transparency requirements under the Act. So perhaps they figure if they're going to be forced to disclose anyway, they might as well do it proactively. [1] If it's even coherent to talk about their feed ranking system as a single algorithm — see the other comments in this thread. [2] https://www.theverge.com/2022/4/23/23036976/eu-digital-services-act-finalized-algorithms-targeted-advertising https://www.theverge.com/2022/4/23/23036976/eu-digital-servi... [3] https://www.statista.com/statistics/242606/number-of-active-twitter-users-in-selected-countries/ https://www.statista.com/statistics/242606/number-of-active-...
- tmaly 4y agoI could see GPT-3 being added in the empty space.
- deleted 4y ago[deleted]
- u1tron 4y agoIt's already gone.