14 ms·

Many people have commented that it is empty. However what they do not realize is that there has never actually been an algorithm and that is why it is empty.

bberenberg 4y ago
I feel like this is a line from The Program (https://www.programaudioseries.com/ https://www.programaudioseries.com/)
bastardoperator 4y ago
This is most likely the correct answer. I doubt there is a single piece of code or algorithm that controls all of twitter. From my limited perspective it's just a catch phrase to simplify what is likely tens of thousands of lines of code.
- andrewmcwatters 4y ago
  Well, that's an algorithm. If you have to draw a diagram showing how edge cache nodes affect a user's results based on whether or not they signed in from New York or Phoenix, or if the downtime of a cluster affects the relevance rankings of tweets for a photography enthusiast, then guess what... That's an algorithm.
  - bastardoperator 4y ago
    I don't disagree, I might even argue that twitter is comprised of many algorithms. We can argue all software is algorithms, or we can just call it software.
    - andrewmcwatters 4y ago
      That just moves the terminology around and doesn't answer people's questions. Everyone would simply start asking, "what software or lines of code directly affect my timeline?" The "timeline algorithm" is a perfectly normal way to describe what people ask about.
  - TimTheTinker 4y ago
    This shouldn't be downvoted. Any black box with inputs and outputs is a function over discrete data, otherwise known as an algorithm.
    - samhw 4y ago
      > a function over discrete data, otherwise known as an algorithm 'A function over discrete data' is a far broader concept / far larger set than 'an algorithm'.
  - mhh__ 4y ago
    Yes it's an algorithm, but as software engineers we shouldn't pander to people who actually think there is a small flow chart dictating what happens in your search results. It's idiotic or dishonest.
    - donatj 4y ago
      I mean I'm sure Google has somewhere a UML chart of the entire process of ordering results. Your insistence otherwise is disingenuous at best, either that or you've never actually worked on any sort of major software, at least with any level of reliability/dependability.
      - mhh__ 4y ago
        Sure they have a chart but it's not "if !idpol -> make woke instead". The level of discourse I'm trying to convey like what we saw in the Zuck/Pichai hearings in Congress. No understanding of the domain at all. You or I know how software actually works, your average politician or man on the street does not.
      - fmobus 4y ago
        > I mean I'm sure Google has somewhere a UML chart of the entire process of ordering results. No, there isn't.
  - bee_rider 4y ago
    I think it is unfortunate that "the algorithm" has become the popularized phrase for the thing, though. Yes, clearly an algorithm which evaluates the entire state of the universe and uses that as an input is still an algorithm. But it is also clearly not what people are looking for. I'm pretty sure people expect to be able look at The Algorithm and find the part that is trying to destroy <whatever their hobby-horse is>. I bet whatever we see will lean more toward "big dumb pile of A/B tests."
influx 4y ago
There's not an algorithm to order timelines?
- jonwinstanley 4y ago
  If it’s a list of tweets ordered based on a kajillion ML data points that varies per user is it still an algorithm? And does every user have their own algorithm? And could it be made readable to a human?
  - abduhl 4y ago
    Yes, irrelevant, and the code somewhere is readable. Take an example like gravity: is there an algorithm for how gravity acts even though it varies per each molecule based on a kajillion other molecules? Of course there is.
    - pavlov 4y ago
      - zbentley 4y ago
        Computer systems, even AI/ML-driven ones, aren't uncomprehended the way some areas of science that we haven't developed a strong understanding of are. Systems comprised of (or generated by) comprehensible parts can be comprehended, we just often opt not to put in that work. There's a reason model interpretability is such a significant and increasingly-demanded area of expertise.
    - marcinzm 4y ago
      If I have your DNA can I recreate a perfect copy of you as you are now including your mind? It is after all the code you were made with.
      - vasco 4y ago
        The point is not to recreate it or explain it, or to have a way to say "tweet X will get rank Y" but to open source what there is. It doesn't matter if no single person can explain it, it'll still be open source and readable and the types of inputs will be known. But this doesn't look like a discussion on open source and more about politics at this point. I for one haven't worked at FAANG and think it'll be super interesting to read this code, I can't believe software engineers are complaining about a potentially super complex bleeding edge codebase for timeline recommendations / ranking being open sourced. This is going to be great reading material if it ever gets published!
andrewmcwatters 4y ago
If there is a total set of data, and a subset of it is produced for users, there is an algorithm. `SELECT TOP` is an algorithm. A ML model with a trillion parameters is an algorithm. There is no "no algorithm."
- sverhagen 4y ago
  You're not wrong. But would it be useful to define better what we mean with algorithm then? Like: we mean "non-obvious algorithm"?
  - andrewmcwatters 4y ago
    You and I might be technical enough people to suss out little details, but I suspect the average person probably also wants to know whether or not something like Twitter testing a feature flag (Yes, really!) is affecting their timeline. If it affects results in the slightest, I think that's what people are asking about. Yeah, if the new senior dev screwed up a deployment and now football fans in Dallas are suddenly getting fewer posts about the most recent athlete scandal all because a regional deployment was the only thing doing relevance record keeping, then yes, that's a part of "the algorithm."
- deleted 4y ago
  [deleted]
crate_barre 4y ago
Twitter is mostly an infrastructure operation imho, someone school me. It’s a scale-based problem space, how do you get all these tweets out in real time at minimum, and at best, how do you do some level of topic bucketing on top of it.
- Karrot_Kream 4y ago
  Right but "the algorithm" is inextricably tied in with scaling. "the algorithm" is designed to handle updates at scale and the model probably has different parts updating to different events on different cadences.
ltbarcly3 4y ago
Yes there is an algorithm. It may not be concise, it may not be found in one place, and nobody alive may be able to explain it to you, but the existence of timelines that are ordered certainly implies the existence of 'the algorithm' that is used to order timelines. I mean even if timelines were totally random, or based on some external facts, there is an algorithm that is being used to order them. This isn't just an academic distinction. Claiming 'there is no algorithm' because the algorithm is intentionally or unintentionally obfuscated or complicated has implications if that claim of 'no algorithm' is accepted. If my algorithm for approving mortgage applications is explicitly racist, I can just spread it's functionality across myriad services owned by lots of teams, make it almost impossible to figure out how it works, and then avoid any responsibility by saying 'there is no algorithm to decide loan approvals'? That would be bullshit!
AviationAtom 4y ago
You don't think recent news of the EU compelling social media companies to disclose their algorithms has anything to do with this?
- layer8 4y ago
  No, the repository is a joke (IME). Regarding the EU, the details haven’t even been nailed down yet, and the timeline is that it will only become a requirement in 2024 I believe. The goal also isn’t to publish “the algorithm”, but to give researchers and civil society representatives access to the training data and information about the most important factors controlling the algorithm, so they can make assessments on the basis of that. It is not about open-sourcing code.
  - AviationAtom 4y ago
    They don't necessarily have to publish code to the repo though, they could just as well publish papers that document their algorithm.
WORMS_EAT_WORMS 4y ago
This would be some pretty funny semantics... but, however it works, message is the same: Transparency for how these work - Search results - Comment Order - Timeline Order - Trends - Human vs code Personalization in general. Big gigantic “why” when it happens to you
yifanl 4y ago
As far as I'm concerned, the algorithm is and can only ever be an implementation detail. If the leadership of Twitter changes overnight, it'd be highly likely the types of content that gets recommended will subtly/not-so-subtly change... because the algorithm has been changed. It's an interchangeable function, it would only be publicized if it's clear to leadership that it wouldn't affect their revenue if people started trying to game towards the published algorithm.
deleted 4y ago
[deleted]
bilekas 4y ago
This, personally I was thinking in light of the new upcomming sale, the current Twitter staff might decide to 'open source' some algorithm. Which would be an effective method of protest against a sale like that. But there definitely is a relationship algo that could be considered theirs, like all social medias inflating the bubbles users all feel.
omoikane 4y ago
That sounds like a variant of this joke: https://picturesofpeoplescanningqrcodes.tumblr.com/ https://picturesofpeoplescanningqrcodes.tumblr.com/
- I-M-S 4y ago
  Joke's on them, in the last 2 years QR codes made the greatest upset of all times
NaturalPhallacy 4y ago
We have a leak revealing a "trends blacklist" at the very least. Some of their measures are detectable too: https://taishin-miyamoto.com/ShadowBan/ https://taishin-miyamoto.com/ShadowBan/
stjohnswarts 4y ago
There's clearly an (probably shifting daily) algorithm. I'm not sure what the statement the site is trying to make. Even a random number generator that selects tweets at random for your feed is an algorithm.
temp8964 4y ago
Here’s the play: interpret “the algorithm” in a stupid way, and then claim it doesn’t exist, and then make a joke about it on company repo. You know what? This does demonstrate the internal problems inside Twitter and shows the need for shakeup.
Diesel555 4y ago
Of note for anyone confused and reading the comments now, the link was to an empty, but real, twitter repository at https://github.com/twitter/the-algorithm https://github.com/twitter/the-algorithm. Now it is a 404.

bberenberg 4y ago

I feel like this is a line from The Program (https://www.programaudioseries.com/ https://www.programaudioseries.com/)

bastardoperator 4y ago

This is most likely the correct answer. I doubt there is a single piece of code or algorithm that controls all of twitter. From my limited perspective it's just a catch phrase to simplify what is likely tens of thousands of lines of code.

andrewmcwatters 4y ago
Well, that's an algorithm. If you have to draw a diagram showing how edge cache nodes affect a user's results based on whether or not they signed in from New York or Phoenix, or if the downtime of a cluster affects the relevance rankings of tweets for a photography enthusiast, then guess what... That's an algorithm.
- bastardoperator 4y ago
  I don't disagree, I might even argue that twitter is comprised of many algorithms. We can argue all software is algorithms, or we can just call it software.
  - andrewmcwatters 4y ago
    That just moves the terminology around and doesn't answer people's questions. Everyone would simply start asking, "what software or lines of code directly affect my timeline?" The "timeline algorithm" is a perfectly normal way to describe what people ask about.
- TimTheTinker 4y ago
  This shouldn't be downvoted. Any black box with inputs and outputs is a function over discrete data, otherwise known as an algorithm.
  - samhw 4y ago
    > a function over discrete data, otherwise known as an algorithm 'A function over discrete data' is a far broader concept / far larger set than 'an algorithm'.
- mhh__ 4y ago
  Yes it's an algorithm, but as software engineers we shouldn't pander to people who actually think there is a small flow chart dictating what happens in your search results. It's idiotic or dishonest.
  - donatj 4y ago
    I mean I'm sure Google has somewhere a UML chart of the entire process of ordering results. Your insistence otherwise is disingenuous at best, either that or you've never actually worked on any sort of major software, at least with any level of reliability/dependability.
    - mhh__ 4y ago
      Sure they have a chart but it's not "if !idpol -> make woke instead". The level of discourse I'm trying to convey like what we saw in the Zuck/Pichai hearings in Congress. No understanding of the domain at all. You or I know how software actually works, your average politician or man on the street does not.
    - fmobus 4y ago
      > I mean I'm sure Google has somewhere a UML chart of the entire process of ordering results. No, there isn't.
- bee_rider 4y ago
  I think it is unfortunate that "the algorithm" has become the popularized phrase for the thing, though. Yes, clearly an algorithm which evaluates the entire state of the universe and uses that as an input is still an algorithm. But it is also clearly not what people are looking for. I'm pretty sure people expect to be able look at The Algorithm and find the part that is trying to destroy <whatever their hobby-horse is>. I bet whatever we see will lean more toward "big dumb pile of A/B tests."

influx 4y ago

There's not an algorithm to order timelines?

jonwinstanley 4y ago
If it’s a list of tweets ordered based on a kajillion ML data points that varies per user is it still an algorithm? And does every user have their own algorithm? And could it be made readable to a human?
- abduhl 4y ago
  Yes, irrelevant, and the code somewhere is readable. Take an example like gravity: is there an algorithm for how gravity acts even though it varies per each molecule based on a kajillion other molecules? Of course there is.
  - pavlov 4y ago
    - zbentley 4y ago
      Computer systems, even AI/ML-driven ones, aren't uncomprehended the way some areas of science that we haven't developed a strong understanding of are. Systems comprised of (or generated by) comprehensible parts can be comprehended, we just often opt not to put in that work. There's a reason model interpretability is such a significant and increasingly-demanded area of expertise.
  - marcinzm 4y ago
    If I have your DNA can I recreate a perfect copy of you as you are now including your mind? It is after all the code you were made with.
    - vasco 4y ago
      The point is not to recreate it or explain it, or to have a way to say "tweet X will get rank Y" but to open source what there is. It doesn't matter if no single person can explain it, it'll still be open source and readable and the types of inputs will be known. But this doesn't look like a discussion on open source and more about politics at this point. I for one haven't worked at FAANG and think it'll be super interesting to read this code, I can't believe software engineers are complaining about a potentially super complex bleeding edge codebase for timeline recommendations / ranking being open sourced. This is going to be great reading material if it ever gets published!

andrewmcwatters 4y ago

If there is a total set of data, and a subset of it is produced for users, there is an algorithm. `SELECT TOP` is an algorithm. A ML model with a trillion parameters is an algorithm. There is no "no algorithm."

sverhagen 4y ago
You're not wrong. But would it be useful to define better what we mean with algorithm then? Like: we mean "non-obvious algorithm"?
- andrewmcwatters 4y ago
  You and I might be technical enough people to suss out little details, but I suspect the average person probably also wants to know whether or not something like Twitter testing a feature flag (Yes, really!) is affecting their timeline. If it affects results in the slightest, I think that's what people are asking about. Yeah, if the new senior dev screwed up a deployment and now football fans in Dallas are suddenly getting fewer posts about the most recent athlete scandal all because a regional deployment was the only thing doing relevance record keeping, then yes, that's a part of "the algorithm."
deleted 4y ago
[deleted]

crate_barre 4y ago

Twitter is mostly an infrastructure operation imho, someone school me. It’s a scale-based problem space, how do you get all these tweets out in real time at minimum, and at best, how do you do some level of topic bucketing on top of it.

Karrot_Kream 4y ago
Right but "the algorithm" is inextricably tied in with scaling. "the algorithm" is designed to handle updates at scale and the model probably has different parts updating to different events on different cadences.

ltbarcly3 4y ago

Yes there is an algorithm. It may not be concise, it may not be found in one place, and nobody alive may be able to explain it to you, but the existence of timelines that are ordered certainly implies the existence of 'the algorithm' that is used to order timelines. I mean even if timelines were totally random, or based on some external facts, there is an algorithm that is being used to order them. This isn't just an academic distinction. Claiming 'there is no algorithm' because the algorithm is intentionally or unintentionally obfuscated or complicated has implications if that claim of 'no algorithm' is accepted. If my algorithm for approving mortgage applications is explicitly racist, I can just spread it's functionality across myriad services owned by lots of teams, make it almost impossible to figure out how it works, and then avoid any responsibility by saying 'there is no algorithm to decide loan approvals'? That would be bullshit!

AviationAtom 4y ago

You don't think recent news of the EU compelling social media companies to disclose their algorithms has anything to do with this?

layer8 4y ago
No, the repository is a joke (IME). Regarding the EU, the details haven’t even been nailed down yet, and the timeline is that it will only become a requirement in 2024 I believe. The goal also isn’t to publish “the algorithm”, but to give researchers and civil society representatives access to the training data and information about the most important factors controlling the algorithm, so they can make assessments on the basis of that. It is not about open-sourcing code.
- AviationAtom 4y ago
  They don't necessarily have to publish code to the repo though, they could just as well publish papers that document their algorithm.

WORMS_EAT_WORMS 4y ago

This would be some pretty funny semantics... but, however it works, message is the same: Transparency for how these work - Search results - Comment Order - Timeline Order - Trends - Human vs code Personalization in general. Big gigantic “why” when it happens to you

yifanl 4y ago

As far as I'm concerned, the algorithm is and can only ever be an implementation detail. If the leadership of Twitter changes overnight, it'd be highly likely the types of content that gets recommended will subtly/not-so-subtly change... because the algorithm has been changed. It's an interchangeable function, it would only be publicized if it's clear to leadership that it wouldn't affect their revenue if people started trying to game towards the published algorithm.

deleted 4y ago

[deleted]

bilekas 4y ago

This, personally I was thinking in light of the new upcomming sale, the current Twitter staff might decide to 'open source' some algorithm. Which would be an effective method of protest against a sale like that. But there definitely is a relationship algo that could be considered theirs, like all social medias inflating the bubbles users all feel.

omoikane 4y ago

That sounds like a variant of this joke: https://picturesofpeoplescanningqrcodes.tumblr.com/ https://picturesofpeoplescanningqrcodes.tumblr.com/

I-M-S 4y ago
Joke's on them, in the last 2 years QR codes made the greatest upset of all times

NaturalPhallacy 4y ago

We have a leak revealing a "trends blacklist" at the very least. Some of their measures are detectable too: https://taishin-miyamoto.com/ShadowBan/ https://taishin-miyamoto.com/ShadowBan/

stjohnswarts 4y ago

There's clearly an (probably shifting daily) algorithm. I'm not sure what the statement the site is trying to make. Even a random number generator that selects tweets at random for your feed is an algorithm.

temp8964 4y ago

Here’s the play: interpret “the algorithm” in a stupid way, and then claim it doesn’t exist, and then make a joke about it on company repo. You know what? This does demonstrate the internal problems inside Twitter and shows the need for shakeup.

Diesel555 4y ago

Of note for anyone confused and reading the comments now, the link was to an empty, but real, twitter repository at https://github.com/twitter/the-algorithm https://github.com/twitter/the-algorithm. Now it is a 404.