10 ms·

I've worked on very large scale recommendation systems at a FAANG. If Twitter's system resembles anything like ours, the concept of publishing or open sourcing

by axg11 4y ago

I've worked on very large scale recommendation systems at a FAANG. If Twitter's system resembles anything like ours, the concept of publishing or open sourcing "the algorithm" doesn't make sense. Even if we were to open source all associated code and publish all related documents it would be very difficult to make sense of the entire system. That is precisely why companies such as Twitter A/B test the hell out of everything. What most people think of as "the algorithm" is a complex system that receives many inputs (maybe hundreds) and has dependencies on many other internal Twitter services. Tweets likely pass through multiple filtering steps as well as scoring before you ever see them. Each of these steps is highly contextual, depending on: location, past tweets, verification status, etc. You can attempt to predict the effect of a certain change, but you never know the actual outcome until you test it. I think what will ultimately happen is that _some_ details will be published. Elon will parade that around as a victory for free speech as Twitter is now more "open". In reality, nothing of value will be gained as "the algorithm" isn't a simple function.

Railsify 4y ago
I get what you're saying. I'd imagine this repo will be closer to pseudo code than "complete" code for all systems that a tweet flows through. For example, the system that flags/remove picture of hotdogs would likely be represented as "if image contains hotdog: weight: -500"
- zeckalpha 4y ago
  That could be a starting point for a DSL that realizes the algorithm.
hama_industries 4y ago
This is likely in response to upcoming EU legislation on algorithm transparency[0]. It's not useful, but they'll need to do it eventually. [0]: https://mashable.com/article/eu-digital-services-act-big-tech?amp https://mashable.com/article/eu-digital-services-act-big-tec...
mhh__ 4y ago
Of course it doesn't make sense. I think it's just a dog whistle to the people who believe google have a guy in a room somewhere turning the "conservative search results" lever down a notch during elections.
- tyingq 4y ago
  Not saying Google is turning down conservative search results, but they absolutely can. Using the same cycle of human raters and tweaking weights they used to push down comparison shopping sites. See http://graphics.wsj.com/google-ftc-report/ http://graphics.wsj.com/google-ftc-report/
  - mhh__ 4y ago
    It's the man in a room part I was emphasising. It's simultaneously massively incorrect about how software actually works, misunderstands social dynamics inside companies, and massively stokes the fire when it comes to conspiracy theories along these lines (in this case at least)
    - nullc 4y ago
      So you're saying google doesn't have infrastructure to allow a human specified list of keywords or domain names to "Twiddle" the results returned in varrious services such as news, search, and youtube for the purpose of artificial promotion or de-boosting?
    - mistermann 4y ago
      Are you claiming that you have comprehensive objective knowledge of what did and did not happen in privileged Twitter operations across time, or is this more like your opinion on the matter? And if the latter, how is it that some people's opinions "don't make sense" or "are dog whistles", but yours are epistemically flawless (assuming you actually believe them to be true, of course)?
    - lupire 4y ago
      There absolutely are explicit lists of controversial words that get weights on the models for up/down ranking or hiding content or flagging for manual review. It's also true that these lists are only a small part of what affects your fringe website or twitter accounts.
cscurmudgeon 4y ago
Just because it has evolved to be a complex monster that is unnaccountable doesn't mean it has to be that way in the future. There are ways to translating trained ML models and associated systems into understandable hierarchical rules. Twitter's timeline is NOT AGI.
temp8964 4y ago
I don’t understand people like you. Just because Elon Musk used the term “the algorithm” in conversation, it must be interpreted literally? And for someone like Elon Musk, he doesn’t know the complexity and need you to point out?
- stefan_ 4y ago
  It's so weird. We get it, data scientists, your implementations are a garbage mess of last minute paper submission levels and you yourself are blind to half the things happening. That doesn't mean an implicit algorithm doesn't still exist. The term is perfectly applicable and it would probably do you a world of good to abstract it to the point where you can legitimately put it into a single repo.
dsugarman 4y ago
Even if publishing the underlying algorithm isn't useful, they should be able to do research on what the end result of endless a/b testing has created in practice. I'm sure you could find the highest level heuristics and that would be extremely valuable to both the public and decision makers internally.
- nojito 4y ago
  You won’t know what they were A/B testing though.
andrewmcwatters 4y ago
For someone who worked on recommendation systems, you really don't seem to understand the concept of an "algorithm" across the abstraction of multiple systems and at different layers of the stack you worked on. In fact, a lot of people here really think what people are talking about is the equivalent of what is handled in a subroutine. No, what people are talking about when they talk about "the algorithm" is anything affecting the result set they're reading. Concepts like eventual consistency and edge computing are... well... a part of a model which laypeople, and even reasonably technical people call an "algorithm." Being pedantic about whether or not this happens in an SQL query, or across multiple codebases, or by region, doesn't escape the question.
- mistermann 4y ago
  Absolutely agree with what you say, but I wanted to pick a nit here: > Being pedantic about whether or not this happens in an SQL query, or across multiple codebases, or by region, doesn't escape the question. Actually, epistemic ~"muddying of the waters" is a well proven technique to control perceptions and public discourse. If it works on HN folks, I expect it would work much more easily on amateurs.
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
34679 4y ago
I don't believe the point was ever to convey an complete understanding of the system, but to allow for the identification of "red flags": Specific functions that would be deemed controversial or counter to equal opportunity speech.
- madeofpalk 4y ago
  Is the idea somewhere in the-algorithm there's a function called `derankGOPMembers()`? Twitter's been pretty transparent in how it "deranks" certain accounts [1]. What more would come from opening the code that certainly not include the actual database of "no no terms" (if you were to believe that exists)? [1] https://blog.twitter.com/en_us/topics/company/2022/our-ongoing-approach-to-the-war-in-ukraine https://blog.twitter.com/en_us/topics/company/2022/our-ongoi...
  - 34679 4y ago
    The idea is that people should be able to find out if there is such a function. It's taking a stance that Twitter has nothing to hide. In my opinion, it's a great way to build trust with users, something that all of the popular social media platforms lack.
  - arthurcolle 4y ago
    take a look at the Twitch leak sometime. There was a hard-coded list of bad words (largely racial epithets & other 4chan constructions). Presumably there is a way to create some kind of config file to penalize "bad users" of Twitter, I know these things existed in other orgs I've worked at
  - EricE 4y ago
    Twitter is transparent in how they do shadow bans? They still deny they are a thing but it's clear to see the network effects of them. Twitter is transparent - that's the most hilarious thing I have read today!
    - madeofpalk 4y ago
      Has there ever been any actual indication of twitter shadow bans being a real thing, or are we putting it in the same "weird conspiracy theory" category as "facebook listens using microphone to target ads"? I follow a few people in "gamer/twitch twitter" and every now and then this meme pops up of twitter secretly deranking "go live" twitch.tv tweets, which is much more palatable to these people than the reality of just no one caring about their boring tweets.
kringo 4y ago
Understood and several of us have dealt with large complex systems. Open sourcing algorithm or code is not about everyone go and analyze the same, instead when controversy or issues arise it'll be readily available for independent experts to review it.
- axg11 4y ago
  My point isn't that it's impossible to analyze. It _is_ impossible to analyze without the context of all of Twitter's other services though. Twitter are not going to open source all of their services.
- dmarcos 4y ago
  I agree. I doubt the Twitter moderation system is more complicated than the Linux kernel or any of the open source compilers. Sufficiently motivated individuals contribute, find a fix bugs all the time.
KrishnaShripad 4y ago
The openness for the algorithm is to make public how the algorithm works internally. It has nothing to do with how "novel" or "good" or "bad" the algorithm is. It is just a way to check if there is anything fishy going on in there. I don't see what the fuss is about if it only makes the defacto town square of the internet more transparent and open.
- michaelmrose 4y ago
  Twitter isn't the town square of the internet. The internet is the town square of the internet and you aren't owed a fair ranking in relation to a certain search term user or combination thereof because no singular definition of fair.
  - KrishnaShripad 4y ago
    > Twitter isn't the town square of the internet Maybe to you. You are entitled to your opinion. I won't censor you for having a contrary opinion. > owed a fair ranking Well we are getting it whether you like it or not. > because no singular definition of fair Of course there is. If New York Post's tweet about Hunter Biden's story gets blocked, I would like to see Taylor Lorenz's story about LibsofTikTok get blocked in a "fair" system. Since both are doxxing after all.
suadeo 4y ago
As someone who’s also worked in this area, I disagree with this take. There is typically clear objective function of a recommendation system. What Twitter is optimizing for is what’s of interest here. And some of the hidden business rules. It’s likely these are specified in the code in an obvious way. How exactly they achieve that is the part that is complex and relatively indecipherable. It’s possible that it’s designed in such a way the optimization objectives are also unclear, but that would indicate a bad design and be to the detriment of the company and users.
- arriu 4y ago
  Yeah seriously. If they are not able to put into words or diagrams what the algorithm is doing then the company itself has no idea how it works. And that to me would suggest it’s far from optimal. Many complicated research papers have had no issues describing their models at a high level. This should be no different.
- axg11 4y ago
  Twitter PMs would argue they've already communicated the objective function: relevance and engagement. My point is that the devil is in the details and implementation. These details are likely something that no one person understands and no one person is able to fix. The concept of being able to extract "the algorithm", factor it out from the codebase and share it with the public doesn't make sense to me. It won't be possible to fully understand how Twitter serves recommendations and ranks posts without understanding how all the different services at Twitter interact. Are they planning on open sourcing all of Twitter? Highly doubtful.
lamontcg 4y ago
Did you just tell me to go write a map/reduce function in Erlang?
actuator 4y ago
Yeah, this makes no sense. There is no golden algorithm that Facebook, Tiktok or Twitter has figured out. All these feed rankings are complex combinations of features, models coupled with weights and filters. On top of this abuse detection layers are added. Unless Musk is planning to open source user data to show what all the "scores" and "features" for all the entities are and how they were reached to, this will make no sense. The whole argument against some people being downranked has been, why me? Just writing a whitepaper to tell the general methodology, is not going to make that go away. On top of that, exposing every vector through which you measure and stop abuse, will just allow for more sophisticated abuse.
- vrfvr 4y ago
  Can you mention the source of your information?
dillondoyle 4y ago
It would be gamed 100%.
tonguez 4y ago
it’s almost like this thread is nothing but people being pedantic and saying, there isn’t just one algorithm, it’s multiple algorithms. yes do you really think elon doesn’t know that? it’s almost like he’s just trying to get the point across in the most simple and basic way possible. pointing out that the recommendation algorithm isn’t just one algorithm isn’t profound at all. this entire conversation is mostly just people who want to argue and point out that they know something about how large tech companies work. congrats, why don’t you tell elon since you’re obviously so much smarter
Traster 4y ago
It's so difficult as someone who works in technology to tolerate this idea that there is "an algorithm". Maybe this will get flagged on HN but the overwhelming feeling is to just scream "don't be such a fucking idiot, this is a $43Bn company it doesn't boil down to 50 lines of code" and hell, there are plenty of examples of 50 lines of code at my company I could spend a month really understanding and even then not understand the full ramifications. It's a stupid persons idea of how tech works.
- farias0 4y ago
  I don't think that most people actually believe "the algorithm" is like a Python script or something.
  - Kiro 4y ago
    No, it's much worse. The general public thinks it's a one line mathematical formula. The secret sauce.
  - Avicebron 4y ago
    agreed, I see this more as a simplification of how a network of interdependent systems operate, just because its not simple doesn't mean there isn't a fundamental structure the farther and farther you zoom out.
  - Traster 4y ago
    The average person on twitter isn't a software engineer. They won't know what python is. They don't know what an algorithm is. So let's be clear about what the baseline is here. There's no point talking to them about how you distribute queries using micro-services or whaever bullshit twitter's engineering team bought into this season. This idea that "the algorithm" is something you can just "publish" is a pernicious lie told by people like Musk - who knows it isn't true - to the general public who don't know better. The "algorithm" in reality is probably farcical calls to cusotm APIs that no current employees understand well enough to modify which is why Twitter hasn't changed anything in coming up to a decade - which is when all the engineering talent left.
MangoCoffee 4y ago
i'm looking forward to see what's going to be on it.
colonwqbang 4y ago
How did you work with a system that couldn't be understood even with access to full source and documentation? Surely your engineering process wasn't purely stochastic.
risyachka 4y ago
As you probably know, the algorithm is NOT the code, never has been. It can be pseudo-code or diagram or whatever that can be used to understand what logic lies behind decision making.
topspin 4y ago
> the concept of publishing or open sourcing "the algorithm" doesn't make sense Whatever. He paid for it. Private company. Do what it wants. > it would be very difficult to make sense of the entire system No. Not buying that. Difficult isn't the same as impossible and, if only to game the system (harder,) people will figure it out. And even if it isn't 100% possible to reproduce the results based on what is released significant insights will still emerge. Further, there is some ceiling on the complexity. Twitter operates at scale and that means they can't actually burn 52kWh of power for every tweet or store TiBs of metadata for every user to do the analysis or take 30 minutes to publish. Likely it's a pretty efficient system and, therefore, limited in complexity.
memish 4y ago
Even just releasing the audit trail of shadowbans, upranking, downranking would be meaningful. Full transparency of actions taken in the past and going forward.
- stjohnswarts 4y ago
  This will never happen, not even under Elon. It makes them wide open to litigation.
  - KrishnaShripad 4y ago
    > It makes them wide open to litigation How? What specifically opens them up to litigation? It is a private company. It can do anything it wants as long as it is within the bounds of law. They have every right to ban anyone. Even on flimsy grounds. The idea here is to expose all the moral wrongs and bring it to the fore. Not that they are legally in the wrong (most cases they aren't). An audit is a good place to start.
    - stjohnswarts 4y ago
      Not every country follows section 230, actually only one does.
      - KrishnaShripad 4y ago
        I don't think transparency of internal Company controls/audits has anything to do with Section 230. Section 230, in brief, only provides immunity for social media providers from being responsible for the content that is posted by users on their platform. It has nothing to do with internal company policies. Rather, Section 230 actually enables internet companies to moderate content through the Good Samaritan protection. So, it is actually worse that Twitter was moderating content in majority countries (except USA) where Section 230 wasn't even recognized. Revealing what exactly happened behind the scenes would not be a sufficient reason for litigation. When Twitter was pulled up in India for its opaque moderation policies, it tried to quote American laws for its defense. In fact, when Jan 6th happened in USA, Twitter was quick to ban accounts of those who took part in the protests. 70,000 accounts were purged from Twitter. However, Twitter refused to ban accounts of those Khalistani terrorists who vandalized the Red Fort on Jan 26th in India and only partially complied with Government of India's orders. This double standard was visible to majority of Indians. So it is not like Twitter actually follows the laws set by the Country it is operating in either. There are multiple instances where Twitter has refused to follow directions by Government of India or by the Courts in India. You can read more about it here: https://www.hindustantimes.com/india-news/have-to-follow-indian-law-govt-s-warning-to-twitter-101613068057143.html https://www.hindustantimes.com/india-news/have-to-follow-ind... So yes, moderation policies cannot be opaque. It has to be transparent. The reason for transparency is so that we know exactly why and for what reason was an account banned/shadow banned. If the Government of India sends a legal request for take down of accounts, it has to be complied with. Twitter cannot decide to invoke US laws in India. Also, read this to understand more about this issue: https://techcrunch.com/2021/08/10/twitter-now-in-compliance-with-india-new-it-rules-government-says/ https://techcrunch.com/2021/08/10/twitter-now-in-compliance-...
dekhn 4y ago
I guess I kind of think about these things not as algorithms, but as a collection of frontends and backends. Collectively, any human request (typing [ ramen shops near me ] as an example) will be handled by a bunch of different code, typically that code is structured as RPCs. We can think of the main interaction as being a query which is an RPC payload. The contents contain the user request and a wide amount of other context (either referenced by a collection of keys like cookies, or materialized like fields that specify the user's age) and the response is a web page which contains sections (the web search response to the query, as well as the ads; either these could be rendered to two different frames, or interspersed, by the result presentation engine). That query -> frontend translates into a tree or a graph of requests which collect up various bits of contextual data required to satisfy the query. For example, the query terms might be rewritten slightly and then sent to a web search backend which searches/ranks documents and returns the top matching documents on the organic web, or sent to an ads backend that returns the top matching bidders for those query terms. Again, just RPC/responce, although the actual context that the frontend and backend systems are dealing with, and use to modify the result, are truly enormous. Each of those backend systems itself was produced with an enormous amount of data processing and contextual data that is available at serving time. All of this is implemented using various algorithms; everything from the TCP algorithms that manage bandwidth to the neural networks doing inference on the joint product of the user context and the query context and the ad context, and the logging system that writes the queries and their clicks to centralized storage for more ML training. In theory though you could set up a system that compiled the full web stack, and ran the end to end of a user query, dumping all the intermediate RPCs, etc, from a modestly sized instantiation of the production system. and people could sit down and inspection what terms affected query result order, or which pages were omitted at which part of the filtering, or what data was logged. It would be hell for a team to maintain and keep up to date wrt the production system, but many folks do this any way to have a simple version of the system around so they can make quick changes and see if it breaks part of the complex system without doing a full deployment.
karmasimida 4y ago
Let alone those 10s of GB of embeddings which might be user identifiable, without those, how do they claim to 'open source' the algorithm?
abledon 4y ago
theres gonna be a few crazy guys who devote their entire time to understanding some obscure part of this algorithm

by axg11 4y ago

Railsify 4y ago

I get what you're saying. I'd imagine this repo will be closer to pseudo code than "complete" code for all systems that a tweet flows through. For example, the system that flags/remove picture of hotdogs would likely be represented as "if image contains hotdog: weight: -500"

zeckalpha 4y ago
That could be a starting point for a DSL that realizes the algorithm.

hama_industries 4y ago

This is likely in response to upcoming EU legislation on algorithm transparency[0]. It's not useful, but they'll need to do it eventually. [0]: https://mashable.com/article/eu-digital-services-act-big-tech?amp https://mashable.com/article/eu-digital-services-act-big-tec...

mhh__ 4y ago

Of course it doesn't make sense. I think it's just a dog whistle to the people who believe google have a guy in a room somewhere turning the "conservative search results" lever down a notch during elections.

tyingq 4y ago
Not saying Google is turning down conservative search results, but they absolutely can. Using the same cycle of human raters and tweaking weights they used to push down comparison shopping sites. See http://graphics.wsj.com/google-ftc-report/ http://graphics.wsj.com/google-ftc-report/
- mhh__ 4y ago
  It's the man in a room part I was emphasising. It's simultaneously massively incorrect about how software actually works, misunderstands social dynamics inside companies, and massively stokes the fire when it comes to conspiracy theories along these lines (in this case at least)
  - nullc 4y ago
    So you're saying google doesn't have infrastructure to allow a human specified list of keywords or domain names to "Twiddle" the results returned in varrious services such as news, search, and youtube for the purpose of artificial promotion or de-boosting?
  - mistermann 4y ago
    Are you claiming that you have comprehensive objective knowledge of what did and did not happen in privileged Twitter operations across time, or is this more like your opinion on the matter? And if the latter, how is it that some people's opinions "don't make sense" or "are dog whistles", but yours are epistemically flawless (assuming you actually believe them to be true, of course)?
  - lupire 4y ago
    There absolutely are explicit lists of controversial words that get weights on the models for up/down ranking or hiding content or flagging for manual review. It's also true that these lists are only a small part of what affects your fringe website or twitter accounts.

cscurmudgeon 4y ago

Just because it has evolved to be a complex monster that is unnaccountable doesn't mean it has to be that way in the future. There are ways to translating trained ML models and associated systems into understandable hierarchical rules. Twitter's timeline is NOT AGI.

temp8964 4y ago

I don’t understand people like you. Just because Elon Musk used the term “the algorithm” in conversation, it must be interpreted literally? And for someone like Elon Musk, he doesn’t know the complexity and need you to point out?

stefan_ 4y ago
It's so weird. We get it, data scientists, your implementations are a garbage mess of last minute paper submission levels and you yourself are blind to half the things happening. That doesn't mean an implicit algorithm doesn't still exist. The term is perfectly applicable and it would probably do you a world of good to abstract it to the point where you can legitimately put it into a single repo.

dsugarman 4y ago

Even if publishing the underlying algorithm isn't useful, they should be able to do research on what the end result of endless a/b testing has created in practice. I'm sure you could find the highest level heuristics and that would be extremely valuable to both the public and decision makers internally.

nojito 4y ago
You won’t know what they were A/B testing though.

andrewmcwatters 4y ago

For someone who worked on recommendation systems, you really don't seem to understand the concept of an "algorithm" across the abstraction of multiple systems and at different layers of the stack you worked on. In fact, a lot of people here really think what people are talking about is the equivalent of what is handled in a subroutine. No, what people are talking about when they talk about "the algorithm" is anything affecting the result set they're reading. Concepts like eventual consistency and edge computing are... well... a part of a model which laypeople, and even reasonably technical people call an "algorithm." Being pedantic about whether or not this happens in an SQL query, or across multiple codebases, or by region, doesn't escape the question.

mistermann 4y ago
Absolutely agree with what you say, but I wanted to pick a nit here: > Being pedantic about whether or not this happens in an SQL query, or across multiple codebases, or by region, doesn't escape the question. Actually, epistemic ~"muddying of the waters" is a well proven technique to control perceptions and public discourse. If it works on HN folks, I expect it would work much more easily on amateurs.

deleted 4y ago

[deleted]

deleted 4y ago

[deleted]

34679 4y ago

I don't believe the point was ever to convey an complete understanding of the system, but to allow for the identification of "red flags": Specific functions that would be deemed controversial or counter to equal opportunity speech.

madeofpalk 4y ago
Is the idea somewhere in the-algorithm there's a function called `derankGOPMembers()`? Twitter's been pretty transparent in how it "deranks" certain accounts [1]. What more would come from opening the code that certainly not include the actual database of "no no terms" (if you were to believe that exists)? [1] https://blog.twitter.com/en_us/topics/company/2022/our-ongoing-approach-to-the-war-in-ukraine https://blog.twitter.com/en_us/topics/company/2022/our-ongoi...
- 34679 4y ago
  The idea is that people should be able to find out if there is such a function. It's taking a stance that Twitter has nothing to hide. In my opinion, it's a great way to build trust with users, something that all of the popular social media platforms lack.
- arthurcolle 4y ago
  take a look at the Twitch leak sometime. There was a hard-coded list of bad words (largely racial epithets & other 4chan constructions). Presumably there is a way to create some kind of config file to penalize "bad users" of Twitter, I know these things existed in other orgs I've worked at
- EricE 4y ago
  Twitter is transparent in how they do shadow bans? They still deny they are a thing but it's clear to see the network effects of them. Twitter is transparent - that's the most hilarious thing I have read today!
  - madeofpalk 4y ago
    Has there ever been any actual indication of twitter shadow bans being a real thing, or are we putting it in the same "weird conspiracy theory" category as "facebook listens using microphone to target ads"? I follow a few people in "gamer/twitch twitter" and every now and then this meme pops up of twitter secretly deranking "go live" twitch.tv tweets, which is much more palatable to these people than the reality of just no one caring about their boring tweets.

kringo 4y ago

Understood and several of us have dealt with large complex systems. Open sourcing algorithm or code is not about everyone go and analyze the same, instead when controversy or issues arise it'll be readily available for independent experts to review it.

axg11 4y ago
My point isn't that it's impossible to analyze. It _is_ impossible to analyze without the context of all of Twitter's other services though. Twitter are not going to open source all of their services.
dmarcos 4y ago
I agree. I doubt the Twitter moderation system is more complicated than the Linux kernel or any of the open source compilers. Sufficiently motivated individuals contribute, find a fix bugs all the time.

KrishnaShripad 4y ago

The openness for the algorithm is to make public how the algorithm works internally. It has nothing to do with how "novel" or "good" or "bad" the algorithm is. It is just a way to check if there is anything fishy going on in there. I don't see what the fuss is about if it only makes the defacto town square of the internet more transparent and open.

michaelmrose 4y ago
Twitter isn't the town square of the internet. The internet is the town square of the internet and you aren't owed a fair ranking in relation to a certain search term user or combination thereof because no singular definition of fair.
- KrishnaShripad 4y ago
  > Twitter isn't the town square of the internet Maybe to you. You are entitled to your opinion. I won't censor you for having a contrary opinion. > owed a fair ranking Well we are getting it whether you like it or not. > because no singular definition of fair Of course there is. If New York Post's tweet about Hunter Biden's story gets blocked, I would like to see Taylor Lorenz's story about LibsofTikTok get blocked in a "fair" system. Since both are doxxing after all.

suadeo 4y ago

As someone who’s also worked in this area, I disagree with this take. There is typically clear objective function of a recommendation system. What Twitter is optimizing for is what’s of interest here. And some of the hidden business rules. It’s likely these are specified in the code in an obvious way. How exactly they achieve that is the part that is complex and relatively indecipherable. It’s possible that it’s designed in such a way the optimization objectives are also unclear, but that would indicate a bad design and be to the detriment of the company and users.

arriu 4y ago
Yeah seriously. If they are not able to put into words or diagrams what the algorithm is doing then the company itself has no idea how it works. And that to me would suggest it’s far from optimal. Many complicated research papers have had no issues describing their models at a high level. This should be no different.
axg11 4y ago
Twitter PMs would argue they've already communicated the objective function: relevance and engagement. My point is that the devil is in the details and implementation. These details are likely something that no one person understands and no one person is able to fix. The concept of being able to extract "the algorithm", factor it out from the codebase and share it with the public doesn't make sense to me. It won't be possible to fully understand how Twitter serves recommendations and ranks posts without understanding how all the different services at Twitter interact. Are they planning on open sourcing all of Twitter? Highly doubtful.

lamontcg 4y ago

Did you just tell me to go write a map/reduce function in Erlang?

actuator 4y ago

Yeah, this makes no sense. There is no golden algorithm that Facebook, Tiktok or Twitter has figured out. All these feed rankings are complex combinations of features, models coupled with weights and filters. On top of this abuse detection layers are added. Unless Musk is planning to open source user data to show what all the "scores" and "features" for all the entities are and how they were reached to, this will make no sense. The whole argument against some people being downranked has been, why me? Just writing a whitepaper to tell the general methodology, is not going to make that go away. On top of that, exposing every vector through which you measure and stop abuse, will just allow for more sophisticated abuse.

vrfvr 4y ago
Can you mention the source of your information?

dillondoyle 4y ago

It would be gamed 100%.

tonguez 4y ago

it’s almost like this thread is nothing but people being pedantic and saying, there isn’t just one algorithm, it’s multiple algorithms. yes do you really think elon doesn’t know that? it’s almost like he’s just trying to get the point across in the most simple and basic way possible. pointing out that the recommendation algorithm isn’t just one algorithm isn’t profound at all. this entire conversation is mostly just people who want to argue and point out that they know something about how large tech companies work. congrats, why don’t you tell elon since you’re obviously so much smarter

Traster 4y ago

It's so difficult as someone who works in technology to tolerate this idea that there is "an algorithm". Maybe this will get flagged on HN but the overwhelming feeling is to just scream "don't be such a fucking idiot, this is a $43Bn company it doesn't boil down to 50 lines of code" and hell, there are plenty of examples of 50 lines of code at my company I could spend a month really understanding and even then not understand the full ramifications. It's a stupid persons idea of how tech works.

farias0 4y ago
I don't think that most people actually believe "the algorithm" is like a Python script or something.
- Kiro 4y ago
  No, it's much worse. The general public thinks it's a one line mathematical formula. The secret sauce.
- Avicebron 4y ago
  agreed, I see this more as a simplification of how a network of interdependent systems operate, just because its not simple doesn't mean there isn't a fundamental structure the farther and farther you zoom out.
- Traster 4y ago
  The average person on twitter isn't a software engineer. They won't know what python is. They don't know what an algorithm is. So let's be clear about what the baseline is here. There's no point talking to them about how you distribute queries using micro-services or whaever bullshit twitter's engineering team bought into this season. This idea that "the algorithm" is something you can just "publish" is a pernicious lie told by people like Musk - who knows it isn't true - to the general public who don't know better. The "algorithm" in reality is probably farcical calls to cusotm APIs that no current employees understand well enough to modify which is why Twitter hasn't changed anything in coming up to a decade - which is when all the engineering talent left.

MangoCoffee 4y ago

i'm looking forward to see what's going to be on it.

colonwqbang 4y ago

How did you work with a system that couldn't be understood even with access to full source and documentation? Surely your engineering process wasn't purely stochastic.

risyachka 4y ago

As you probably know, the algorithm is NOT the code, never has been. It can be pseudo-code or diagram or whatever that can be used to understand what logic lies behind decision making.

topspin 4y ago

> the concept of publishing or open sourcing "the algorithm" doesn't make sense Whatever. He paid for it. Private company. Do what it wants. > it would be very difficult to make sense of the entire system No. Not buying that. Difficult isn't the same as impossible and, if only to game the system (harder,) people will figure it out. And even if it isn't 100% possible to reproduce the results based on what is released significant insights will still emerge. Further, there is some ceiling on the complexity. Twitter operates at scale and that means they can't actually burn 52kWh of power for every tweet or store TiBs of metadata for every user to do the analysis or take 30 minutes to publish. Likely it's a pretty efficient system and, therefore, limited in complexity.

memish 4y ago

Even just releasing the audit trail of shadowbans, upranking, downranking would be meaningful. Full transparency of actions taken in the past and going forward.

stjohnswarts 4y ago
This will never happen, not even under Elon. It makes them wide open to litigation.
- KrishnaShripad 4y ago
  > It makes them wide open to litigation How? What specifically opens them up to litigation? It is a private company. It can do anything it wants as long as it is within the bounds of law. They have every right to ban anyone. Even on flimsy grounds. The idea here is to expose all the moral wrongs and bring it to the fore. Not that they are legally in the wrong (most cases they aren't). An audit is a good place to start.
  - stjohnswarts 4y ago
    Not every country follows section 230, actually only one does.
    - KrishnaShripad 4y ago
      I don't think transparency of internal Company controls/audits has anything to do with Section 230. Section 230, in brief, only provides immunity for social media providers from being responsible for the content that is posted by users on their platform. It has nothing to do with internal company policies. Rather, Section 230 actually enables internet companies to moderate content through the Good Samaritan protection. So, it is actually worse that Twitter was moderating content in majority countries (except USA) where Section 230 wasn't even recognized. Revealing what exactly happened behind the scenes would not be a sufficient reason for litigation. When Twitter was pulled up in India for its opaque moderation policies, it tried to quote American laws for its defense. In fact, when Jan 6th happened in USA, Twitter was quick to ban accounts of those who took part in the protests. 70,000 accounts were purged from Twitter. However, Twitter refused to ban accounts of those Khalistani terrorists who vandalized the Red Fort on Jan 26th in India and only partially complied with Government of India's orders. This double standard was visible to majority of Indians. So it is not like Twitter actually follows the laws set by the Country it is operating in either. There are multiple instances where Twitter has refused to follow directions by Government of India or by the Courts in India. You can read more about it here: https://www.hindustantimes.com/india-news/have-to-follow-indian-law-govt-s-warning-to-twitter-101613068057143.html https://www.hindustantimes.com/india-news/have-to-follow-ind... So yes, moderation policies cannot be opaque. It has to be transparent. The reason for transparency is so that we know exactly why and for what reason was an account banned/shadow banned. If the Government of India sends a legal request for take down of accounts, it has to be complied with. Twitter cannot decide to invoke US laws in India. Also, read this to understand more about this issue: https://techcrunch.com/2021/08/10/twitter-now-in-compliance-with-india-new-it-rules-government-says/ https://techcrunch.com/2021/08/10/twitter-now-in-compliance-...

dekhn 4y ago

I guess I kind of think about these things not as algorithms, but as a collection of frontends and backends. Collectively, any human request (typing [ ramen shops near me ] as an example) will be handled by a bunch of different code, typically that code is structured as RPCs. We can think of the main interaction as being a query which is an RPC payload. The contents contain the user request and a wide amount of other context (either referenced by a collection of keys like cookies, or materialized like fields that specify the user's age) and the response is a web page which contains sections (the web search response to the query, as well as the ads; either these could be rendered to two different frames, or interspersed, by the result presentation engine). That query -> frontend translates into a tree or a graph of requests which collect up various bits of contextual data required to satisfy the query. For example, the query terms might be rewritten slightly and then sent to a web search backend which searches/ranks documents and returns the top matching documents on the organic web, or sent to an ads backend that returns the top matching bidders for those query terms. Again, just RPC/responce, although the actual context that the frontend and backend systems are dealing with, and use to modify the result, are truly enormous. Each of those backend systems itself was produced with an enormous amount of data processing and contextual data that is available at serving time. All of this is implemented using various algorithms; everything from the TCP algorithms that manage bandwidth to the neural networks doing inference on the joint product of the user context and the query context and the ad context, and the logging system that writes the queries and their clicks to centralized storage for more ML training. In theory though you could set up a system that compiled the full web stack, and ran the end to end of a user query, dumping all the intermediate RPCs, etc, from a modestly sized instantiation of the production system. and people could sit down and inspection what terms affected query result order, or which pages were omitted at which part of the filtering, or what data was logged. It would be hell for a team to maintain and keep up to date wrt the production system, but many folks do this any way to have a simple version of the system around so they can make quick changes and see if it breaks part of the complex system without doing a full deployment.

karmasimida 4y ago

Let alone those 10s of GB of embeddings which might be user identifiable, without those, how do they claim to 'open source' the algorithm?

abledon 4y ago

theres gonna be a few crazy guys who devote their entire time to understanding some obscure part of this algorithm