5 ms·

I don't understand the concept of open-sourcing "the algorithm". First of all, "the algorithm" is probably hundreds of thousands of lines of code, including al

I don't understand the concept of open-sourcing "the algorithm". First of all, "the algorithm" is probably hundreds of thousands of lines of code, including all the tedious boilerplate like cache policies and multi-AZ logic. And second of all, doesn't the algorithm include machine learning components, which are trained on terabytes of data? That data will likely be impossible to open source. And open sourcing the neural nets without the training data is mostly meaningless from a transparency perspective?

WORMS_EAT_WORMS 5y ago
Probably right but I think trying to be transparent is better than not trying at all. Gesture carries weight to the users too. Not sure any big company has tried this before. I could be wrong, but either way looking forward to it / FWIW hope it catches on.
threeseed 5y ago
You can open source the model code. And developers will be able to train a model using it on a subset of Twitter data. Just that the quality of the outcome won't be the same as having the full set of Twitter data.
mrkramer 5y ago
Ever heard of Pseudocode which happens to be human-readable? If they are really going do it they will release source code for programmers and in general computer scientists to analyse and on top of that they will release pseudocode which non-techincal people can somewhat understand.
ProAm 5y ago
Or is it six cats in a basement with a laser pointer and a mouse? The point of releasing it is to let people know exactly why they see the tweets they do in the order they do. I hope Elon just goes back to time base ordering of tweets.
- LordDragonfang 5y ago
  I personally know a FAANG employee whose full time job is building tools to try and help understand why the company's recommendation algorithm picks the things it does (and more specifically, predict how changes to the algo will affect that). Even the people who build these systems barely know what the algorithm is going to do, much less why. It will be a herculean task to try and convey that to an average user.
  - shrimpx 5y ago
    Right. Also, it takes a new eng hire N years to gain a working understanding of the small subset of code that their team produces.
- nthot 5y ago
  Disclaimer: I'm pretty new to twitter, so I may be misunderstanding something. On my Twitter Home Screen there are three little stars on the upper right of the central container which allow you to toggle between "top tweets" (ie. 'The Algorithm') and "latest tweets" which is the time based ordering.
  - UncleMeat 5y ago
    What you are misunderstanding is that there is more social clout to be gained by complaining about twitter than exploring the available features. The ability to sort by latest doesn't stop people from feeling self righteous by demanding this feature exist.
  - stjohnswarts 5y ago
    You understand it, it's just like instagram, facebook, reddit etc, you let them pick something for you via some algo or you order by date.
- buu700 5y ago
  Maybe it's a room full of lava lamps.
vijaybritto 5y ago
The whole point of the repo seems to just show that there's nothing called as "the algorithm" It's probably something like 100s of 1000s of algorithms doing different things
owaislone 5y ago
Open sourcing is in this is not about the implementation or the CS algorithm. It is more about transparency. I think the idea is that the public should know how tweets are ranked, why tweets show up in timelines and which timelines, what makes tweets popular etc. Imagine Google publishing a document detailing how their system ranks pages aka publishes internal SEO rules officially. I don't know if it is a good idea or not. People with enough resources might be able to game the system (if they don't do it already).
- Traster 5y ago
  The only change is that people with resource would stop guessing how to game the system, and start employing people to ensure they systematically game the system.
- rootusrootus 5y ago
  > if they don't do it already That's an interesting point. A practical description of the algorithm from the perspective of someone trying to game it may be more useful than anything Twitter or Google would release.
jonas21 5y ago
This is pretty standard in the machine learning world. You'll open-source the code and weights trained on a public data set (these are often licensed specifically for non-commercial use). But in production, you'll be using different weights trained on a proprietary data set.
thorgutierrez 5y ago
If it's too complicated, there is a good chance Elon will ask to simplify it until it can be open sourced.
imtemplain 5y ago

WORMS_EAT_WORMS 5y ago

Probably right but I think trying to be transparent is better than not trying at all. Gesture carries weight to the users too. Not sure any big company has tried this before. I could be wrong, but either way looking forward to it / FWIW hope it catches on.

threeseed 5y ago

You can open source the model code. And developers will be able to train a model using it on a subset of Twitter data. Just that the quality of the outcome won't be the same as having the full set of Twitter data.

mrkramer 5y ago

Ever heard of Pseudocode which happens to be human-readable? If they are really going do it they will release source code for programmers and in general computer scientists to analyse and on top of that they will release pseudocode which non-techincal people can somewhat understand.

ProAm 5y ago

Or is it six cats in a basement with a laser pointer and a mouse? The point of releasing it is to let people know exactly why they see the tweets they do in the order they do. I hope Elon just goes back to time base ordering of tweets.

LordDragonfang 5y ago
I personally know a FAANG employee whose full time job is building tools to try and help understand why the company's recommendation algorithm picks the things it does (and more specifically, predict how changes to the algo will affect that). Even the people who build these systems barely know what the algorithm is going to do, much less why. It will be a herculean task to try and convey that to an average user.
- shrimpx 5y ago
  Right. Also, it takes a new eng hire N years to gain a working understanding of the small subset of code that their team produces.
nthot 5y ago
Disclaimer: I'm pretty new to twitter, so I may be misunderstanding something. On my Twitter Home Screen there are three little stars on the upper right of the central container which allow you to toggle between "top tweets" (ie. 'The Algorithm') and "latest tweets" which is the time based ordering.
- UncleMeat 5y ago
  What you are misunderstanding is that there is more social clout to be gained by complaining about twitter than exploring the available features. The ability to sort by latest doesn't stop people from feeling self righteous by demanding this feature exist.
- stjohnswarts 5y ago
  You understand it, it's just like instagram, facebook, reddit etc, you let them pick something for you via some algo or you order by date.
buu700 5y ago
Maybe it's a room full of lava lamps.

vijaybritto 5y ago

The whole point of the repo seems to just show that there's nothing called as "the algorithm" It's probably something like 100s of 1000s of algorithms doing different things

owaislone 5y ago

Open sourcing is in this is not about the implementation or the CS algorithm. It is more about transparency. I think the idea is that the public should know how tweets are ranked, why tweets show up in timelines and which timelines, what makes tweets popular etc. Imagine Google publishing a document detailing how their system ranks pages aka publishes internal SEO rules officially. I don't know if it is a good idea or not. People with enough resources might be able to game the system (if they don't do it already).

Traster 5y ago
The only change is that people with resource would stop guessing how to game the system, and start employing people to ensure they systematically game the system.
rootusrootus 5y ago
> if they don't do it already That's an interesting point. A practical description of the algorithm from the perspective of someone trying to game it may be more useful than anything Twitter or Google would release.

jonas21 5y ago

This is pretty standard in the machine learning world. You'll open-source the code and weights trained on a public data set (these are often licensed specifically for non-commercial use). But in production, you'll be using different weights trained on a proprietary data set.

thorgutierrez 5y ago

If it's too complicated, there is a good chance Elon will ask to simplify it until it can be open sourced.

imtemplain 5y ago