34 ms·
From https://github.com/twitter/the-algorithm/blob/7f90d0ca342b928b479b512ec51ac2c3821f5922/home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_c
by tric 4y ago
From https://github.com/twitter/the-algorithm/blob/7f90d0ca342b928b479b512ec51ac2c3821f5922/home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/HomeTweetTypePredicates.scala#L225 https://github.com/twitter/the-algorithm/blob/7f90d0ca342b92...
(
"author_is_elon",
candidate =>
candidate
.getOrElse(AuthorIdFeature, None).contains(candidate.getOrElse(DDGStatsElonFeature, 0L))),
(
"author_is_power_user",
candidate =>
candidate
.getOrElse(AuthorIdFeature, None)
.exists(candidate.getOrElse(DDGStatsVitsFeature, Set.empty[Long]).contains)),
(
"author_is_democrat",
candidate =>
candidate
.getOrElse(AuthorIdFeature, None)
.exists(candidate.getOrElse(DDGStatsDemocratsFeature, Set.empty[Long]).contains)),
(
"author_is_republican",
candidate =>
candidate
.getOrElse(AuthorIdFeature, None)
.exists(candidate.getOrElse(DDGStatsRepublicansFeature, Set.empty[Long]).contains)),
)
- jawns 4y agoThe author_is_elon flag doesn't surprise me, but the two political designators are somewhat shocking. I'd sure like to know what changes based on what Twitter knows about your political affiliation.
- 6nf 4y agoSo many questions. How are users tagged D or R? Is that a manual process or automated somehow? What is the effect of these tags? Can I find out if my Twitter account is in one of those buckets?
- deleted 4y ago[deleted]
- moffkalast 4y agoEspecially for people that aren't... you know.. Americans. Unless they mean actual public figure party members which are known and probably verified.
- userbinator 4y agoThen it probably means which side you'd be on if you were American.
- smegger001 4y agoyou could probably algorithmically determine it in most cases based on any number of indicators from phrases used, to communities interacted with, which hashtages are included, which cohort retweets and likes most etc... thats not even getting into simply tagging political figures with the party they officially affiliate themselves with
- abracadaniel 4y agoAnd how are they choosing to balance them, per capita, or just both sides should get 50%? It seems pretty clear they are making editorial decisions here. Does that break their section 230 protections?
- astrange 4y ago"Section 230 protections" have nothing to do with "editorial decisions"
- KarlKemp 4y agoPlease read something. Preferably section 230, which is short. Alternatively something entirely different, as long as that keeps you occupied.
- krapp 4y ago>It seems pretty clear they are making editorial decisions here. Does that break their section 230 protections? No. https://www.techdirt.com/2020/06/23/hello-youve-been-referred-here-because-youre-wrong-about-section-230-communications-decency-act/ https://www.techdirt.com/2020/06/23/hello-youve-been-referre...
- cma 4y agoLikely they are tracking performance verified politician accounts based on registered party affiliation. Why republican should count equal in the evaluation metric to democrat when nunerically there are less republican voters, let alone proportions on Twitter, is another question.
- stouset 4y agoI suspect that these are used for metrics tracking rather than being fed back into the recommendation engine. But there's no real way to know for sure given the limited release. These predicates aren't actually used anywhere in the code that's been made available.
- krapp 4y agoThe repo suggests it's about tracking engagement metrics[0], so Team Red people see more Team Red content and vice versa. Nothing nefarious. [0]https://github.com/twitter/the-algorithm/blob/7f90d0ca342b928b479b512ec51ac2c3821f5922/home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/RequestQueryFeatureHydrator.scala#L86 https://github.com/twitter/the-algorithm/blob/7f90d0ca342b92...
- jeron 4y agoYou say it’s not nefarious but isn’t that how echo chambers are created?
- krapp 4y agoI don't believe echo chambers are nefarious - there's no hidden agenda involved with them. That's just how recommendation algorithms work, and it's what most people want. But if someone finds some code that suppresses recommendations from a specific political ideology across the board, that would be nefarious, IMO.
- thfuran 4y agoEcho chambers may not be nefarious, but they are insidious.
- krapp 4y agoThey can be if they're involuntary and inescapable, but neither is the case for Twitter. It's designed around letting you curate your own feed, but it also constantly throws random stuff in through retweets and quote tweets - which is what people hate the most about the platform.
- ClumsyPilot 4y ago[flagged]
- 4y ago
- jen20 4y agoIt's not that shocking... Half the people that got promoted on my timeline were perpetually candidates for elections I couldn't vote in, and they _self-identified_ as Republican or Democrat in their own bios, or via the registration of their candidacy... This is why I exclusively used to use Twitter in the "people I follow only" mode, and simply shut my account down when they pushed harder on the algorithm.
- jandrese 4y agoI thought it was interesting how it explicitly doesn't boost independents. So much of the two-party system is self-reinforcing.
- wahnfrieden 4y agoOr anti-statists! Not everyone who engages with politics is a bootlicker for authoritarianism but it sure feels like there’s no space made for this perspective (obviously)
- emodendroket 4y agoThe vast majority of self-proclaimed independents vote with one party just as reliably as registered members.
- philistine 4y agoAmerican's lax attitude towards cultivating more than two parties is literally killing the republic from the inside.
- airstrike 4y agoYou'd think a country that played a central role in a global, decades-long unstable regime of bipolar power that routinely pushed mankind to the brink of nuclear oblivion would know better than to have a bipolar electoral system
- robertlagrant 4y ago> routinely pushed mankind to the brink of nuclear oblivion I'm struggling to think of a reason why this is anything but bad faith nonsense.
- weaksauce 4y agohave you never heard of the many times we almost came to setting off full scale nuclear warfare because of the bipolar power war of the ussr v usa et.al.? the only thing that saved us was cooler heads that prevailed on both sides.
- JustSomeNobody 4y agoExactly. I don't see NPA (no party affiliation) anywhere.
- ldoughty 4y agoDid you not get the memo? "If you're not with us, you're against us". It's probably more of a conservative/liberal identifier based on US political party ideals... And they likely would filter any metrics from this by the users country
- partiallypro 4y agoFacebook guesses your political affiliation as well, you can even look über your settings to see what they guessed.
- deleted 4y ago[deleted]
- GaryNumanVevo 4y agoI wonder who's on the "VIT" (Very Important Tweeter) list?
- minimaxir 4y agoThere's a few: https://www.rollingstone.com/culture/culture-news/twitter-vip-list-boosted-tweets-report-1234704651/ https://www.rollingstone.com/culture/culture-news/twitter-vi...
- wahnfrieden 4y agoPeople like Ben Shapiro, Glenn Greenwald, @catturd2
- fortuna86 4y agoRight wing high engagement accounts. Through in one or two accounts like @AOC for "balance"
- coolspot 4y agoBTW: How @aoc got three-letter handle?
- fknorangesite 4y agoProbably exactly the most boring, mundane way you would expect: they managed to contact the owner of what was probably a dead/inactive account and then paid them some money for it.
- rhaway84773 4y agoAOC is also not a very common/popular three letter combination.
- foderking 4y ago
- minimaxir 4y agoThe full list of model features in that file is interesting. I am surprised at the number of inherently redundant and colinear features, though. (e.g. has_1_image, has_2_images, has_3_images, has_4_images)
- qyph 4y agoThose aren't redundant or collinear though? Maybe you are surprised they didn't encode this as an integer "num_images"? It is fairly common to one hot encode ordinal variables with only a few common/possible values this way.
- minimaxir 4y agoTrue, it still seems odd to encode an explicitly ordinal variable as categorical (particularly one with a small finite range, in contrast to the follower logarithmic bucket ones), but Twitter's layout is weird enough that it could be a impactful difference in terms of engagement.
- disgruntledphd2 4y agoThis is (weirdly) common in production ML codebases written by software engineers. Like you, I have no idea why unless it's a memory optimisation (where you count 4+ as many).
- ladon86 4y agoHaving every column as a boolean (0/1) means you can treat it as a bitmap. As an (entirely fictional) example, imagine if you wanted to get the features of a thread instead of a single tweet. You could do it as a union of all the tweets: threadFeatures = tweet1 | tweet2 | tweet2
- disgruntledphd2 4y agoOk that makes lots of sense from an engineering perspective. It's pretty insane from a statistical perspective though, which I think was the original point.
- CathalMullan 4y agoOnly used for metrics, apparently. [0] /** * These author ID lists are used purely for metrics collection. We track how often we are * serving Tweets from these authors and how often their tweets are being impressed by users. * This helps us validate in our A/B experimentation platform that we do not ship changes * that negatively impacts one group over others. */ [0]: https://github.com/twitter/the-algorithm/blob/7f90d0ca342b928b479b512ec51ac2c3821f5922/home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/RequestQueryFeatureHydrator.scala#L87 https://github.com/twitter/the-algorithm/blob/7f90d0ca342b92...
- minimaxir 4y agoThe original code is a part of the home-mixer service, which is the "Main service used to construct and serve the Home Timeline." I suspect the flag corresponds to weights not present in the repo.
- roughly 4y agoI expect they're tracking the red team/blue team metrics because of the political shitstorm that's been the GOP's assertions they're being silenced by The Algorithm.
- panarky 4y agoThe fallacy of false equivalence systematized in code. Now one side can spew as much disinfo and incitement to violence as it likes, and any algorithm change that prevents this shit from getting amplified will be rejected as bias. BSaaS = Both Sides as a Service
- eldritch_4ier 4y ago[flagged]
- deleted 4y ago[deleted]
- AdamH12113 4y agoI read the code snippet before I saw the link and thought you were joking, but yeah, there really is an author_is_elon flag right there in the main branch.
- bikeformind 4y agoI opened this thread just to verify this would be top comment, good job hn
- sschueller 4y agoWell someone just asked about it in the live spaces[1] Elon is hosting and he said that should not be there. An engineer said afterwards it is just for metrics but then Elon chimed in again and said "we should get rid of it, it should be gone." [1] https://twitter.com/elonmusk/status/1641880448061120513?s=20 https://twitter.com/elonmusk/status/1641880448061120513?s=20
- JustSomeNobody 4y agoDoesn't necessarily mean he didn't want it there in the first place. Why else would it be there?
- deleted 4y ago[deleted]
- JoshCole 4y agoThere are millions of lines of code existent. There are thousands added. Your prior for it being Elon's addition should be something like 10,000/1,000,000 or roughly 1/100. The prior that it wasn't Elon's change is going to be something like 99/100. When you add the additional information that Elon wants the code removed, but existing Twitter engineers think it appropriate to keep this actually increases the probability of it being added by the existing Twitter engineers and decreases the probability it was added due to Elon. Obviously, these are rough numbers, but hopefully seeing any numbers at all helps you to get an intuition for the math.
- jsnell 4y agoWhy is lines of code the appropriate input here? Here's a different computation that is at least as plausible: There are hundreds of millions of users. Let's say 300M. Only a single one is special-cased in this code: the narcisistic CEO who reportedly went ballistic when his engagement metrics went down. The prior that it's a change done in response to his demands is 299999999/300000000. (But of course it was added by existing Twitter engineers. The odds of Musk being able to actually commit code to their repository are zero. Even if he had the permissions, the man simply does not have the technical acumen to make even a trivial change.)
- minimaxir 4y agoUpdate: Elon was asked about these in a Twitter Space, he says it's not appropriate and will be removed from the codebase. Additionally, from another Twitter engineer, the Democrat/Republican flags are apparently 10 years old and not important and do not have high feature importance.
- lhnz 4y agoI think the decade old comment related to a different part of the code regarding the number of followers you have in relation to the number of accounts you follow. (Everybody on the call wants to remove this: I wonder why they haven't yet.)
- delecti 4y agoChesterton's Fence. In a sufficiently large system, you should be hesitant to remove things unless you're sure you know why it was added, and all the things that have come to depend on it since. I've definitely been hesitant to remove things I was pretty confident weren't used anymore, just because I didn't want to deal with the repercussions if I was wrong.
- nonethewiser 4y agoI didn’t know there was a name for this. Thanks for sharing. I’ve definitely been bitten by this. You always have to weigh the chance you break something against the upside. If you’re actually fixing a bug, fine. But just refactoring to make something cleaner? Or deleting because it seems like it’s not doing anything, even after doing some research? Think again.
- djtango 4y agoTrue for many things - doctors are now a lot more hesitant to remove the appendix for example. (it serves as a buffer for your gut flora iirc)
- sillysaurusx 4y ago
- deleted 4y ago[deleted]
- culi 4y agowhat is vits? private val DarkRequestAnnotation = "clnt/has_dark_request" private val Democrats = "democrats" private val Republicans = "republicans" private val Elon = "elon" private val Vits = "vits"
- nanidin 4y agoVery Important Tweeters, a play on VIP perhaps?
- stefanos82 4y agoVery Important Tweets, I presume?
- mshafrir 4y agoVery Important Tweeters
- camjohnson26 4y agoHere's a function definition: /** * This function returns the top most followed and verified userIds truncated to topK */ def vits(
- spyder 4y agoVery likely this: https://create.twitter.com/en/articles/best-practice/2021/partnering-with-vits-to-drive-conversation-on-twitter https://create.twitter.com/en/articles/best-practice/2021/pa...
- schemescape 4y agoDid they not expect people to notice suspicious code like this? Or did they leave this in just so they could hold its removal up as an example of listening to the community?
- bitshiftfaced 4y agoCould be that those in charge of preparing this open sourced repository did it begrudgingly, and so they perceived the fact that it looked bad as a positive thing. "Hey, you wanted us to release the code. Happy now?"
- dmix 4y agoWhy are you assuming this knowledge is harmful to them? What do you think it means for their business? No other social media platform will have this sort of accountability and public pressure to be better like having their recommendation algorithms public.
- philistine 4y agoCoded as if the only two political parties on the planet were the Rs and the Ds. Shameful.
- grishka 4y ago*as if the US is the only country out there
- ibraheemdev 4y ago> But we are deleting this bs. I only learned about it now! Will be gone by tomorrow. https://twitter.com/elonmusk/status/1641908130274525187?t=5tpzoUT5mUK9qptCjKbPxw&s=19 https://twitter.com/elonmusk/status/1641908130274525187?t=5t...
- commandlinefan 4y agoAt first I thought this post was a joke - and it was actually a pretty good joke. Yikes.
- RoyGBivCap 4y agoElon just said in the space "that shouldn't be there. Consider it gone"
- tomberin 4y ago[flagged]
- jdnordy 4y agogithub link now show a warning at the top of the page: > This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. Is this new? Perhaps Twitter already removed the code from their main branch? Or was this just a joke from the beginning?
- tric 4y agoYup, they removed it. https://github.com/twitter/the-algorithm/commit/ec83d01dcaebf369444d75ed04b3625a0a645eb9 https://github.com/twitter/the-algorithm/commit/ec83d01dcaeb... Here's what Musk said: > But we are deleting this bs. I only learned about it now! Will be gone by tomorrow. https://twitter.com/elonmusk/status/1641908130274525187 https://twitter.com/elonmusk/status/1641908130274525187
- maccam912 4y agoJust removed https://github.com/twitter/the-algorithm/commit/ec83d01dcaebf369444d75ed04b3625a0a645eb9 https://github.com/twitter/the-algorithm/commit/ec83d01dcaeb...
- camjohnson26 4y agoClearly embarrassing. This is the third commit in the repo.
- qwertox 4y agoThere are no guarantees that the actual code which is run will have this removed.
- MichaelMoser123 4y agonever seen a pull requests with so many comments, and where the comments to the pull request read like ... twitter :-)
- cbeach 4y agoThe “author_is_elon” flag may have been assigned to him because Elon’s Twitter account has the most followers on the platform. So, for technical / performance reasons, changes to the algos might want to be benchmarked against this account in particular, because it’s the account most likely to be at the centre of capacity- / load-related issues.
- null4bl3 4y agoSure. Every serious application has a special flag for their boss man right?
- adharmad 4y agoAlso this: https://github.com/twitter/the-algorithm/pull/160 https://github.com/twitter/the-algorithm/pull/160
- camjohnson26 4y agoThat's a pull request from a random person
- nelox 4y agoIt’s April Fools’ Day where I live.
- fnands 4y agoLegit was my first thought when I saw the headline. But apparently it's real
- iaseiadit 4y agoIf you buy a company for $44B and take it private, I for one say you should get your own flag.
- gregcoombe 4y ago"It's one flag, Michael. How much could it cost, $44B?"
- girafffe_i 4y ago[dead]
- hn2017 4y agoPer Zoe: The code that was released today doesn't show the parts that actually alter the scores of Elon and other users. The part of the code referenced below just tracks Elon stats (from what I know). Employees removed most PII before the code was released. https://twitter.com/ZoeSchiffer/status/1641902570921943044?s=20 https://twitter.com/ZoeSchiffer/status/1641902570921943044?s...
- deleted 4y ago[deleted]
- gigglesupstairs 4y agoWait, am I missing something here or author name is cleary mentioned as elon here while musk’s twitter id it @elonmusk? Why is everyone assuming this code is about elon?
- CameronNemo 4y agoAre you serious? What other Elon do you think it refers to?
- gigglesupstairs 4y agoI mean yeah it must be elon we know but what I was mostly curious about was if it’s actually meant for him only, why didn’t they use his twitter handle? And just elon? I am not a developer I must mention it looks like. And I was genuinely curious.
- slim 4y ago"author_is_elon" (notice the _ ) is a name the developer writing that code came up with for a category of twitter users. it has nothing to do with twitter handles. he could have chosen any name, but since that category includes only elon musk, he named it like that.
- gigglesupstairs 4y agoRight, this makes sense. Thanks.
- nothngssimpl 4y ago„This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.“ Isn’t this top of the page disclaimer relevant for you? It seems not to be part of the main branch.
- bdw5204 4y agoHow hard would it be to replace this entire algorithm with the following pseudocode? If !user.follows_author(author) then don't show tweet on timeline Else if tweet.timestamp is later than all other tweets show tweet first This is vastly superior to any other possible recommendation algorithm because users can choose what tweets they see/don't see by whom they follow and everybody has an equal chance to have their tweets seen by their followers. When Twitter moved away from this, it rendered my timeline useless so I started just pulling up people's profiles to read their tweets in order and eventually deleted my (pseudonymous) account that had several thousand followers. Almost nobody was seeing my tweets anyway thanks to this algorithm and deleting the account did not prevent me from browsing accounts I'm interested in. All Elon needed to do to fix Twitter was to reverse all of the bad changes they've made since 2015 or so and restore the platform to what it was in the late 00s/early 10s.
- arcatech 4y agoHow is that a recommendation algorithm? The point of the "recommendation" part is to show you things you wouldn't normally see.
- afrcnc 4y agoAs a European I find this very offensive
- steele 4y agoOccam's Razors: engineers worried about their jobs (and potentially residency) appease a volatile narcissist as fast as possible.
- pessimizer 4y agoIf anybody is actually reading this thread, it looks like twitter is using "author_is_democrat" and "author_is_republican" to evaluate "Community Notes." As with all of the media outlets that elevate these two private clubs into the arbiters of truth, votes for Community Notes have to be relatively balanced between the two parties. Bipartisanship is a trash metric for determining truth, but absolutely none of the people raging at Musk in this thread would disagree with it.
- emehrkay 4y agowhy arent these strings constants?