9 ms·

Twitter and the Internet War

54 points by dsr12 9y ago

danso 9y ago
This is the excerpt I've seen being tweeted about this piece: At the same time, her defenders say, Harvey has been forced to clean up a mess that Twitter should have fixed years ago. Twitter’s backend was initially built on Ruby on Rails, a rudimentary web-application framework that made it nearly impossible to find a technical solution to the harassment problem. If Twitter’s co-founders had known what it would become, a third former executive told me, “you never would have built it on a Fisher-Price infrastructure.” Instead of building a product that could scale alongside the platform, former employees say, Twitter papered over its problems by hiring more moderators. “Because this is just an ass-backward tech company, let’s throw non-scalable, low-tech solutions on top of this low-tech, non-scalable problem.” Setting aside the whole "Ruby/Rails is slow" discussion, I would've loved to see more elaboration by these sources about how a web framework makes moderation and policing "nearly impossible". Compared to what? The PHP scripts that became Facebook?
- maxpupmax 9y ago
  Yeah, that doesn't make any sense to me either. I think it sounds like the complaint is actually that they failed to build out features that enabled easier moderation and better verification (mentioned a paragraph later). My understanding is that Twitter moved away from Ruby/Rails back in 2010-2012 anyway, making the comparison more confusing.
  - bshepard 9y ago
    More info on this here: https://www.quora.com/Why-did-Twitter-move-away-from-Ruby-on-Rails https://www.quora.com/Why-did-Twitter-move-away-from-Ruby-on...
- taneq 9y ago
  That's bizarre. They have a social policy problem, not a technical problem. Short of building a general human-level AI, it's not clear how their tech stack has any bearing on the issue.
- lkrubner 9y ago
  I followed this story back in 2006 to 2008. I’m sorry I don’t have time to write a real comment with the crucial details. The basic plot is that Twitter initially set out to be a CMS sort of like WordPress, but then they realized they had actually built a messaging platform, which needed a different architecture. You can catch a glimpse of the issues in “Amateur hour is over at Twitter” from 2008. https://www.google.com/amp/s/techcrunch.com/2008/04/23/amateur-hour-over-at-twitter/amp/ https://www.google.com/amp/s/techcrunch.com/2008/04/23/amate... Of course, it is a stretch to suggest that in ten years they couldn’t reinvent the architecture. It took them many years to get beyond the “fail whale”. And it is surprising who they thought of as experts at scaling: “And this week they hired two scaling experts – John Kalucki and Steve Jenson (“known for his work scaling Blogger and Blogspot”).” No offense to those guys, but Blogger and Blogspot were also notorious failures at scaling. Bloggers problems were not fixed till a year or two after they were bought by Google.
  - liquidgecka 9y ago
    Hi! I am one of the dudes that actually helped kill the fail whale so I feel I can speak to this topic. =) Steve and John were both good developers that wrote some seriously critical components of the system and were a big part of getting twitter more stable. But reading Techcrunch to figure out what was happening inside of Twitter is asking for serious misinformation. At that time it was very common for people with no idea what they were talking about to write articles about what was going on inside of Twitter, yet do so with authority that they didn't really have. By the time a team of reliability focused people started to get hired Twitter was finally getting smart about not letting employees/outsides control the messaging sent to Techcrunch and the like. Twitter has completely reinvented the architecture since I left so that is not impossible by any means. Lets be clear about the rate that Twitter was growing though. In any 9 month window while I worked there they never failed to grow an order of magnitude in request volume. When I started we would have spikes of a few thousand requests per second. We had a major site outage at 451 tweets per second and we simply couldn't keep up. 9 months later we were spiking 50k requests per second, and we never dropped below 451 tps (barring outage). By the time I left both of those numbers simply couldn't be seen in the graphs anymore. Every time we got technology in place to deal with the scaling issues we were already behind the curve because of how fast things were growing. It felt like being in a car that has amazing 0-60 time, but it just kept accelerating well beyond 60mph.
- booleandilemma 9y ago
  I’m a .NET developer, not a ruby apologist, but fisher-price infrastructure? Gimme a break. Someone go tell GitHub, AirBnb, Hulu, and Bloomberg.
  - akhilcacharya 9y ago
    Bloomberg?
    - deleted 9y ago
      [deleted]
    - adventured 9y ago
      They work with ruby on rails, eg - https://job-openings.monster.com/Full-Stack-Senior-Software-Engineer-Vault-New-York-NY-US-Bloomberg/31/21445955-46af-49f8-a793-515208dd9e5b https://job-openings.monster.com/Full-Stack-Senior-Software-... https://job-openings.monster.com/Senior-Web-Engineer-BLAW-New-York-NY-US-Bloomberg/31/45377f1b-6c57-40bc-9c32-53aad319332a https://job-openings.monster.com/Senior-Web-Engineer-BLAW-Ne... Bloomberg is fairly well known for having top notch technical capabilities (in their general space). What they build tends to work, work well, and have high reliability.
      - akhilcacharya 9y ago
        I've worked at Bloomberg, I did not know they used RoR in large external facing apps. Only heard about it for smaller internal sites. My team worked on things very far from BLaw and BVault implementation details though.
- aptwebapps 9y ago
  The most charitable interpretation I can come up with is that this is a distortion of someone claiming that they couldn't build better anti-harassment tools because of excessive technical debt and maybe a lack technical acumen or focus.
  - intended 9y ago
    Thats not charity - thats donating credibility. Whoever was quoted has precisely Zero current understanding of moderation and the challenges it faces. And worse - they haven't reflected on what "technical acumen" resulting on anti-harassment tools means. Effective anti harassment tools can suss out intent, and by nature are also censorship tools. Effective tools which can identify intent are so far past invasion of privacy, that they may as well be categorized as assaults on specific ideas. If twitter does come up with this magical tool, then it's a sure bet that other nation states will also have this magic algorithm. This rule applies to all lesser tools for "anti-harassment". The solution sadly IS hiring more moderators, and journalists vetting. Which I suspect is impossible at the scale and message type twitter operates on.
- dawhizkid 9y ago
  I assume this "executive" was a non-technical mid-level manager. I don't think either the author of this story or her source has any idea what they're talking about w.r.t web frameworks.
- mattbierner 9y ago
  The only way I can understand the argument is that their early UX / technical design influenced the community and culture that we now see. From the start, it seems like Twitter was designed for hottakes and emotional broadcasting, and countless little design decisions emphasized this (everything from the original 140 character limit to the UX around retweeting and followers.) The modern platform’s culture grew out of these early decisions and now embodies them in many ways, so much so that it’s difficult now to imagine that twitter could ever have been anything different. But what if following had only been bi-directional for example? What if the platform never displayed retweet or star count? What if muting and blocking had been built in from the start? What if you could only @mention people who follow you? It’s very possible that twitter would never have become Twitter if these things had been different, but that’s also my point. So while it may not be a technical root cause in the classic coding sense, I can buy the argument that many of twitter’s early technical and design decisions (as well as their early platform policies and who the first adopters were) led to their present difficulties
- liquidgecka 9y ago
  Hi, early Twitter employee here. The reliability team called the infrastructure Fisher-Price internally so this wasn't just one random executive coming up with a term on his/her own. The problem wasn't ruby. The problem was the way that Twitter used Ruby. We had one big mono repo with every single function and every form of business logic baked into a single place. That logic relied on monkey patching and all sorts of crazy horrible glue to keep it working together. Every time we had to scale up we would glue infra in place to keep things working while we came up with a real solution (which never really materialized). In my time there we had memcache instances which held timelines. Populating them took hours/days and while they were unpopulated the site was offline. Rebooting/restarting the caches was simply not an option. We had a data sharding strategy that was temporal. We would spin up a new database cluster every few weeks to handle all of the incoming tweets and failing to spin up a new cluster in time meant we would have a global site outage. Don't even get me started on the "load bearing mac mini". In reality the only problem rails really contributed on its own was that it could only process a single request per process at a time. Each machine would spin up 16 or 32 processes to handle requests in parallel but each process needed its own connection to the database, to memcache, etc. At one point we had something like 100k processes all trying to talk to a single mysql master. Much of this could have been mitigated by better design of course, but rails encourages models that don't scale up to crazy dimensions. In reality moderation was virtually impossible because we were in a 24/7 fight with ourselves about how to keep the system alive for the next couple of days. Constant infighting, managerial changes (I had 9 different managers in 3 years), focus changes (we didn't finish the last major site redesign before starting the next one) and a general unwillingness to pause features long enough to stabilize the system meant we were always on the losing end of a infra battle.
  - stryk 9y ago
    I don't know nearly enough about large scale projects or the startup world maybe this is a silly question but when it became apparent that it was blowing up far beyond what you were expecting, was it not an option to have another team (hire or split existing) in parallel, to start writing a re-implmentation in another language or system that would work better for your new quickly expanding needs? Was that just not feasible?, no doubt it was a crazy hectic time for you folks back then, I can't even really imagine what that was like.
icelancer 9y ago
Lost? How is Twitter losing the Internet War? Their platform is more influential than ever and helped to seriously impact a presidential election (the results of which not too many people in media are happy with, fair enough). Yes, the platform is a failure in so many ways - failure to protect the identity and safety of the people on it. (Though as someone who has served a subpoena to Twitter, let me tell you something - the legal team isn't exactly handing over data easily. It was a huge pain in the ass with tons of individual privacy concerns the whole way.) And the platform is failing revenue-wise, yes. But if it's about the Internet War, so to speak? Twitter is at the top of it all. No one has to like the externalities - I sure don't - but their influence is undeniable.
- oceanghost 9y ago
  You're going to think I'm trolling, but I'm not-- What value does Twitter provide to ordinary people? I have friends who are celebrities in their industries or trying to market themselves-- but other than that it seems like the most hostile place imaginable. Why would I want to participate in that?
  - craftyguy 9y ago
    In the US, most ordinary people pay attention to twitter's top customer: donald trump
    - olivermarks 9y ago
      I suspect the vast majority of people read what Trump said on Twitter on their favorite news site, rather than using Twitter itself.
  - marssaxman 9y ago
    I wonder the same. It has always seemed to me that Twitter gets a disproportionately large share of media attention simply because Twitter happens to be personally useful to the people doing the reporting. For those of us who are less interested in drama-hunting and reputation-mongering, its utility is unclear.
ohiovr 9y ago
I lurved the hate they had on ruby. I've never used ruby but strikes me as childish to blame all your ills on software that they developed to run their whole company on. I mean they could have used PHP, a real professional language! Just like what Facebook used :D
iisbum 9y ago
What a cop out. If they really wanted to solve the problem all it takes it consuming their own firehose API and writing back to delete flagged content. You really don't need to insert the moderation into the "backend", you just have to want to solve the problem, instead of accepting the problem because it fuels your growth.
- itronitron 9y ago
  This article is an interesting variant on the 'we're trying but it's a really hard problem' puff PR piece.
anonytrary 9y ago
This smells like boring FUD. Twitter is an incredible tool for real-time information, and everyone already uses it. Twitters content problems are great problems to have, similar to the problems Facebook faces. I think Twitter has a very bright future. Using Facebook, I have always felt that the website tried to "force" me into where it thought I belonged in the social graph. Twitter has a much lower barrier to entry. I prefer the lurker-first philosophy. Facebook tries way too hard to engineer interactions (so does Twitter, but I think it's not as bad).
mark242 9y ago
Part of the problem is that Twitter encouraged automated signups in the early days, blazing the trail for gigantic bot-farms that we all talk about today. They pushed popular rss-to-tweet gateways, wordpress plugins for auto-tweeting blog posts, etc. There should have been gigantic red flags waving when the hypergrowth of Twitter really started, because you knew these weren't all people signing up for one account. I'm not sure if Twitter can ever put pandora back in the box, but at the very least, requiring a mobile number is a start. It's hard(-ish) to generate thousands of bots if you have to have a unique phone number with multifactor for each signup.
flashman 9y ago
> There are two main components to Harvey’s job, this person told me: to formulate a clear set of rules for what constitutes abusive speech, and to be consistent in enforcing them. The odds are against Harvey being the first person in human history to solve this problem without false positives and negatives.
- otterley 9y ago
  The big question in my eyes is not how a human can reduce the error rate, but rather, which category of errors one should bias towards. I know where I stand on this one: be biased against the mean-spirited.
  - proofbygazing 9y ago
    Wow you're really great
- gaius 9y ago
  without false positives and negatives. To be fair to Twitter they are consistent here. What looks like false results to an outsider is clearly policy.
pg_bot 9y ago
Well at least we now know the solution to all of society's problems is making Ruby on Rails fast. Perhaps one day we can let out a collective sigh of relief once we remove the global interpreter lock.
firasd 9y ago
My general feeling when it comes to questions about harassment on Twitter is that people look at it too much as a policy issue ("you're not banning and deleting enough accounts according to my ideological stance"), which has its place, but there's a lot of product aspects that make it such a shouty place. A good example is the way the 'quote tweet' feature is often used to start a pile-on. It's not a bad feature in itself, but there is a significant portion of usage that lends itself to starting food fights. Could things be improved by, for example, a setting that limits quote-tweets to people who follow you? There's a lot of dynamics like that which can be explored.
- kenning 9y ago
  That's a cool idea, if you really want to quote-tweet someone you can follow them, do it and then unfollow them, so you're not limiting behavior really, but it would still reduce the number of times it happened. However I think twitter used to have a lot of weird secret features like how DM's could accidentally become public if you formatted them wrong, and another secret rule to twitter could be a bad idea.
- philwelch 9y ago
  Jeff Vogel: > Twitter was designed, from Day 1, to enable any random person to send messages directly to any public figure. In other words, from Day 1, it was designed to be an abuse and harassment engine. It's not a bug. It's a feature. All that abuse and controversy is how it gets clicks and money. http://jeff-vogel.blogspot.com/2016/04/how-i-deal-with-harassment-abuse-and.html http://jeff-vogel.blogspot.com/2016/04/how-i-deal-with-haras... Twitter's feature set is perfectly optimized for harassment and abuse. Changing it to prevent harassment and abuse would kill the product.
continuations 9y ago
> Twitter’s backend was initially built on Ruby on Rails, a rudimentary web-application framework that made it nearly impossible to find a technical solution to the harassment problem. This is probably a new low in journalism.
- kenning 9y ago
  Can you elaborate? I don't know much about ror and would like to know more.
  - ericd 9y ago
    The web framework (Ruby on Rails) has nothing to do with the ability to build anti-harassment tech.
    - deleted 9y ago
      [deleted]
    - liquidgecka 9y ago
      Its not described well at all in the article, but the instance on sticking with a RoR framework well beyond what it could rightfully scale up to is what kept us from building a great, great many features, least of which was harassment tooling. It wasn't RoR's fault beyond being the framework selected early on, and dogmatically stuck too long after it stopped helping us move forward quickly.
- elcapitan 9y ago
  Apparently the rudimentary framework that Vanity Fair uses made it nearly impossible for the author to find a technical solution to write a fact based article.
- TAForObvReasons 9y ago
  Put the line in the context, and include the actual quotes from the former employees and executives: > If Twitter’s co-founders had known what it would become, a third former executive told me, “you never would have built it on a Fisher-Price infrastructure.” Instead of building a product that could scale alongside the platform, former employees say, Twitter papered over its problems by hiring more moderators. “Because this is just an ass-backward tech company, let’s throw non-scalable, low-tech solutions on top of this low-tech, non-scalable problem.” It sounds like the original people who spoke to the journalist left the impression that there were deficiencies in the base technology which may have led to the current circumstances.
JumpCrisscross 9y ago
“I think what religion and politics have in common is that they become part of people's identity, and people can never have a fruitful argument about something that's part of their identity. By definition they're partisan. Which topics engage people's identity depends on the people, not the topic. For example, a discussion about a battle that included citizens of one or more of the countries involved would probably degenerate into a political argument. But a discussion today about a battle that took place in the Bronze Age probably wouldn't. No one would know what side to be on. So it's not politics that's the source of the trouble, but identity. When people say a discussion has degenerated into a religious war, what they really mean is that it has started to be driven mostly by people's identities. ... More generally, you can have a fruitful discussion about a topic only if it doesn't engage the identities of any of the participants. What makes politics and religion such minefields is that they engage so many people's identities. But you could in principle have a useful conversation about them with some people. And there are other topics that might seem harmless, like the relative merits of Ford and Chevy pickup trucks, that you couldn't safely talk about with others.“ http://paulgraham.com/identity.html?viewfullsite=1 http://paulgraham.com/identity.html?viewfullsite=1
petraeus 9y ago
Twitter chased growth at the expense of quality, quality moderation, and quality infrastructure. Thats all it boils down to.
Aurelia_Cotta 9y ago
I am not a coder by any means, but I am a heavy social media user and know a lot about politics, sociology, psych, organizing, health, comms---and I admit, until I read this thread, I liked this article because it gave me a better reason for all the screw ups, beyond, "The Executives are dithering and have no business skills and social skills." Or the theory "The Executives do nothing because they don't care if women and vulnerable people die." Which is far more disturbing....it can't be true, even if it feels true. I would give anything if they'd listen to users who have been around awhile. So few people worked there and also used it at the same time. (And users longingly miss the Fail Whale logo...) As a user (250,000+ tweets) under a pseudonym, twitter has incredible uses, things Facebook and other platforms didn't do, because they were so closed and hard to search. They created echo Chambers, because you could only see people you already knew or were slightly connected too. Or worse, your mom or mother-in-law could find you. Twitter was so open--if I want to discuss philosophy or Japanese food or an MRI result, 24/7 I just search and people who love that are there. And pseudonyms had to exist, because thousands of people in real life have the same name. (just like the rest of the Internet back then and yes, now too) They thought people would just post status updates like, hey eating lunch, but we did way way more. They made it 140 chars with 20 chars reserved for names because many people had expensive tiny data plans, but could update by text message; allowing a much wider demographic to use it, and across multiple countries, even low tech ones with limited access. The most critical piece tho is that users invented everything good about twitter (Sorry creators--but we did). TW allowed some html symbols, so a user put an @ in front of a user name to reply, and it worked! Users also invented hashtags, the first one was for organizing BarCamp--which was kind of a conference not about alcohol and the original manual RT, and MT for modified tweet, and commenting on top of someone's tweet, or at the end of it and we figured out how to shrink long urls to post links. bit.ly had no purpose til then, and many users became Developers and got full access to the API and we crowdsourced hundreds of changes and ideas, from pictures to videos to emoji to gifs to analytics, to accessible apps for people with vision, hearing, speech issues. All while making lots and lots of jokes. They didn't start out with a heavy respect for pseudonyms, and privacy, and free speech--but TW learned it fast after the Green Iran Revolution, and after many patient users wanted to keep privacy because they had rare diseases, and mental health issues, and parents of kids with autism and speech issues took to it and found each other and felt less alone. It was less complicated than blogging and writing long stories on laptops after events happened. I could do everything from live tweeting a doctor's appt to an ER visit and get reactions from friends who could tell me what to ask. We still tweet everything from recipes to exact instructions on how to ride a bike, how to organize groups like #occupy to crowd sourcing Flu symptoms and rashes, to who is watching what TV show and how cool it is to watch live sports "together" even for people who can't leave the house that night and meet. Weekly hour long Chats take place under special hashtags like #hcsm for Health Care Social Media or #meded for medical education. (doctors and science fans and academics found twitter and they debate articles, techniques, crowdsource diagnoses) Police and emergency responders and good Samaritans have used it to (swear to god, it's true) befriend people who sound troubled or suicidal and validate their pain and sadness and find them help. It is to this day, the only platform I know that allows people from many different areas to find like-minded people and to bring together people across multiple subject areas. Nothing else does it quite the same way. Especially with the ridiculous algorithms other platforms use.

54 points by dsr12 9y ago

This is the excerpt I've seen being tweeted about this piece: At the same time, her defenders say, Harvey has been forced to clean up a mess that Twitter should have fixed years ago. Twitter’s backend was initially built on Ruby on Rails, a rudimentary web-application framework that made it nearly impossible to find a technical solution to the harassment problem. If Twitter’s co-founders had known what it would become, a third former executive told me, “you never would have built it on a Fisher-Price infrastructure.” Instead of building a product that could scale alongside the platform, former employees say, Twitter papered over its problems by hiring more moderators. “Because this is just an ass-backward tech company, let’s throw non-scalable, low-tech solutions on top of this low-tech, non-scalable problem.” Setting aside the whole "Ruby/Rails is slow" discussion, I would've loved to see more elaboration by these sources about how a web framework makes moderation and policing "nearly impossible". Compared to what? The PHP scripts that became Facebook?

maxpupmax 9y ago
Yeah, that doesn't make any sense to me either. I think it sounds like the complaint is actually that they failed to build out features that enabled easier moderation and better verification (mentioned a paragraph later). My understanding is that Twitter moved away from Ruby/Rails back in 2010-2012 anyway, making the comparison more confusing.
- bshepard 9y ago
  More info on this here: https://www.quora.com/Why-did-Twitter-move-away-from-Ruby-on-Rails https://www.quora.com/Why-did-Twitter-move-away-from-Ruby-on...
taneq 9y ago
That's bizarre. They have a social policy problem, not a technical problem. Short of building a general human-level AI, it's not clear how their tech stack has any bearing on the issue.
lkrubner 9y ago
I followed this story back in 2006 to 2008. I’m sorry I don’t have time to write a real comment with the crucial details. The basic plot is that Twitter initially set out to be a CMS sort of like WordPress, but then they realized they had actually built a messaging platform, which needed a different architecture. You can catch a glimpse of the issues in “Amateur hour is over at Twitter” from 2008. https://www.google.com/amp/s/techcrunch.com/2008/04/23/amateur-hour-over-at-twitter/amp/ https://www.google.com/amp/s/techcrunch.com/2008/04/23/amate... Of course, it is a stretch to suggest that in ten years they couldn’t reinvent the architecture. It took them many years to get beyond the “fail whale”. And it is surprising who they thought of as experts at scaling: “And this week they hired two scaling experts – John Kalucki and Steve Jenson (“known for his work scaling Blogger and Blogspot”).” No offense to those guys, but Blogger and Blogspot were also notorious failures at scaling. Bloggers problems were not fixed till a year or two after they were bought by Google.
- liquidgecka 9y ago
  Hi! I am one of the dudes that actually helped kill the fail whale so I feel I can speak to this topic. =) Steve and John were both good developers that wrote some seriously critical components of the system and were a big part of getting twitter more stable. But reading Techcrunch to figure out what was happening inside of Twitter is asking for serious misinformation. At that time it was very common for people with no idea what they were talking about to write articles about what was going on inside of Twitter, yet do so with authority that they didn't really have. By the time a team of reliability focused people started to get hired Twitter was finally getting smart about not letting employees/outsides control the messaging sent to Techcrunch and the like. Twitter has completely reinvented the architecture since I left so that is not impossible by any means. Lets be clear about the rate that Twitter was growing though. In any 9 month window while I worked there they never failed to grow an order of magnitude in request volume. When I started we would have spikes of a few thousand requests per second. We had a major site outage at 451 tweets per second and we simply couldn't keep up. 9 months later we were spiking 50k requests per second, and we never dropped below 451 tps (barring outage). By the time I left both of those numbers simply couldn't be seen in the graphs anymore. Every time we got technology in place to deal with the scaling issues we were already behind the curve because of how fast things were growing. It felt like being in a car that has amazing 0-60 time, but it just kept accelerating well beyond 60mph.
booleandilemma 9y ago
I’m a .NET developer, not a ruby apologist, but fisher-price infrastructure? Gimme a break. Someone go tell GitHub, AirBnb, Hulu, and Bloomberg.
- akhilcacharya 9y ago
  Bloomberg?
  - deleted 9y ago
    [deleted]
  - adventured 9y ago
    They work with ruby on rails, eg - https://job-openings.monster.com/Full-Stack-Senior-Software-Engineer-Vault-New-York-NY-US-Bloomberg/31/21445955-46af-49f8-a793-515208dd9e5b https://job-openings.monster.com/Full-Stack-Senior-Software-... https://job-openings.monster.com/Senior-Web-Engineer-BLAW-New-York-NY-US-Bloomberg/31/45377f1b-6c57-40bc-9c32-53aad319332a https://job-openings.monster.com/Senior-Web-Engineer-BLAW-Ne... Bloomberg is fairly well known for having top notch technical capabilities (in their general space). What they build tends to work, work well, and have high reliability.
    - akhilcacharya 9y ago
      I've worked at Bloomberg, I did not know they used RoR in large external facing apps. Only heard about it for smaller internal sites. My team worked on things very far from BLaw and BVault implementation details though.
aptwebapps 9y ago
The most charitable interpretation I can come up with is that this is a distortion of someone claiming that they couldn't build better anti-harassment tools because of excessive technical debt and maybe a lack technical acumen or focus.
- intended 9y ago
  Thats not charity - thats donating credibility. Whoever was quoted has precisely Zero current understanding of moderation and the challenges it faces. And worse - they haven't reflected on what "technical acumen" resulting on anti-harassment tools means. Effective anti harassment tools can suss out intent, and by nature are also censorship tools. Effective tools which can identify intent are so far past invasion of privacy, that they may as well be categorized as assaults on specific ideas. If twitter does come up with this magical tool, then it's a sure bet that other nation states will also have this magic algorithm. This rule applies to all lesser tools for "anti-harassment". The solution sadly IS hiring more moderators, and journalists vetting. Which I suspect is impossible at the scale and message type twitter operates on.
dawhizkid 9y ago
I assume this "executive" was a non-technical mid-level manager. I don't think either the author of this story or her source has any idea what they're talking about w.r.t web frameworks.
mattbierner 9y ago
The only way I can understand the argument is that their early UX / technical design influenced the community and culture that we now see. From the start, it seems like Twitter was designed for hottakes and emotional broadcasting, and countless little design decisions emphasized this (everything from the original 140 character limit to the UX around retweeting and followers.) The modern platform’s culture grew out of these early decisions and now embodies them in many ways, so much so that it’s difficult now to imagine that twitter could ever have been anything different. But what if following had only been bi-directional for example? What if the platform never displayed retweet or star count? What if muting and blocking had been built in from the start? What if you could only @mention people who follow you? It’s very possible that twitter would never have become Twitter if these things had been different, but that’s also my point. So while it may not be a technical root cause in the classic coding sense, I can buy the argument that many of twitter’s early technical and design decisions (as well as their early platform policies and who the first adopters were) led to their present difficulties
liquidgecka 9y ago
Hi, early Twitter employee here. The reliability team called the infrastructure Fisher-Price internally so this wasn't just one random executive coming up with a term on his/her own. The problem wasn't ruby. The problem was the way that Twitter used Ruby. We had one big mono repo with every single function and every form of business logic baked into a single place. That logic relied on monkey patching and all sorts of crazy horrible glue to keep it working together. Every time we had to scale up we would glue infra in place to keep things working while we came up with a real solution (which never really materialized). In my time there we had memcache instances which held timelines. Populating them took hours/days and while they were unpopulated the site was offline. Rebooting/restarting the caches was simply not an option. We had a data sharding strategy that was temporal. We would spin up a new database cluster every few weeks to handle all of the incoming tweets and failing to spin up a new cluster in time meant we would have a global site outage. Don't even get me started on the "load bearing mac mini". In reality the only problem rails really contributed on its own was that it could only process a single request per process at a time. Each machine would spin up 16 or 32 processes to handle requests in parallel but each process needed its own connection to the database, to memcache, etc. At one point we had something like 100k processes all trying to talk to a single mysql master. Much of this could have been mitigated by better design of course, but rails encourages models that don't scale up to crazy dimensions. In reality moderation was virtually impossible because we were in a 24/7 fight with ourselves about how to keep the system alive for the next couple of days. Constant infighting, managerial changes (I had 9 different managers in 3 years), focus changes (we didn't finish the last major site redesign before starting the next one) and a general unwillingness to pause features long enough to stabilize the system meant we were always on the losing end of a infra battle.
- stryk 9y ago
  I don't know nearly enough about large scale projects or the startup world maybe this is a silly question but when it became apparent that it was blowing up far beyond what you were expecting, was it not an option to have another team (hire or split existing) in parallel, to start writing a re-implmentation in another language or system that would work better for your new quickly expanding needs? Was that just not feasible?, no doubt it was a crazy hectic time for you folks back then, I can't even really imagine what that was like.

icelancer 9y ago

Lost? How is Twitter losing the Internet War? Their platform is more influential than ever and helped to seriously impact a presidential election (the results of which not too many people in media are happy with, fair enough). Yes, the platform is a failure in so many ways - failure to protect the identity and safety of the people on it. (Though as someone who has served a subpoena to Twitter, let me tell you something - the legal team isn't exactly handing over data easily. It was a huge pain in the ass with tons of individual privacy concerns the whole way.) And the platform is failing revenue-wise, yes. But if it's about the Internet War, so to speak? Twitter is at the top of it all. No one has to like the externalities - I sure don't - but their influence is undeniable.

oceanghost 9y ago
You're going to think I'm trolling, but I'm not-- What value does Twitter provide to ordinary people? I have friends who are celebrities in their industries or trying to market themselves-- but other than that it seems like the most hostile place imaginable. Why would I want to participate in that?
- craftyguy 9y ago
  In the US, most ordinary people pay attention to twitter's top customer: donald trump
  - olivermarks 9y ago
    I suspect the vast majority of people read what Trump said on Twitter on their favorite news site, rather than using Twitter itself.
- marssaxman 9y ago
  I wonder the same. It has always seemed to me that Twitter gets a disproportionately large share of media attention simply because Twitter happens to be personally useful to the people doing the reporting. For those of us who are less interested in drama-hunting and reputation-mongering, its utility is unclear.

ohiovr 9y ago

I lurved the hate they had on ruby. I've never used ruby but strikes me as childish to blame all your ills on software that they developed to run their whole company on. I mean they could have used PHP, a real professional language! Just like what Facebook used :D

iisbum 9y ago

What a cop out. If they really wanted to solve the problem all it takes it consuming their own firehose API and writing back to delete flagged content. You really don't need to insert the moderation into the "backend", you just have to want to solve the problem, instead of accepting the problem because it fuels your growth.

itronitron 9y ago
This article is an interesting variant on the 'we're trying but it's a really hard problem' puff PR piece.

anonytrary 9y ago

This smells like boring FUD. Twitter is an incredible tool for real-time information, and everyone already uses it. Twitters content problems are great problems to have, similar to the problems Facebook faces. I think Twitter has a very bright future. Using Facebook, I have always felt that the website tried to "force" me into where it thought I belonged in the social graph. Twitter has a much lower barrier to entry. I prefer the lurker-first philosophy. Facebook tries way too hard to engineer interactions (so does Twitter, but I think it's not as bad).

mark242 9y ago

Part of the problem is that Twitter encouraged automated signups in the early days, blazing the trail for gigantic bot-farms that we all talk about today. They pushed popular rss-to-tweet gateways, wordpress plugins for auto-tweeting blog posts, etc. There should have been gigantic red flags waving when the hypergrowth of Twitter really started, because you knew these weren't all people signing up for one account. I'm not sure if Twitter can ever put pandora back in the box, but at the very least, requiring a mobile number is a start. It's hard(-ish) to generate thousands of bots if you have to have a unique phone number with multifactor for each signup.

flashman 9y ago

> There are two main components to Harvey’s job, this person told me: to formulate a clear set of rules for what constitutes abusive speech, and to be consistent in enforcing them. The odds are against Harvey being the first person in human history to solve this problem without false positives and negatives.

otterley 9y ago
The big question in my eyes is not how a human can reduce the error rate, but rather, which category of errors one should bias towards. I know where I stand on this one: be biased against the mean-spirited.
- proofbygazing 9y ago
  Wow you're really great
gaius 9y ago
without false positives and negatives. To be fair to Twitter they are consistent here. What looks like false results to an outsider is clearly policy.

pg_bot 9y ago

Well at least we now know the solution to all of society's problems is making Ruby on Rails fast. Perhaps one day we can let out a collective sigh of relief once we remove the global interpreter lock.

firasd 9y ago

My general feeling when it comes to questions about harassment on Twitter is that people look at it too much as a policy issue ("you're not banning and deleting enough accounts according to my ideological stance"), which has its place, but there's a lot of product aspects that make it such a shouty place. A good example is the way the 'quote tweet' feature is often used to start a pile-on. It's not a bad feature in itself, but there is a significant portion of usage that lends itself to starting food fights. Could things be improved by, for example, a setting that limits quote-tweets to people who follow you? There's a lot of dynamics like that which can be explored.

kenning 9y ago
That's a cool idea, if you really want to quote-tweet someone you can follow them, do it and then unfollow them, so you're not limiting behavior really, but it would still reduce the number of times it happened. However I think twitter used to have a lot of weird secret features like how DM's could accidentally become public if you formatted them wrong, and another secret rule to twitter could be a bad idea.
philwelch 9y ago
Jeff Vogel: > Twitter was designed, from Day 1, to enable any random person to send messages directly to any public figure. In other words, from Day 1, it was designed to be an abuse and harassment engine. It's not a bug. It's a feature. All that abuse and controversy is how it gets clicks and money. http://jeff-vogel.blogspot.com/2016/04/how-i-deal-with-harassment-abuse-and.html http://jeff-vogel.blogspot.com/2016/04/how-i-deal-with-haras... Twitter's feature set is perfectly optimized for harassment and abuse. Changing it to prevent harassment and abuse would kill the product.

continuations 9y ago

> Twitter’s backend was initially built on Ruby on Rails, a rudimentary web-application framework that made it nearly impossible to find a technical solution to the harassment problem. This is probably a new low in journalism.

kenning 9y ago
Can you elaborate? I don't know much about ror and would like to know more.
- ericd 9y ago
  The web framework (Ruby on Rails) has nothing to do with the ability to build anti-harassment tech.
  - deleted 9y ago
    [deleted]
  - liquidgecka 9y ago
    Its not described well at all in the article, but the instance on sticking with a RoR framework well beyond what it could rightfully scale up to is what kept us from building a great, great many features, least of which was harassment tooling. It wasn't RoR's fault beyond being the framework selected early on, and dogmatically stuck too long after it stopped helping us move forward quickly.
elcapitan 9y ago
Apparently the rudimentary framework that Vanity Fair uses made it nearly impossible for the author to find a technical solution to write a fact based article.
TAForObvReasons 9y ago
Put the line in the context, and include the actual quotes from the former employees and executives: > If Twitter’s co-founders had known what it would become, a third former executive told me, “you never would have built it on a Fisher-Price infrastructure.” Instead of building a product that could scale alongside the platform, former employees say, Twitter papered over its problems by hiring more moderators. “Because this is just an ass-backward tech company, let’s throw non-scalable, low-tech solutions on top of this low-tech, non-scalable problem.” It sounds like the original people who spoke to the journalist left the impression that there were deficiencies in the base technology which may have led to the current circumstances.

JumpCrisscross 9y ago

“I think what religion and politics have in common is that they become part of people's identity, and people can never have a fruitful argument about something that's part of their identity. By definition they're partisan. Which topics engage people's identity depends on the people, not the topic. For example, a discussion about a battle that included citizens of one or more of the countries involved would probably degenerate into a political argument. But a discussion today about a battle that took place in the Bronze Age probably wouldn't. No one would know what side to be on. So it's not politics that's the source of the trouble, but identity. When people say a discussion has degenerated into a religious war, what they really mean is that it has started to be driven mostly by people's identities. ... More generally, you can have a fruitful discussion about a topic only if it doesn't engage the identities of any of the participants. What makes politics and religion such minefields is that they engage so many people's identities. But you could in principle have a useful conversation about them with some people. And there are other topics that might seem harmless, like the relative merits of Ford and Chevy pickup trucks, that you couldn't safely talk about with others.“ http://paulgraham.com/identity.html?viewfullsite=1 http://paulgraham.com/identity.html?viewfullsite=1

petraeus 9y ago

Twitter chased growth at the expense of quality, quality moderation, and quality infrastructure. Thats all it boils down to.

Aurelia_Cotta 9y ago

I am not a coder by any means, but I am a heavy social media user and know a lot about politics, sociology, psych, organizing, health, comms---and I admit, until I read this thread, I liked this article because it gave me a better reason for all the screw ups, beyond, "The Executives are dithering and have no business skills and social skills." Or the theory "The Executives do nothing because they don't care if women and vulnerable people die." Which is far more disturbing....it can't be true, even if it feels true. I would give anything if they'd listen to users who have been around awhile. So few people worked there and also used it at the same time. (And users longingly miss the Fail Whale logo...) As a user (250,000+ tweets) under a pseudonym, twitter has incredible uses, things Facebook and other platforms didn't do, because they were so closed and hard to search. They created echo Chambers, because you could only see people you already knew or were slightly connected too. Or worse, your mom or mother-in-law could find you. Twitter was so open--if I want to discuss philosophy or Japanese food or an MRI result, 24/7 I just search and people who love that are there. And pseudonyms had to exist, because thousands of people in real life have the same name. (just like the rest of the Internet back then and yes, now too) They thought people would just post status updates like, hey eating lunch, but we did way way more. They made it 140 chars with 20 chars reserved for names because many people had expensive tiny data plans, but could update by text message; allowing a much wider demographic to use it, and across multiple countries, even low tech ones with limited access. The most critical piece tho is that users invented everything good about twitter (Sorry creators--but we did). TW allowed some html symbols, so a user put an @ in front of a user name to reply, and it worked! Users also invented hashtags, the first one was for organizing BarCamp--which was kind of a conference not about alcohol and the original manual RT, and MT for modified tweet, and commenting on top of someone's tweet, or at the end of it and we figured out how to shrink long urls to post links. bit.ly had no purpose til then, and many users became Developers and got full access to the API and we crowdsourced hundreds of changes and ideas, from pictures to videos to emoji to gifs to analytics, to accessible apps for people with vision, hearing, speech issues. All while making lots and lots of jokes. They didn't start out with a heavy respect for pseudonyms, and privacy, and free speech--but TW learned it fast after the Green Iran Revolution, and after many patient users wanted to keep privacy because they had rare diseases, and mental health issues, and parents of kids with autism and speech issues took to it and found each other and felt less alone. It was less complicated than blogging and writing long stories on laptops after events happened. I could do everything from live tweeting a doctor's appt to an ER visit and get reactions from friends who could tell me what to ask. We still tweet everything from recipes to exact instructions on how to ride a bike, how to organize groups like #occupy to crowd sourcing Flu symptoms and rashes, to who is watching what TV show and how cool it is to watch live sports "together" even for people who can't leave the house that night and meet. Weekly hour long Chats take place under special hashtags like #hcsm for Health Care Social Media or #meded for medical education. (doctors and science fans and academics found twitter and they debate articles, techniques, crowdsource diagnoses) Police and emergency responders and good Samaritans have used it to (swear to god, it's true) befriend people who sound troubled or suicidal and validate their pain and sadness and find them help. It is to this day, the only platform I know that allows people from many different areas to find like-minded people and to bring together people across multiple subject areas. Nothing else does it quite the same way. Especially with the ridiculous algorithms other platforms use.