5 ms·
This is the excerpt I've seen being tweeted about this piece: At the same time, her defenders say, Harvey has been forced to clean up a mess that Twitter shoul
by danso 9y ago
This is the excerpt I've seen being tweeted about this piece:
At the same time, her defenders say, Harvey has been forced to clean up a mess that Twitter should have fixed years ago. Twitter’s backend was initially built on Ruby on Rails, a rudimentary web-application framework that made it nearly impossible to find a technical solution to the harassment problem. If Twitter’s co-founders had known what it would become, a third former executive told me, “you never would have built it on a Fisher-Price infrastructure.” Instead of building a product that could scale alongside the platform, former employees say, Twitter papered over its problems by hiring more moderators. “Because this is just an ass-backward tech company, let’s throw non-scalable, low-tech solutions on top of this low-tech, non-scalable problem.”
Setting aside the whole "Ruby/Rails is slow" discussion, I would've loved to see more elaboration by these sources about how a web framework makes moderation and policing "nearly impossible". Compared to what? The PHP scripts that became Facebook?
- maxpupmax 9y agoYeah, that doesn't make any sense to me either. I think it sounds like the complaint is actually that they failed to build out features that enabled easier moderation and better verification (mentioned a paragraph later). My understanding is that Twitter moved away from Ruby/Rails back in 2010-2012 anyway, making the comparison more confusing.
- bshepard 9y agoMore info on this here: https://www.quora.com/Why-did-Twitter-move-away-from-Ruby-on-Rails https://www.quora.com/Why-did-Twitter-move-away-from-Ruby-on...
- taneq 9y agoThat's bizarre. They have a social policy problem, not a technical problem. Short of building a general human-level AI, it's not clear how their tech stack has any bearing on the issue.
- lkrubner 9y agoI followed this story back in 2006 to 2008. I’m sorry I don’t have time to write a real comment with the crucial details. The basic plot is that Twitter initially set out to be a CMS sort of like WordPress, but then they realized they had actually built a messaging platform, which needed a different architecture. You can catch a glimpse of the issues in “Amateur hour is over at Twitter” from 2008. https://www.google.com/amp/s/techcrunch.com/2008/04/23/amateur-hour-over-at-twitter/amp/ https://www.google.com/amp/s/techcrunch.com/2008/04/23/amate... Of course, it is a stretch to suggest that in ten years they couldn’t reinvent the architecture. It took them many years to get beyond the “fail whale”. And it is surprising who they thought of as experts at scaling: “And this week they hired two scaling experts – John Kalucki and Steve Jenson (“known for his work scaling Blogger and Blogspot”).” No offense to those guys, but Blogger and Blogspot were also notorious failures at scaling. Bloggers problems were not fixed till a year or two after they were bought by Google.
- liquidgecka 9y agoHi! I am one of the dudes that actually helped kill the fail whale so I feel I can speak to this topic. =) Steve and John were both good developers that wrote some seriously critical components of the system and were a big part of getting twitter more stable. But reading Techcrunch to figure out what was happening inside of Twitter is asking for serious misinformation. At that time it was very common for people with no idea what they were talking about to write articles about what was going on inside of Twitter, yet do so with authority that they didn't really have. By the time a team of reliability focused people started to get hired Twitter was finally getting smart about not letting employees/outsides control the messaging sent to Techcrunch and the like. Twitter has completely reinvented the architecture since I left so that is not impossible by any means. Lets be clear about the rate that Twitter was growing though. In any 9 month window while I worked there they never failed to grow an order of magnitude in request volume. When I started we would have spikes of a few thousand requests per second. We had a major site outage at 451 tweets per second and we simply couldn't keep up. 9 months later we were spiking 50k requests per second, and we never dropped below 451 tps (barring outage). By the time I left both of those numbers simply couldn't be seen in the graphs anymore. Every time we got technology in place to deal with the scaling issues we were already behind the curve because of how fast things were growing. It felt like being in a car that has amazing 0-60 time, but it just kept accelerating well beyond 60mph.
- booleandilemma 9y agoI’m a .NET developer, not a ruby apologist, but fisher-price infrastructure? Gimme a break. Someone go tell GitHub, AirBnb, Hulu, and Bloomberg.
- akhilcacharya 9y agoBloomberg?
- deleted 9y ago[deleted]
- adventured 9y agoThey work with ruby on rails, eg - https://job-openings.monster.com/Full-Stack-Senior-Software-Engineer-Vault-New-York-NY-US-Bloomberg/31/21445955-46af-49f8-a793-515208dd9e5b https://job-openings.monster.com/Full-Stack-Senior-Software-... https://job-openings.monster.com/Senior-Web-Engineer-BLAW-New-York-NY-US-Bloomberg/31/45377f1b-6c57-40bc-9c32-53aad319332a https://job-openings.monster.com/Senior-Web-Engineer-BLAW-Ne... Bloomberg is fairly well known for having top notch technical capabilities (in their general space). What they build tends to work, work well, and have high reliability.
- akhilcacharya 9y agoI've worked at Bloomberg, I did not know they used RoR in large external facing apps. Only heard about it for smaller internal sites. My team worked on things very far from BLaw and BVault implementation details though.
- aptwebapps 9y agoThe most charitable interpretation I can come up with is that this is a distortion of someone claiming that they couldn't build better anti-harassment tools because of excessive technical debt and maybe a lack technical acumen or focus.
- intended 9y agoThats not charity - thats donating credibility. Whoever was quoted has precisely Zero current understanding of moderation and the challenges it faces. And worse - they haven't reflected on what "technical acumen" resulting on anti-harassment tools means. Effective anti harassment tools can suss out intent, and by nature are also censorship tools. Effective tools which can identify intent are so far past invasion of privacy, that they may as well be categorized as assaults on specific ideas. If twitter does come up with this magical tool, then it's a sure bet that other nation states will also have this magic algorithm. This rule applies to all lesser tools for "anti-harassment". The solution sadly IS hiring more moderators, and journalists vetting. Which I suspect is impossible at the scale and message type twitter operates on.
- dawhizkid 9y agoI assume this "executive" was a non-technical mid-level manager. I don't think either the author of this story or her source has any idea what they're talking about w.r.t web frameworks.
- mattbierner 9y agoThe only way I can understand the argument is that their early UX / technical design influenced the community and culture that we now see. From the start, it seems like Twitter was designed for hottakes and emotional broadcasting, and countless little design decisions emphasized this (everything from the original 140 character limit to the UX around retweeting and followers.) The modern platform’s culture grew out of these early decisions and now embodies them in many ways, so much so that it’s difficult now to imagine that twitter could ever have been anything different. But what if following had only been bi-directional for example? What if the platform never displayed retweet or star count? What if muting and blocking had been built in from the start? What if you could only @mention people who follow you? It’s very possible that twitter would never have become Twitter if these things had been different, but that’s also my point. So while it may not be a technical root cause in the classic coding sense, I can buy the argument that many of twitter’s early technical and design decisions (as well as their early platform policies and who the first adopters were) led to their present difficulties
- liquidgecka 9y agoHi, early Twitter employee here. The reliability team called the infrastructure Fisher-Price internally so this wasn't just one random executive coming up with a term on his/her own. The problem wasn't ruby. The problem was the way that Twitter used Ruby. We had one big mono repo with every single function and every form of business logic baked into a single place. That logic relied on monkey patching and all sorts of crazy horrible glue to keep it working together. Every time we had to scale up we would glue infra in place to keep things working while we came up with a real solution (which never really materialized). In my time there we had memcache instances which held timelines. Populating them took hours/days and while they were unpopulated the site was offline. Rebooting/restarting the caches was simply not an option. We had a data sharding strategy that was temporal. We would spin up a new database cluster every few weeks to handle all of the incoming tweets and failing to spin up a new cluster in time meant we would have a global site outage. Don't even get me started on the "load bearing mac mini". In reality the only problem rails really contributed on its own was that it could only process a single request per process at a time. Each machine would spin up 16 or 32 processes to handle requests in parallel but each process needed its own connection to the database, to memcache, etc. At one point we had something like 100k processes all trying to talk to a single mysql master. Much of this could have been mitigated by better design of course, but rails encourages models that don't scale up to crazy dimensions. In reality moderation was virtually impossible because we were in a 24/7 fight with ourselves about how to keep the system alive for the next couple of days. Constant infighting, managerial changes (I had 9 different managers in 3 years), focus changes (we didn't finish the last major site redesign before starting the next one) and a general unwillingness to pause features long enough to stabilize the system meant we were always on the losing end of a infra battle.
- stryk 9y agoI don't know nearly enough about large scale projects or the startup world maybe this is a silly question but when it became apparent that it was blowing up far beyond what you were expecting, was it not an option to have another team (hire or split existing) in parallel, to start writing a re-implmentation in another language or system that would work better for your new quickly expanding needs? Was that just not feasible?, no doubt it was a crazy hectic time for you folks back then, I can't even really imagine what that was like.