10 ms·
Oh my, this is amazing. I'm not sure the engineers realized despite their secrecy, it would be noticed by the press immediately after deploy. But the best par
by jrochkind1 8y ago
Oh my, this is amazing.
I'm not sure the engineers realized despite their secrecy, it would be noticed by the press immediately after deploy.
But the best part is how Google engineers immediately on seeing it figured "oh yeah, we should do that too" (although they apparently got the necessary approvals however that was done at Google, it was easier to do because they figured "well, youtube must have done due dillegence before doing it.")
Amazing!
I don't know how they didn't all get fired. Like, ALL of em, including everyone who set up the special "OldTuber" priv long before.
But... it worked! This is a hacker story for the history books, it sounds like the kind of thing programmers did 20+ years ago for nothing except the reward of doing it right (against their own career interests), that I feel like doesn't happen so much in a more professionalized industry.
- ljm 8y agoIt's a fascinating tech story, but it would easily turn into a horror if we weren't talking about something like a video hosting platform. Imagine working at a medical or financial company and having secretive 'old timer' permissions that basically backdoored the company's engineering processes. Even if the engineers saw a good reason to do so. I mean, who are we to say that this was doing it right? It turned out to be beneficial but there are thousands of ways that this could explode in your face if you tried it yourself.
- jrochkind1 8y agoYes. I love the story, and if I had been in charge I don't see how I could do anything but fire the whole lot of them. Although... the number of people involved in the "OldTubers" backdoor seems to have been pretty substantial, I don't know if they could really have fired all of them... But who is anyone to say if doing anything is "right"? We're all equally qualified to have a moral compass.
- derefr 8y agoI mean, Google just paid a ton of money to acquihire this team, and then the acquihired team is going to turn around and fire all the people Google paid for? That doesn't sound like an arrangement Google would be happy with. (Which is to say, with more generality: people who get acquihired are basically temporarily invulnerable to office politics.)
- ascar 8y agoThe engineers involved were not part of YouTube at the time of acquisition. YouTube also wasn't really acquihire. The site had a huge user base Google wanted.
- ljm 8y agoI'd keep them on board but I'd get rid of that backdoor and encourage them to adapt to the new situation, for a number of reasons: - 'OldTubers' vs. the new Googlers is a clear us v. them thing. Bad for culture, especially when you get privileges on top. - Their plan was actually successful so there must have been a thing or two to learn on the product side, even if it's as simple as accepting engineering input and understanding the difficulty of maintaining stuff (as opposed to relying on the analytics). - There's some good thinking and ingenuity in there, despite the methodology. I'd want to keep hold of that talent while dealing with burgeoning elitism. Up to them to quit if they don't like the changes. Fire them if their contribution begins to suffer.
- jrochkind1 8y agoYes, you're right, there are options available other than 'firing' and that's the right one. This is why I'm not a manager. :)
- user5994461 8y agoIt's not a us v. them thing since they were not OldTuber and the article says that many other people were granted the access in spite of not being OldTuber.
- bluedino 8y agoThey’d quit and you’d kill morale. Something Yahoo would do.
- pjc50 8y agoThis may be a management culture thing, but I would regard that happening as a massive red flag and immediately start looking to quit. Formal warning, perhaps, if it actually contravenes a written policy, but in this case is unlikely to have been one. There was no actual financial or reputation loss to the company.
- HeWhoLurksLate 8y agoBut mah autonomy!!! THB, meh.
- Cyberspy 8y agoIt doesn't seem though, from the story, that the 'OldTubers' group was exposed. It seems like the 'why' was more important than the 'how'
- darawk 8y agoThis is exactly the sort of riskless sentiment that causes something like IE6 to persist forever. > I mean, who are we to say that this was doing it right? It turned out to be beneficial but there are thousands of ways that this could explode in your face if you tried it yourself. How would this explode in your face, exactly? What was the risk here?
- saagarjha 8y agoFrom what it looks like, this is quite literally the ability to get code into the main codebase with no oversight. Even without malicious intentions, it seems very easy to do something awful by accident.
- derefr 8y agoThe main codebase... of a video sharing site. With no SLAs, and (at the time) no partner video-hosting agreements. The most awful thing that could happen to 2006-era YouTube: they could stop delivering ad impressions for an hour or two. Oh wait, no; 2006-era YouTube didn't even have ads yet.
- jrochkind1 8y agoI suppose more awful thing could be an accidental XSS or other vulnerability.
- darawk 8y agoWhich would then compromise what? People's youtube accounts?
- 0xDEFC0DE 8y agoIE6 had plenty of nasty vulns by the time this banner had gone up (06/14/2009): https://www.cvedetails.com/vulnerability-list/vendor_id-26/product_id-124/version_id-8371/Microsoft-IE-6.0.html https://www.cvedetails.com/vulnerability-list/vendor_id-26/p... That said, the people with this access probably knew WTF they were doing.
- soulofmischief 8y agoThey weren't working at a medical or financial company so your argument is pointless.
- user17843 8y agoThis is not comparable imho. No medical or financial company has this vibrant engineering IT culture like the web startups 20 years ago had. Everything was new and exciting, and things moved foreward exactly due to the flexibility of all involved, the uncertainty, the flat hierarchies, and the fact that many rules either didn't exist, or only existed on paper. This is what happens everytime a new market emerges, where structures need some time to settle. Once they have settled down, procedures become standardized, but they also become so boring that all innovation is lost. What happend at Youtube 12 years ago is exactly what happens right now at some Cryptocurrency start ups. The start ups that are succesful are exactly those start ups where people like this Youtube engineer do not get fired. If YT had such an authoritarian attitude in 2009, they would never have become succesful. It's easy for the blog author to say "I have no idea why we weren't fired" but I bet back then it was all fun and games. (Getting fired isn't even that bad, what really harms people is when everyone around them starts to tell them things like "You could have ruined everything!") And something similar to the excitement and fluidity within the Youtube of 12 years ago happened 80 years ago in pharmaceutical companies, or 200 years ago within a Bank, when such structures were still freshly formed.
- user5994461 8y agoWhat makes you think it's different in financial organizations?
- user17843 8y agoI just implied it is different because the industry was so young, it is solidified now, thus resembling other industries. (i.e. in the corporate world the hierarchies in big tech now resemble the hierarchies in big [insert any other industry])
- antoineMoPa 8y agoIn my experience, I think it is a question of developper count and company culture. Larger corporations have processes & checks that smaller businesses do not always have.
- deleted 8y ago[deleted]
- jeromebaek 8y agoThis is correct, but there's a complication: YouTube today is a lot more than the YouYube of 2006. Today, people's livelihoods depend on YouTube. Naturally, I'd expect more responsible engineers.
- megaremote 8y agoIf enough developers want to do something, you can't stop them. You can make it take more developers, that is the best you can do.
- profalseidol 8y agoPretty sure there was enough developers whom hated JavaScript. What happened is that made more if it. A reality that we would still be supporting IE6 today because so many businesses still use it seems realistic to me. Maybe we would have developed a facade for it.. This brave actions of the OldTuber saved us a ton of effort. Hopefully somebody from Oracle sneak in Clojure in the JVM!
- baud147258 8y ago> Imagine working at a medical or financial company and having secretive 'old timer' permissions that basically backdoored the company's engineering processes Have you heard about Kerviel? A trader that cost his banks a few billions of euros? He was able to do this because he could bypass reviews by his boss and the internal compliance team. https://en.wikipedia.org/wiki/J%C3%A9r%C3%B4me_Kerviel https://en.wikipedia.org/wiki/J%C3%A9r%C3%B4me_Kerviel Edit: to be fair, it look like he took the blame for all the failings of the company
- makapuf 8y agoThe whole question about Kerviel is if his management let him "bypass" regulatory safety nets or if he did it by himself.
- ufmace 8y agoThis seems like a kind of silly objection. It's kind of like - I have some knives in my kitchen for cutting food, as does most of the world's population. I, or anyone else, could pick one up and go out and stab somebody. 99% of the world is capable of not doing that. Can't we presume that all of this setup happened because everyone involved knew perfectly well that it was just a moderately popular video hosting site? And that they would all have behaved differently if they knew that peoples' life savings and medical histories were at risk? Did it go out of style somehow to presume that people were mostly sane most of the time?
- AceJohnny2 8y ago> I don't know how they didn't all get fired. Like, ALL of em, including everyone who set up the special "OldTuber" priv long before. I can think of 2 reasons: 1. The gamble paid off. It turned out the time was ripe. Thus, there was really no negative impact on the organization that people had to be punished for. 2. The OldTubers were the experienced backbone of the group. Firing everyone with that privilege would likely have destroyed the group's technical expertise. YouTube as it was would have died. That latter point is a bit of a strong negotiation position that many engineers don't realize they have. For example in my team we're having a lot of difficulty filling job openings (we're picky and frankly our interviewing pipeline sucks). As such, there's a significant cost/benefit calculation to firing someone. Maybe the person did something egregious, but is it likely to happen again? I know some people in my team are... not great, but are they so bad that we would be better off without them? (I must reluctantly admit, no)
- Zak 8y agoI can think of a third reason: everybody in the industry wanted IE6 dead. It was expensive to maintain support for an old, broken browser and often meant new features couldn't be added. The people who should have fired the conspirators were glad that the conspiracy succeeded even if they had objections to the methods used.
- neilv 8y agoI'm really glad this method of execution was an honest banner. (The biggest problem here might've been cutting colleagues out of the loop it sounds like they were probably supposed to be in.) When I started reading the story, I was afraid it was going to go in the direction of sneaky sabotage, like intentionally causing performance problems or crashes. There've been accusations of that happening, within the last year. Earlier, there was a long history in the computer industry of sneaky sabotage of competitors, which was suggested by the (perhaps only a joke, in that particular case) mantra "DOS isn't done, till Lotus doesn't run".
- closeparen 8y ago
- Causality1 8y agoSomething that stuck out to me was how concerned Google was that even while recommending other browsers next to Chrome, that putting Chrome first in the list could rankle antitrust regulators. Fast forward to 2019 and Windows 10 makes you fight multiple menus and "suggestions" just to switch your default browser away from Edge.
- tatersolid 8y agoYour comment is demonstrably nonsense. Windows 10 only suggests Edge if you make IE11 your default browser. If you choose Firefox or Chrome it never suggests that you should switch to Edge. Microsoft’s lawyers are not stupid.
- buffalo2391 8y agoThis isn't exactly accurate, I've had multiple menus telling me to switch to edge from chrome.
- pbhjpbhj 8y agoPerhaps I'm not remembering it right but doesn't it day something like "you should use a stable modern browser instead! Are you sure you don't want to keep using Edge"??
- TeMPOraL 8y agoIt's more like "Windows has a modern browser called Edge, you might want to try it out; are you sure you want to change the default?".
- ddingus 8y agoThey are missing an awesome opportunity. Provide a place for people to say why they won't. A lot of that input will be garbage. Some seriously laughable in a good way, a few gems. Why not get them?
- terinjokes 8y ago
- user5994461 8y ago>>> I don't know how they didn't all get fired. Like, ALL of em, including everyone who set up the special "OldTuber" priv long before. That's really blowing it out of proportions. It's a minor button addition to the UI and they paid attention to not be sued by the European commission for it for anti competitive behavior. It's a job well done with nothing to criticize. It's a BAU example of how works get done in a large bureaucratic organization.
- Iv 8y agoThis part is why it worked: "He had come in on an otherwise normal day to find email from every major tech news publication asking why the second largest website on the planet was threatening to cut off access to nearly a fifth of its user base. Fortunately for us, the publications had already settled on a narrative that this was a major benefit to the Internet. By their call, YouTube was leading the charge towards making the web a faster, safer experience for all of its users." Go with the Zeitgeist, and you'll have a lot of wind in your sails.
- underwater 8y ago> I don't know how they didn't all get fired. That seems extreme. I've worked in some organizations where the assumption is that engineers aren't capable of making product decisions. They are merely walking, talking machines that are there to implement the vision of the PM and designers. In other orgs engineers are very much in tune with how product decisions are made, what the business is optimizing for, and what constraints they need to be aware of. They will generally work with the product owners to shape the user experience. I'm guessing that YouTube was much more in the latter category.
- solveit 8y agoYes, but >They will generally work with the product owners is exactly what the engineers didn't do in this story. The company was completely blindsided.
- westoque 8y agoWhat's funny is we actually used YouTube as our "gauge" on what browsers to support. We also built the narrative that: "If YouTube doesn't work on this browser, then we will not support it"
- nl 8y agoI don't know how they didn't all get fired. Like, ALL of em, including everyone who set up the special "OldTuber" priv long before. Because this is how everything on the web used to be done. Google won the internet partly because they moved away from this first and best, but product decision making by engineering and overriding approval processes in the name of a good outcome were pretty much standard back then.
- DonHopkins 8y agoIt's just like Murder on the Orient Express! https://en.wikipedia.org/wiki/Murder_on_the_Orient_Express_(2017_film) https://en.wikipedia.org/wiki/Murder_on_the_Orient_Express_(... >The next morning, Poirot discovers Ratchett was murdered during the night, having been stabbed a dozen times. [...] >With the train back on track, Poirot concludes that justice is impossible in this case, as Cassetti deserved death; for the first time, Poirot will have to live with a lie and imbalance.
- tjpnz 8y agoThey got visits from PR and Legal but there's no mention of a Project Manager - the sort of person I could imagine being the most pissed. Do PMs at YT not rule over their engineers with an iron fist?
- stingraycharles 8y agoPerhaps there was no PM because there was no official Project?
- zrobotics 8y agoThis is a classic "better to ask for forgiveness than ask for permission" story. They didn't get fired because it turned out not to be a huge negative PR hit. Honestly they were just lucky in that regard, if the press had been negative about the banner they likely would have been disciplined.