16 ms·
> I run a YouTube channel that is over 10 years old, has over 770,000 subscribers, almost 600 public videos, 120,000,000 views When writing software involved i
by pdkl95 6y ago
> I run a YouTube channel that is over 10 years old, has over 770,000 subscribers, almost 600 public videos, 120,000,000 views
When writing software involved in managing a live, public, massively multi-user system, the traditional unix-style commands that are immediate, often silent, and capable of damaging effects become a really easy way to shoot yourself in the foot. Worse, some commands might let you accidentally shoot everyone's foot on a typo. The traditional example is accidentally typing something like "rm -f * .bak" (note the extra space after the star).
For a good discussion of this type of problem, I highly recommend Bryan Cantrill's talk[1] about the time an operator accidentally rebooted an entire datacenter with a single miss-typed command.
The general solution to this is building sanity checks into the software. The user just asked to reformat 500 hosts, but almost all previous uses olf the 'reformat' command affected less than 10 hosts. Maybe we should ask for verification from an actual human if they really intended to run this unusually destructive command.
Why doesn't YouTube have this kind of sanity check in their automated takedown/strike/channel-deletion tools? Google wrote automation that can decide to delete a channel with a long history and many successful videos. Why doesn't that automation have basic sanity checks that ask for operator input when asked to do an unusually destructive action like deleting a 10 year old channel with a huge history?
[1] https://www.youtube.com/watch?v=30jNsCVLpAE https://www.youtube.com/watch?v=30jNsCVLpAE
- Tokkemon 6y agoBecause writing good software is haaaarddd. /s
- tmpz22 6y agoBecause building it won’t get you a promotion at google.
- eru 6y agoThat's what we called 'promotion oriented programming'.
- tgtweak 6y agoWhere is Bryan anyway he's been unusually quiet and I know he lurks on HN. bcantrill where art thou
- cellularmitosis 6y agoHave you heard about Oxide Computer Company? https://oxide.computer/team/ https://oxide.computer/team/
- hahajk 6y agoand if you start implementing these sanity checks, they better be everywhere. Operators will start to rely on them and might become careless - they believe the system will catch any egregious typo. The one spot where the sanity check isn’t implemented will become more vulnerable.
- Ensorceled 6y agoHow could some sanity checks be worse than NO sanity checks at all? In the YouTube case, I mean. This is happening ALL the time.
- hamandcheese 6y agoFor the same reason human babysitters of autonomous vehicles don't succeed in preventing pedestrians from getting run over.
- Ensorceled 6y agoRight, but a human babysitter is better than NO babysitter.
- lmm 6y agoAn inconsistently applied sanity check creates a false sense of security. Like how a safety barrier that isn't strong enough to take someone's weight might be worse than no barrier.
- fiddlerwoaroof 6y agoYeah, some people recommend something like alias rm=‘rm -i’ Unfortunately, as soon as you’re on another computer or at someone else’s shell, you’ll be more comfortable thinking less about rm-ing things and the sanity check won’t be there.
- pwg 6y agoThe better recommendation is to alias rm -i to del: alias del='rm -i' And then get in the habit of using 'del' unless you really want a no-confirmation delete. Then when you are on another computer or at someone else's shell you get: del: command not found Instead of having a bunch of files silently deleted.
- Bedon292 6y agoReminds me of when s3 went down in 2017, and took a chunk of the internet with it (to include their own status dashboard) due to someone entering a command wrong. https://aws.amazon.com/message/41926/ https://aws.amazon.com/message/41926/
- mattl 6y agoOther than websites and maybe some web-ish apps not too much else would be affected by S3 outage, a DNS outage would affect more of the Internet vs the web.
- megous 6y agoAll of Google was gone from my country due to bungled router configuration for half a day some time ago. Now that was fun to watch. Everything from 8.8.8.8 to all their backend APIs for android, to all their web properties, went poof.
- mattl 6y agoYikes. Time to use a few different providers?
- CreepGin 6y agoWebsites and web-ish apps are not a chunk of the internet? Also there are tons of non-web-ish apps that depend on S3.
- dylan604 6y agoi work on multiple projects that are 100% data crunching with content stored on S3 and accessed solely by ec2 instances. there is no UI or public access. yet if S3 went down, we'd be up a certain creek without a certain self-propelling device and a big whole in the boat without a bucket (physically and in the cloud)
- Bedon292 6y agoMaybe "the internet" was a bit too generic. But it destroyed a lot of tech worker productivity for the day at least. Slack, GitHub, GitLab, BitBucket, Docker, etc all had issues. https://venturebeat.com/2017/02/28/aws-is-investigating-s3-issues-affecting-quora-slack-trello/ https://venturebeat.com/2017/02/28/aws-is-investigating-s3-i...
- pwillia7 6y agoBecause that would be sprints not spent making more money for Alphabet shareholders
- moltar 6y agoBut that’s not true though. Creators create content and make money with ads. No creators = no ad platform.
- edoceo 6y agoThe number of creators harmed is less than the number of other creators, new creators and other factors. Loss of 0.00001% of the content on YT is, literally, no big deal. You, me, all other creators just don't matter. That is BigG just don't give a f---.
- eru 6y agoNot sure your logic holds. If Google is so big that 0.00001% loss of revenue doesn't register for them, they are surely big enough to spare 0.00001% of total engineering time to fix that loss? In fact, I hear that as a common complaint that when working as an engineer at Google most of the time you are just making some system fractions of a percent more efficient. Just to be sure, I am not saying that either your premise or the statement you are inferring is wrong. No opinion on that. I'm just saying that your premises don't lead to the conclusion. (But your conclusion might be right for other reasons.)
- stickfigure 6y agoTwo days ago I restored a customer's data from backup because they typed "ALL" and hit OK on a dialog that said "This will change ALL of your things, type ALL to confirm". This dialog only pops up if N > 100, and normally makes you type the number (ALL is a special case). I just do not know how to make this more idiotproof short of making them fax in the request. No matter what you think is sane, some insane person will prove you wrong. In this case, tiredness + ESL = bad reading comprehension. It's going to happen.
- waheoo 6y agoReverse survivorship bias...
- laughinghan 6y agoIt sounds like you found the solution, you just made it really cumbersome: undo. All confirmation dialogs should be replaced with undo. The happy path has lower friction, and in case of a mistake they'll heave a huge sigh of relief. When possible, it's better in all cases for all users, whether novices or power users. And many things that at first blush seem like undo isn't possible, are actually easy to make undoable with a simple tweak: deleting data? Don't actually delete it until 24 hours later. Sending an email? Wait 10 seconds to actually send it, similar to Gmail's Undo Send.
- stickfigure 6y agoImplementing soft-delete is much easier than "soft-update", which is what this would have been.
- eru 6y agoYou could still wait 10 seconds, and have a 'fake' undo button that aborts. (You can even put up a progress bar to pretend you are doing work during those 10 seconds.) That's purely a UI element and is completely independent of how the actual destructive operation is implemented in the backend nor how hard it would be to reverse.
- kords 6y agoThey have ways to detect the "unusual". If I fill a form really fast, or something is weird in the state of my browser I get prompted to select crosswalks, road-signs or hydrants. It will be great if they would apply them on their algorithms too, when an "unusual" result occurs.
- bigiain 6y agoI'm now imagining a newly emergent dark pattern, where when you try to delete some expensive cloud resource, but before it'll stop charging you you get into one of those insane recaptcha loops that keeps saying "Click on all the Lovecraftian Gods that drive you insane just by looking at them!" or "type the text" where the displayed text is all in Druidic runes or something...
- noir_lord 6y agoSounds like the Land of Azure.
- tatersolid 6y agoThis is actually a pretty effective control. I almost deleted the wrong storage account in Azure the the other day. We do not generally allow deletions of stateful resources to be done via automation after being bitten, and use the Azure lock mechanism. We are now also changing our naming standards from “component-environment” to “environment-component”. In this example I almost deleted “app-prd-sa” instead of “app-dev-sa”. Much harder to do when you lead the name with “production”. Anybody have “safer” naming conventions in use out there that I should be aware of? Didn’t find much authoritative out there via search; naming things is hard.
- zaroth 6y agoGood luck checking in code that will increase the human burden and reduce automation. Imagine how many TPS reports you would have to fill out to get permission to make a system page a human more often at “Google scale”. When the metrics finally hit the dashboards sounds like it would be career limiting. “But we prevented pdkl95’s channel from being deleted!” is definitely going to be a solid defense for why you hobbled their infallible AI from ruthlessly executing its system oversight responsibility.
- mikenew 6y agoAnd yet they've created a widespread perception that YouTube mistreats it's creators, and we see stories like this all the time now. Surely someone at Google is capable of taking the long view. And if the culture at Google prevents that then Google is broken.
- motoboi 6y agoEvery time a person come close the have a long time view, they get senior enough to leave the company or the product team. All companies have a tendency to lost bearing of its mission and focus on the wellbeing of the company owners or executives.
- lmilcin 6y agoYeah. If the channel owner has spent so much effort it would be fair to expect actual human to spend literally a minute to verify it is legit removal case.
- Sophistifunk 6y agoYoutube runs the way it runs because they don't care if a bunch of creators get hosed. All they care about is keeping the copyright wolves at bay.
- Crosseye_Jack 6y agoUnder DCMA, on receiving a valid counter notification they are free to put the content back on the site, remove the strike from the content uploader and let the courts handle the matter between the two parties. https://twitter.com/leonardjfrench/status/1307948170379513857 https://twitter.com/leonardjfrench/status/130794817037951385... It’s only if Google “ignore” valid dmca takedown notifications they are at risk of losing their safe harbour exceptions. How they handle the counter notifications can be pretty hit and miss. (but tbf, most of the time it’s because the person filing the counter notification didn’t fill it out exactly how Google’s bots like) I’ve seen cases where YT just automatically removed the strike from the account and restored the video on receipt of the counter notice. I’ve seen cases where they have locked the account even though the person filing the DMCA notifications has publicly admitted to taking down the video because it was critical of the work and the video only contained small pieces of the original content (exactly what you would expect to be covered under the criticism and comment portions of fair use exception).
- AstralStorm 6y agoThey're not afraid of DMCA, but big money pulling out if they're insufficiently aggressive about claims...
- Crosseye_Jack 6y agoWhich is why they have Content ID. Big money can use the system to claim the ad rev / remove the content (without giving the user a strike) then putting the onus on the uploader of the video to then take the fight to the claimant for breaching their rights.
- rlayton2 6y agoI don't know, but I've read that the counter here is that YouTube's system isn't actually DCMA - they are pursuing copyright over and above that system, and therefore the "protections" in DCMA don't apply - you broke a Google policy, not a law, and therefore you have no recourse.
- nxpnsv 6y agoWeigh the cost of false positive against false negative. In the current landscape YT has little incentive to change.