9 ms·
This is actually funny, because I was involved with the creation of this list, way back in 2004. The whole thing started as a way to stop phishing. I was work
by jedberg 6y ago
This is actually funny, because I was involved with the creation of this list, way back in 2004. The whole thing started as a way to stop phishing.
I was working at eBay/PayPal at the time, and we were finding a bunch of new phishing sites every day. We would keep a list and try to track down the owners of the (almost always hacked) sites and ask them to take it down. But sometimes it would take weeks or months for the site to get removed, so we looked for a better solution. We got together with the other big companies that were being phished (mostly banks) and formed a working group.
One of the things we did was approach the browser vendors and ask them if we could provide them a blacklist of phishing sites, which we already had, would they block those sites at the browser level.
For years, they said no, because they were worried about the liability of accidentally blocking something that wasn't a phishing site. So we all agreed to promise that no site would ever be put on the list without human verification and the lawyers did some lawyer magic to shift liability to the company that put a site on the list.
And thus, the built in blacklist was born. And it worked well for a while. We would find a site, put it on the list, and then all the browsers would block it.
But since then it seems that they have forgotten their fear of liability, as well as their promise that all sites on the list will be reviewed by a human. Now that the feature exists, they have found other uses for it.
And that is your slippery slope lesson for today! :)
- yuvalr1 6y agoThis is an amazing story. It really demonstrates the way we pave our road to hell with good intentions... We should really do something about this issue, where so few companies (arguably, a single one) hold so much power over the most fundamental technology of the era.
- ocdtrekkie 6y agoThe solution is simple: Liability. As soon as it becomes legally infeasible to let algorithms block people, it will stop happening. Make it easy and affordable to submit legal complaints for tech misbehavior and make the penalties hurt.
- vkou 6y agoAh, so you suggest liability for the vendors of the software blocking websites, with, in practice [1], no liability for the operators of a compromised website, if it is phishing/malware? This is a great approach, if your goal is to optimize for increasing the amount of dangerous crap on the web. But, eh, that's surely worth it, because the profitability of startups is more important then little things like the security of the average netizen... [1] Even if you make the operators liable [2], in practice, you'll never be able to collect from most of them. Whereas the blacklist curators are a singular, convenient target... [2] If you can demonstrate how the operators of compromised websites can be held liable for all the harm they cause, I will happily agree that we should do away with blacklists. Unfortunately, the technical and legislative solutions for this are much worse than the disease you are trying to treat.
- smsm42 6y agoSince phishing is not going to go anywhere with or without blacklists - for obvious reasons, e.g. lists can't cover everything, and you can't add sites to the list instantly - I am willing to tolerate a slight increase in fishing which is going to exist anyway in exchange for not having Google (or any other megacorp, or any other organization for that matter) as a gatekeeper of everybody's access to the internet. The potential for abuse of such power is much greater and much more dangerous than the danger from tiny increase of phishing.
- mewse 6y ago> I am willing to tolerate a slight increase in fishing According to Google's most recent transparency report[1], as of December 20th of last year they were blocking around 27,000 malware distribution sites and a little over 2,000,000 phishing sites. In your view, would turning off those blacklists and allowing those >2,000,000 sites to become functional again count as a "slight" increase? (edit: That's a real question, incidentally, not a disagreement or an attempt at a 'zing'; I have no knowledge in this area but went to look up the numbers, and am curious whether 2,000,000 is truly a vanishingly small amount, relative to everything else that's out there that's not already on the list) [1]: https://transparencyreport.google.com/safe-browsing/overview https://transparencyreport.google.com/safe-browsing/overview
- teamspirit 6y agoHere-here! I really wish there was more human involvement in a lot of these seemingly arbitrary AI-taken actions. Everything from app review to websites and more. This heavy reliance on automated systems has led us down this road. Shoot, keep it, just give us the option to guarantee human review - with of course transparency. We don't need anymore "some human looked at this and agreed, the decisions is final, goodbye." I know it's easier said than done, especially when taking the scale of the requests into account, but the alternative has, does, and will continue to do serious harm to the many people and businesses caught in this wide, automated net.
- alfiedotwtf 6y ago> I really wish there was more human involvement in a lot of these seemingly arbitrary AI-taken actions. Narrator: but it was only ever to get worse
- pryce 6y agoIt's interesting how closely the unfolding of this awful scenario has followed an entirely predictable path based on the shifting incentives: now hundreds of thousands of businesses face the same massive hazard of blocklisted without adequate human review, and with mediocre options to respond to it if it occurs. Without a shift in incentives, its unlikely the outlook will improve. Unless the organisations affected (and those vulnerable) can organise and exert enough pressure for google to notice and adjust course, we're probably going to be stuck like this -or worse- for a long time.
- craftinator 6y ago> this awful scenario has followed an entirely predictable path The interesting things about predictable paths is that at the start there are a LOT of them, then over time there becomes just one of them. I don't see that this path was any more predictable at the start than any other.
- Retric 6y agoBlacklisting a site incorrectly seems like a perfectly adequate reason for a defamation lawsuit. So, I think the real issue is with the legal system.
- ehsankia 6y agoAre you implying that the list no longer has a good intention? I wouldn't be surprised if there are multiple orders of magnitude more phishing and hacked websites in 2021 than there was in 2004. Even with human checking, I doubt you'll even have 0% failure rate. Is the solution to just give up on blocking phishing sites?
- cft 6y agoThe solution is for the legitimate sites that are driven out of business by Google AI to sue Google for tortuous interference and libel.
- nearbuy 6y agoThis helps one group and hurts another. If Google is liable for blocking potential malware and phishing pages, they'll either stop blocking it, or adjust their algorithm to strongly err on the side of allowing phishing sites. Businesses become safer, but more regular people will get phished.
- coldtea 6y ago>or adjust their algorithm to strongly err on the side of allowing phishing sites. It'a not the role of Google to disallow phishing sites (as a browser) just like it's not the role of the ISP. Make it hookable so people can chose their own phising protection service.
- qwantim1 6y agoThis was the case with railroads too, only a few controlled the biggest and most transforming and business-integral tech of 1800s. Prior to that it was those that controlled the printing presses. ... History continues to repeat itself.
- __turbobrew__ 6y ago> since then it seems that they have forgotten their fear of liability They most likely have offloaded the liability to a “machine learning algorithm”. It’s easy for companies to point the finger at an algorithm instead of them taking responsibility.
- worik 6y agoWhich then leads them to the awkward place of having to be transparent about how their algorithm work Either take responsibility, or be transparent. But we all want our cake and eat it
- monsieurbanana 6y agoI take offense to this. Sure, I like to eat cake. But if I liked to eat cake as much as Google does, I'd have died of obesity (= have my life ruined by legal issues) a long time ago.
- megous 6y ago2 millions phishing sites and counting... with 40000 websites added each week. https://transparencyreport.google.com/safe-browsing/overview?unsafe=dataset:1;series:malwareDetected,phishingDetected;start:1148194800000;end:1609660800000&lu=unsafe https://transparencyreport.google.com/safe-browsing/overview... I guess the automation started in 2007 or so.
- jedberg 6y agoI think you mean 2017? 2007 is when the feature launched.
- megous 6y agojanuary/february 2007 looks like the time the list jumped from a few hundred to tens of thousands of sites.
- ehsankia 6y agoMalware may be easier to catch, but for phishing, it was fairly small (under 150k) until around 2016 where it starts growing linearly.
- jedberg 6y agoThat was the normal volume of manually identified sites at the time. Before 2007 there weren’t a lot of participants because it was in beta.
- forgotmypw17 6y agoLike some kind of perverse blockchain, no site is ever removed, even though most phishing sites don't live long.
- underwater 6y agoLiability was my first though. How is an assertion that a site contains malware not libel? Site would be easily able to demonstrate lost revenue.
- CrazyCatDog 6y agoSimple solution = let google use their, imperfect (false-positives) filter, allow them to collect $12 / year not to be blacklisted, and google to send all revenue to the Electronic Frontier Foundation or similar internet defending foundations.
- TimTheTinker 6y agoAnother road to hell paved with good intentions. Once everyone’s paying, who’s to stop them from pocketing the money instead? “After careful review, we’ve concluded that the Electronic Frontier Foundation no longer aligns with the goals of Google or its parent company Alphabet Inc. to the extent we require from recipients of our Freedom Fund. We will place these funds in a separate account and use them in ways we believe will be in the best interest of digital freedom, both now and in the future.”
- alexcohn 6y agoWorse, getting such money flow, EFF will get corrupt very soon.
- TimTheTinker 6y agoAbsolutely. There’s nothing like guaranteed money or power to corrupt people.
- TriNetra 6y agoSomething similar I've just read in zero to one (by Blake Masters and Peter Thiel). Peter argues that computers can't replace humans - it'd be foolish to expect that at least for coming decades – strong AI replacing human is the problem of 22nd century. He proposes Complementarity and provides a successful implementation of this idea in PayPal fraud detection system way back in 2002 when purely automated detection algorithms were quickly overcome by determined fraudsters. He went on founding Palantir based on the same idea. >>> In mid-2000, we had survived the dot-com crash and we were growing fast, but we faced one huge problem: we were losing upwards of $10 million to credit card fraud every month. Since we were processing hundreds or even thousands of transactions per minute, we couldn’t possibly review each one—no human quality control team could work that fast. So we did what any group of engineers would do: we tried to automate a solution. First, Max Levchin assembled an elite team of mathematicians to study the fraudulent transfers in detail. Then we took what we learned and wrote software to automatically identify and cancel bogus transactions in real time. But it quickly became clear that this approach wouldn’t work either: after an hour or two, the thieves would catch on and change their tactics. We were dealing with an adaptive enemy, and our software couldn’t adapt in response. The fraudsters’ adaptive evasions fooled our automatic detection algorithms, but we found that they didn’t fool our human analysts as easily. So Max and his engineers rewrote the software to take a hybrid approach: the computer would flag the most suspicious transactions on a well-designed user interface, and human operators would make the final judgment as to their legitimacy. Thanks to this hybrid system—we named it “Igor,” after the Russian fraudster who bragged that we’d never be able to stop him—we turned our first quarterly profit in the first quarter of 2002 (as opposed to a quarterly loss of $29.3 million one year before). The FBI asked us if we’d let them use Igor to help detect financial crime. And Max was able to boast, grandiosely but truthfully, that he was “the Sherlock Holmes of the Internet Underground.” This kind of man-machine symbiosis enabled PayPal to stay in business, which in turn enabled hundreds of thousands of small businesses to accept the payments they needed to thrive on the internet. None of it would have been possible without the man-machine solution—even though most people would never see it or even hear about it.
- apple_innocent 6y ago"For years, they said no, because they were worried about the liability of accidentally blocking something that wasn't a phishing site." Can anyone explain how a web browser author could be liable for using a blacklist. Once past the disclaimer in uppercase that precedes every software install, past the Public Suffix (White)List that browsers include, how do you successfully sue the author of a software program, a web browser, for having a dommainname blacklist. Spamhaus was once ordered to pay $11 million for blacklisting some spammers, but that did not involve a contractual relationship, e.g., a software license, between the spammers and Spamhaus.
- kelnos 6y agoI think the situation is actually exactly like the Spamhaus case you describe: it wouldn't be the browser user that sues, but the blocked website's owner. The website's owner need not have accepted any kind of agreement from the browser maker in order to be harmed by the blocklist.
- apple_innocent 6y agoPerhaps the website would sue the author of the list. That does not explain why this comment suggests a browser author was afraid to use the list. The browser author could easily require the list author to agree that the browser author has no obligations to the list author if the list author gets sued by a website, and the list author must idemnify the browser author if the browser author is named in any suit over the list. The list author must assume all the risk.
- eitland 6y agoCan someone dig out that old agreement to see if Google can be sued big time for this? I doubt it but I must say it would make me happy and that would be weird because Schadenfreude normally isn't my thing.
- SmokyBourbon 6y agoIt’s amazing that a person can write this and not feel any remorse for the pain they caused.
- varispeed 6y agoThat's very interesting. Would you not think for a moment that such mechanism could be abused?
- jedberg 6y agoThe internet was a much kinder trusting place back then. We assumed when the browser makers agreed to not use it for bad things, we believed them.
- varispeed 6y agoI think as always great ideas do not account for human nature...
- Const-me 6y ago> there is no chance in hell that the government will try to break them up. Government is not the only option. Railroads were fixed by Congress. If you want to fix or split Google, writing your representative about your concerns might help.