7 ms·

Google Safe Browsing missed 84% of confirmed phishing sites

296 points by jdup7 7mo ago

passwordoops 7mo ago
Anecdotal and loosely related, but I can say since Gemini was forced into Gmail, much more obvious SPAM passes the filter
xnx 7mo ago
Why should I trust that "Norn Labs" knows what is and is not a phishing site?
supermatt 7mo ago
> When we ran the full dataset through the deep scan, it caught every single confirmed phishing site with zero false negatives. The tradeoff is that it flagged all 9 of the legitimate sites in our dataset as suspicious Huh? Does this mean it just flagged everything as suspicious?
- badgersnake 7mo ago
  lol, return false;
- john_strinlai 7mo ago
  indeed... it seems like it just says everything is phishing... which they go on to say is desirable? "The tradeoff is that it flagged all 9 of the legitimate sites in our dataset as suspicious, which is worth it when you're actively investigating a link you don't trust." so, you dont really need the scanning product at all. if you just assume every website is a phishing website, you will have the same performance as the scanner!
- jdup7 7mo ago
  Yeah probably could have done better at describing the methodology. The dataset is just the confirmed (manually by a human) phishing urls. We only included the FPs to show that the tooling isn't perfect there were many TNs that we did not include. Going forward we could definitely frame these results better.
lich_king 7mo ago
I don't understand the metric they're using. Which is maybe to be expected of an article that looks LLM-written. But they started with ~250 URLs; that's a weirdly small sample. I'm sure there are tens of thousands malicious websites cropping up monthly. And I bet that Safe Browsing flags more than 16% of that? So how did they narrow it down to that small number? Why these sites specifically?... what's the false positive / negative rate of both approaches? What's even going on?
- jdup7 7mo ago
  Probably could have been a bit more descriptive around the dataset. Our tooling pulls in a lot more than 250 URLs but since we are manually confirming them that means a smaller dataset. In other words, out of the urls we pulled in these 250 were confirmed (by a human) as phishing. We did not do any selection beyond that. As for the article LLMs were used to help with the graphs and grammatical checks but that's it. This was our first month of going through this exercise and we definitely want to have larger datasets going forward as we expand capacity for review. As for Safe Browsing catching more than 16% it depends on the timeline at the time these attacks are launched it's likely Safe Browsing catches closer to 0% but as the time goes on that number definitely climbs.
- john_strinlai 7mo ago
  >what's the false positive / negative rate of both approaches the false positive rate is 100%. they just say everything is phishing: "When we ran the full dataset through the deep scan, it caught every single confirmed phishing site with zero false negatives. The tradeoff is that it flagged all 9 of the legitimate sites in our dataset as suspicious, which is worth it when you're actively investigating a link you don't trust."
  - lorenzoguerra 7mo ago
    it's 100% for what they call "deep scan", it's 66.7% for the "automatic scan". Practically unusable anyway
  - eurleif 7mo ago
PunchyHamster 7mo ago
They put them directly in front of search results, why would they not miss them ?
xvector 7mo ago
There's probably like one engineer maintaining this as a side project at the company
- andor 7mo ago
  Yeah, it would be interesting to know how much work is spent on it. I sometimes submit sites when I am targeted by a campaign, but I'm not sure if they end up in their deny-list.
candiddevmike 7mo ago
I'm getting some kind of chrome security warning when using zscaler now. Discussing all of this with non-techies, I think folks are overwhelmed by all of the security warnings they get and have stopped paying attention to them. So what's the point of doing all of this if there isn't some kind of corresponding education on responsible computer use? There needs to be some personal responsibility here, you can't protect people against everything.
dvh 7mo ago
Just yesterday I marked another Gmail phishing scam. This wouldn't be worth mentioning but they are using Google's own service for it. It has to be intentional, there is no other explanation. https://news.ycombinator.com/item?id=46665414 https://news.ycombinator.com/item?id=46665414
- inemesitaffia 7mo ago
  Seen similar with CloudFlare
iqandjoke 7mo ago
But why Apple choose to work with this on Safari?
nico 7mo ago
On a tangent - gmail has a feature to report phishing emails, but it seems like it’s only available on the website. Their mobile app doesn’t seem to have the option (same with “mark as unread”). Is it hidden or just not available?
- bradyd 7mo ago
  The mobile app definitely has mark as unread. It's the envelope icon next to the trashcan (the exact same icon as in the web interface). Never realized there was a report phishing option. I just mark those emails as spam, which is available in the app.
itvision 7mo ago
Criminals can easily show Google crawlers "good" websites. The fact that Safe Browsing even works is already good enough.
7777777phil 7mo ago
Blocklists assume you can separate malicious infrastructure from legitimate infrastructure. Once phishing moves to Google Sites and Weebly that model just doesn't work.
lorenzoguerra 7mo ago
>We also ran the full dataset of 263 URLs (254 phishing, 9 confirmed legitimate) through Muninn's automatic scan. This is the scan that runs on every page you visit without any action on your part. On its own, the automatic scan correctly identified 238 of the 254 phishing sites and only incorrectly flagged 6 legitimate pages. ...so it has a false positive rate of 67%? On a ridiculously small dataset?
- jdup7 7mo ago
  Fair point in isolation that number doesn't look good. The important context is that this dataset was built to test phishing detection, not to measure false positive rates on normal traffic. It's sourced from our threat intelligence tooling so it's almost entirely malicious URLs by design. The 9 clean sites aren't a random sample of everyday browsing. They're sites that were submitted as suspicious and turned out to be legitimate so they're basically the hardest possible set of clean pages to correctly classify. This seems like a common critique and we definitely could have done a better job of explaining the methodology. Going forward we will include numbers from daily use to give a better picture of FP rate.
mholt 7mo ago
I never loved the idea of GSB or centralized blocklists in general due to the consequences of being wrong, or the implications for censorship. So for my masters' thesis about 6-7 years ago now (sheesh) I proposed some alternative, privacy-preserving methods to help keep users safe with their web browsers: https://scholarsarchive.byu.edu/etd/7403/ https://scholarsarchive.byu.edu/etd/7403/ I think Chrome adopted one or two of the ideas. Nowadays the methods might need to be updated especially in a world of LLMs, but regardless, my hope was/is that the industry will refine some of these approaches and ship them.
- notepad0x90 7mo ago
  Block lists will always be used for one reason or another, in this case these are verified malicious sites, there is no subjective analysis element in the equation that could be misconstrued as censorship. But even if there was, censorship implies a right to speech, in this case Google has the right to restrict the speech of it's users if it so wishes, matter of fact, through extensions there are many that do censor their users using Chrome.
  - rstupek 7mo ago
    I know for a fact that GSB contains non-malicious sites in its dataset.
    - notepad0x90 7mo ago
      It is possible for sure. what's your point? spamhaus does too with IPs, abuse.ch does too, every enterprise firewall's reputation list does too. that's the whole point of reputation, if it was reliable 100% it wouldn't be "reputation".
      - rstupek 7mo ago
        You claimed they all are malicious sites or they wouldn’t be included but that’s factually incorrect
sirpilade 7mo ago
But hits 100% of browsing tracking
- blell 7mo ago
  Educate yourself on how it works before you say something like this.
  - sirpilade 7mo ago
    Pun aside, I cannot fully trust a centralized URL checker on a remote server that I don’t own, even if they guarantee that my privacy is safe
notepad0x90 7mo ago
Glass is half empty, I see. How about GSB stopped 16% of phishing sites? that's still huge.
- debo_ 7mo ago
  I guess the glass is 16% full.
  - notepad0x90 7mo ago
    Stop locking your door if someone can just break through the window then. I think you and the author are conflating 16% effective with 84% of sites on GSB are false positives, that's not what that stat means.
    - debo_ 7mo ago
      I don't know what part of my comment led you to believe I was thinking.
- loloquwowndueo 7mo ago
  Would you use anything that was only 16% effective for its claimed purpose? “Tylenol stops headaches in 16% of people” - it’s huge, right? That’s millions of people we’re talking about. Would you use it?
  - mock-possum 7mo ago
    Idk why not? What’re the side effects?
  - HeatrayEnjoyer 7mo ago
    Countless medications have <16% efficacy rate.
  - NekkoDroid 7mo ago
    If the other options would just straight up kill innocent bystanders (e.g. false positives for legit shops) I think that is a tradeoff I am willing to make.
epicprogrammer 7mo ago
Having spent some time in the anti-abuse and Trust & Safety space, I always take these vendor reports with a massive grain of salt. It’s a classic case of comparing apples to vendor-marketing oranges. A headline screaming about an 84% miss rate sounds like a systemic collapse until you look at the radically different constraint envelopes a global default like GSB and a specialized enterprise vendor operate under. The biggest factor here is the false-positive cliff. Google Safe Browsing is the default safety net for billions of clients across Chrome, Safari, and Firefox. If GSB’s false-positive rate ticks up by even a fraction of a percent, they end up accidentally nuking legitimate small businesses, SaaS platforms, or municipal portals off the internet. Because of that massive blast radius, GSB fundamentally has to be deeply conservative. A boutique security vendor, on the other hand, can afford to be highly aggressive because an over-block in a corporate environment just results in a routine IT support ticket. You also have to factor in the ephemeral nature of modern phishing infrastructure and basic selection bias. Threat actors heavily rely on automated DGAs and compromised hosts where the time-to-live for a payload is measured in hours, if not minutes. If a specialized vendor detects a zero-day phishing link at 10:00 AM, and GSB hasn't confidently propagated a global block to billions of edge clients by 10:15 AM, the vendor scores it as a "miss." Add in the fact that vendors naturally test against the specific subset of threats their proprietary engines are tuned to find, and that 84% number starts to make a lot more sense as a top-of-funnel marketing metric rather than a scientific baseline. None of this is to say GSB is perfect right now. It has absolutely struggled to keep up with the recent explosion of automated, highly targeted spear-phishing and MFA-bypass proxy kits. But we should read this report for what it really is: a smart marketing push by a security vendor trying to sell a product, not a sign that the internet's baseline immune system is totally broken.
- Medowar 7mo ago
  > We also ran the full dataset of 263 URLs (254 phishing, 9 confirmed legitimate) through Muninn's automatic scan. This is the scan that runs on every page you visit without any action on your part. On its own, the automatic scan correctly identified 238 of the 254 phishing sites and only incorrectly flagged 6 legitimate pages. [...] The tradeoff is that it flagged all 9 of the legitimate sites in our dataset as suspicious, ... Am I missing something or is that a 66%/100% False Positive Rate on legitimate Sites? If GSB would have that ratio, it would be absolute unusable.. So comparing these two is absolutely wrong...
mrexcess 7mo ago
These statistics would be a lot better if they were compared directly to the same measurements taken from dedicated cloud SWGs/SSEs like Zscaler. My somewhat subjective sense is that the whole industry is in a bit of a rough patch, the miss rate seems to be noticeably climbing all across the board.
pothamk 7mo ago
[flagged]
- tadfisher 7mo ago
  If you are not a bot, I suggest changing your voice so that you are distinguishable from one. You're not wrong, just like you weren't wrong about "one thing that trips people up about asyncio" yesterday, but I noticed the slop-speak immediately. I'm sure others have as well.
  - niwtsol 7mo ago
    Multiple comments that start w/ "what's interesting about" by this user and very similar formatting kind of answers that question on human vs bot. Weird internet we live in these days.
timnetworks 7mo ago
The most dangerous links recently have been from sharepoint.com, dropbox.com, etc. and nobody is going to block those.
- deleted 7mo ago
  [deleted]
varispeed 7mo ago
When Google will remove scams, phishing and other nonsense from their advertising? Especially the scareware stuff, where AI videos say someone might be listened to / hacked and here is the software that will help block it / find it whatnot. Then they collect personal data.
hedora 7mo ago
So, the false negative rate was 84%, but what was the false positive rate? They have a table "AUTOMATIC SCAN RESULTS (263 URLS)" that sort of presents this information. Of the 9 sites that were negatives, they say they incorrectly flagged 6 as phishing. With a false positive rate of 66%, it's not surprising they were able to drive down their false negative rate. Also, the test set of 254 phishing sites with 9 legitimate ones is a strange choice. (Or maybe they need to work on how they present data in tables; tl;dr the supporting text.)
- decimalenough 7mo ago
  The false positive rate was 66% for "automatic scan" and 100% (!) for "deep scan". In other words, you can get these numbers if your deep scan filter is isSuspicious() { return true; }.
  - antonvs 7mo ago
    Brb, applying for YC funding for my new AI-based phishing detection system. (‘return true’ is just a very optimized neural network after all!)
  - dataflow 7mo ago
    I think there might be a confusion here? The 100% seems like the true positive rate (correct detection), not the false positive rate?
    - decimalenough 7mo ago
      Nope, 9 of 9 legit sites were incorrectly flagged: > The tradeoff is that it flagged all 9 of the legitimate sites in our dataset as suspicious
      - dataflow 7mo ago
        Sorry, I think I had my wires crossed somewhere. Yeah, I see now. That's crazy/hilarious.
obblekk 7mo ago
Maybe I’m an outlier but I’d rather this than accidentally block legit sites. Otherwise this becomes just another tool for Google to wall in the subset of the internet they like.
caaqil 7mo ago
Yeah, maybe let's change the title to remove that 84% rate. It's meaningless because it's just 254 websites, given the scale of what Google Safe Browsing deals with. How is this serious? This is a marketing slop. If the title isn't enough indicator, the ending should be: > If you're interested in trying Muninn, it's available as a Chrome extension. We're in an early phase and would genuinely appreciate feedback from anyone willing to give it a shot. And if you run across phishing in the wild, consider submitting it to Yggdrasil so the data can help protect others.
bethekidyouwant 7mo ago
Their example is really dumb. Eventually, you get a fake Microsoft login page, but they clip out the address bar which clearly isn’t a Microsoft address so your auto complete password isn’t going to be put into the form and you’d have to be pretty dumb to type it in my hand or even to know your Microsoft password, it should be some random thing generated by Safari or whatever your password manager is. Not to mention two factor authentication.
- eks391 7mo ago
  Most people are "really dumb" by your standards then. Not only are most people not going to check the URL, but many people don't know how password managers work, and the only reason they use the browser password manager is because it is on by default, and it is saving their collection of 3 reused passwords they manually type at each site when it doesn't auto populate.
dsr_ 7mo ago
Almost all email phishing attempts we receive come from GMail.
kemotep 7mo ago
Default deny and only permitting what you explicitly allow stops 90% of this in a corporate environment. You don’t just leave all your ports open on the firewall and only close the ones exploited. You default deny and only allow the bare minimum you need in.
virken 7mo ago
I'm all for stopping phishing - and the tool sounds great - but I have to say the Web Store Extension listing is very concerning - even with a new company/offering - there's only 4 users - and 1 rating (a 5 of course) - I'd like to try - but seems phishy :-(
virken 7mo ago
I'm all for stopping phishing - and the tool sounds great - but the Web Store Extension listing is very concerning - even with a new company/offering - there's only 4 users - and 1 rating (a 5 of course) - I'd like to try - but seems phishy :-(
thayne 7mo ago
It would be interesting to see how many of the sites safe browsing does block are false positives.
nickphx 7mo ago
"If you're interested in trying Muninn, it's available as a Chrome extension. We're in an early phase " Domain is less than 4 months old.. Software is "early phase".. Already making misleading marketing claims of usefulness..
kopollo 7mo ago
Let me give you a simple detection algorithm. Apply OCR to the screenshot because they often use logos. Also, parse the text from the HTML and compare it to the URL. You can catch a lot of spam this way.You can also examine many parameters in the js html code.
throawayonthe 7mo ago
> ...full dataset of 263 URLs (254 phishing, 9 confirmed legitimate) > ... automatic scan is optimized for precision (keeping false alarms low... really? > When we ran the full dataset through the deep scan, ... it flagged all 9 of the legitimate sites in our dataset as suspicious lol
hulitu 7mo ago
The purpose of "Safe Browsing" is to send your URLs to Google.
nickphx 7mo ago
So I tested out the extension.. First the extension spammed me with "login required".. So I click the notification to be taken to a login page.. Great? Now I have to create an account and verify a link.. Now I can test how great this is against a "fresh" facebook phishing page being actively promoted via Facebook Ads.. hxxps://r7ouhcqzdgae76-fsc0fydmbecefrap.z03.azurefd.net/new2/?utm_medium=paid&utm_source=fb&utm_id=6900429311725&utm_content=6900429312725&utm_t erm=6900429314125&utm_campaign=6900429311725 The "extension" did a "scan". {"url":"https://r7ouhcqzdgae76-fsc0fydmbecefrap.z03.azurefd.net/new2/","mode":"light https://r7ouhcqzdgae76-fsc0fydmbecefrap.z03.azurefd.net/new2..."} response: {"classification":"clean"} great work? If I click "Deep scan".. I see a screenshot blob being sent over.. response: { "classification": "phish", "reasons": [ "Our system has previously flagged this webpage as malicious." ] } So if the site were already flagged, why does the "light" scan not show that?
luckman212 7mo ago
Is it possible to disable Safe Browsing AND also not have to manually click to confirm that "yes, I actually do want to keep the file I just downloaded, thank you" every. single. time.
- burnte 7mo ago
  No. I keep checking every year.
- yogorenapan 7mo ago
  The answer you probably don't want: Linux + Firefox. Or just copy link and wget.
RS-232 7mo ago
This seems like proof to me that Safe Browsing is about surveillance, not protection.
Anamon 7mo ago
Forget about checking Safe Browsing. Try reporting phishing sites to Google if you really want a reason to hate them. I report all (manually verified) phishing sites I get spam for to Safe Browsing, then automatically check the listing status daily, and re-report every week if it's still not blacklisted. Usually, takedown by hosters is faster. Phishing sites that stay up long enough often go for 3-4 months before Google finally deems it apt to block them. I think the entire Safe Browsing team consists of one overworked, underpaid intern. Compare this to Microsoft Defender. The time it takes for a site I report there to get blocked is usually measured in hours, not weeks. This is one thing MS actually seems to be running decently.

296 points by jdup7 7mo ago

Anecdotal and loosely related, but I can say since Gemini was forced into Gmail, much more obvious SPAM passes the filter

xnx 7mo ago

Why should I trust that "Norn Labs" knows what is and is not a phishing site?

supermatt 7mo ago

> When we ran the full dataset through the deep scan, it caught every single confirmed phishing site with zero false negatives. The tradeoff is that it flagged all 9 of the legitimate sites in our dataset as suspicious Huh? Does this mean it just flagged everything as suspicious?

badgersnake 7mo ago
lol, return false;
john_strinlai 7mo ago
indeed... it seems like it just says everything is phishing... which they go on to say is desirable? "The tradeoff is that it flagged all 9 of the legitimate sites in our dataset as suspicious, which is worth it when you're actively investigating a link you don't trust." so, you dont really need the scanning product at all. if you just assume every website is a phishing website, you will have the same performance as the scanner!
jdup7 7mo ago
Yeah probably could have done better at describing the methodology. The dataset is just the confirmed (manually by a human) phishing urls. We only included the FPs to show that the tooling isn't perfect there were many TNs that we did not include. Going forward we could definitely frame these results better.

lich_king 7mo ago

I don't understand the metric they're using. Which is maybe to be expected of an article that looks LLM-written. But they started with ~250 URLs; that's a weirdly small sample. I'm sure there are tens of thousands malicious websites cropping up monthly. And I bet that Safe Browsing flags more than 16% of that? So how did they narrow it down to that small number? Why these sites specifically?... what's the false positive / negative rate of both approaches? What's even going on?

jdup7 7mo ago
Probably could have been a bit more descriptive around the dataset. Our tooling pulls in a lot more than 250 URLs but since we are manually confirming them that means a smaller dataset. In other words, out of the urls we pulled in these 250 were confirmed (by a human) as phishing. We did not do any selection beyond that. As for the article LLMs were used to help with the graphs and grammatical checks but that's it. This was our first month of going through this exercise and we definitely want to have larger datasets going forward as we expand capacity for review. As for Safe Browsing catching more than 16% it depends on the timeline at the time these attacks are launched it's likely Safe Browsing catches closer to 0% but as the time goes on that number definitely climbs.
john_strinlai 7mo ago
>what's the false positive / negative rate of both approaches the false positive rate is 100%. they just say everything is phishing: "When we ran the full dataset through the deep scan, it caught every single confirmed phishing site with zero false negatives. The tradeoff is that it flagged all 9 of the legitimate sites in our dataset as suspicious, which is worth it when you're actively investigating a link you don't trust."
- lorenzoguerra 7mo ago
  it's 100% for what they call "deep scan", it's 66.7% for the "automatic scan". Practically unusable anyway
- eurleif 7mo ago

PunchyHamster 7mo ago

They put them directly in front of search results, why would they not miss them ?

xvector 7mo ago

There's probably like one engineer maintaining this as a side project at the company

andor 7mo ago
Yeah, it would be interesting to know how much work is spent on it. I sometimes submit sites when I am targeted by a campaign, but I'm not sure if they end up in their deny-list.

candiddevmike 7mo ago

I'm getting some kind of chrome security warning when using zscaler now. Discussing all of this with non-techies, I think folks are overwhelmed by all of the security warnings they get and have stopped paying attention to them. So what's the point of doing all of this if there isn't some kind of corresponding education on responsible computer use? There needs to be some personal responsibility here, you can't protect people against everything.

dvh 7mo ago

Just yesterday I marked another Gmail phishing scam. This wouldn't be worth mentioning but they are using Google's own service for it. It has to be intentional, there is no other explanation. https://news.ycombinator.com/item?id=46665414 https://news.ycombinator.com/item?id=46665414

inemesitaffia 7mo ago
Seen similar with CloudFlare

iqandjoke 7mo ago

But why Apple choose to work with this on Safari?

nico 7mo ago

On a tangent - gmail has a feature to report phishing emails, but it seems like it’s only available on the website. Their mobile app doesn’t seem to have the option (same with “mark as unread”). Is it hidden or just not available?

bradyd 7mo ago
The mobile app definitely has mark as unread. It's the envelope icon next to the trashcan (the exact same icon as in the web interface). Never realized there was a report phishing option. I just mark those emails as spam, which is available in the app.

itvision 7mo ago

Criminals can easily show Google crawlers "good" websites. The fact that Safe Browsing even works is already good enough.

7777777phil 7mo ago

Blocklists assume you can separate malicious infrastructure from legitimate infrastructure. Once phishing moves to Google Sites and Weebly that model just doesn't work.

lorenzoguerra 7mo ago

>We also ran the full dataset of 263 URLs (254 phishing, 9 confirmed legitimate) through Muninn's automatic scan. This is the scan that runs on every page you visit without any action on your part. On its own, the automatic scan correctly identified 238 of the 254 phishing sites and only incorrectly flagged 6 legitimate pages. ...so it has a false positive rate of 67%? On a ridiculously small dataset?

jdup7 7mo ago
Fair point in isolation that number doesn't look good. The important context is that this dataset was built to test phishing detection, not to measure false positive rates on normal traffic. It's sourced from our threat intelligence tooling so it's almost entirely malicious URLs by design. The 9 clean sites aren't a random sample of everyday browsing. They're sites that were submitted as suspicious and turned out to be legitimate so they're basically the hardest possible set of clean pages to correctly classify. This seems like a common critique and we definitely could have done a better job of explaining the methodology. Going forward we will include numbers from daily use to give a better picture of FP rate.

mholt 7mo ago

I never loved the idea of GSB or centralized blocklists in general due to the consequences of being wrong, or the implications for censorship. So for my masters' thesis about 6-7 years ago now (sheesh) I proposed some alternative, privacy-preserving methods to help keep users safe with their web browsers: https://scholarsarchive.byu.edu/etd/7403/ https://scholarsarchive.byu.edu/etd/7403/ I think Chrome adopted one or two of the ideas. Nowadays the methods might need to be updated especially in a world of LLMs, but regardless, my hope was/is that the industry will refine some of these approaches and ship them.

notepad0x90 7mo ago
Block lists will always be used for one reason or another, in this case these are verified malicious sites, there is no subjective analysis element in the equation that could be misconstrued as censorship. But even if there was, censorship implies a right to speech, in this case Google has the right to restrict the speech of it's users if it so wishes, matter of fact, through extensions there are many that do censor their users using Chrome.
- rstupek 7mo ago
  I know for a fact that GSB contains non-malicious sites in its dataset.
  - notepad0x90 7mo ago
    It is possible for sure. what's your point? spamhaus does too with IPs, abuse.ch does too, every enterprise firewall's reputation list does too. that's the whole point of reputation, if it was reliable 100% it wouldn't be "reputation".
    - rstupek 7mo ago
      You claimed they all are malicious sites or they wouldn’t be included but that’s factually incorrect

sirpilade 7mo ago

But hits 100% of browsing tracking

blell 7mo ago
Educate yourself on how it works before you say something like this.
- sirpilade 7mo ago
  Pun aside, I cannot fully trust a centralized URL checker on a remote server that I don’t own, even if they guarantee that my privacy is safe

notepad0x90 7mo ago

Glass is half empty, I see. How about GSB stopped 16% of phishing sites? that's still huge.

debo_ 7mo ago
I guess the glass is 16% full.
- notepad0x90 7mo ago
  Stop locking your door if someone can just break through the window then. I think you and the author are conflating 16% effective with 84% of sites on GSB are false positives, that's not what that stat means.
  - debo_ 7mo ago
    I don't know what part of my comment led you to believe I was thinking.
loloquwowndueo 7mo ago
Would you use anything that was only 16% effective for its claimed purpose? “Tylenol stops headaches in 16% of people” - it’s huge, right? That’s millions of people we’re talking about. Would you use it?
- mock-possum 7mo ago
  Idk why not? What’re the side effects?
- HeatrayEnjoyer 7mo ago
  Countless medications have <16% efficacy rate.
- NekkoDroid 7mo ago
  If the other options would just straight up kill innocent bystanders (e.g. false positives for legit shops) I think that is a tradeoff I am willing to make.

epicprogrammer 7mo ago

Having spent some time in the anti-abuse and Trust & Safety space, I always take these vendor reports with a massive grain of salt. It’s a classic case of comparing apples to vendor-marketing oranges. A headline screaming about an 84% miss rate sounds like a systemic collapse until you look at the radically different constraint envelopes a global default like GSB and a specialized enterprise vendor operate under. The biggest factor here is the false-positive cliff. Google Safe Browsing is the default safety net for billions of clients across Chrome, Safari, and Firefox. If GSB’s false-positive rate ticks up by even a fraction of a percent, they end up accidentally nuking legitimate small businesses, SaaS platforms, or municipal portals off the internet. Because of that massive blast radius, GSB fundamentally has to be deeply conservative. A boutique security vendor, on the other hand, can afford to be highly aggressive because an over-block in a corporate environment just results in a routine IT support ticket. You also have to factor in the ephemeral nature of modern phishing infrastructure and basic selection bias. Threat actors heavily rely on automated DGAs and compromised hosts where the time-to-live for a payload is measured in hours, if not minutes. If a specialized vendor detects a zero-day phishing link at 10:00 AM, and GSB hasn't confidently propagated a global block to billions of edge clients by 10:15 AM, the vendor scores it as a "miss." Add in the fact that vendors naturally test against the specific subset of threats their proprietary engines are tuned to find, and that 84% number starts to make a lot more sense as a top-of-funnel marketing metric rather than a scientific baseline. None of this is to say GSB is perfect right now. It has absolutely struggled to keep up with the recent explosion of automated, highly targeted spear-phishing and MFA-bypass proxy kits. But we should read this report for what it really is: a smart marketing push by a security vendor trying to sell a product, not a sign that the internet's baseline immune system is totally broken.

Medowar 7mo ago
> We also ran the full dataset of 263 URLs (254 phishing, 9 confirmed legitimate) through Muninn's automatic scan. This is the scan that runs on every page you visit without any action on your part. On its own, the automatic scan correctly identified 238 of the 254 phishing sites and only incorrectly flagged 6 legitimate pages. [...] The tradeoff is that it flagged all 9 of the legitimate sites in our dataset as suspicious, ... Am I missing something or is that a 66%/100% False Positive Rate on legitimate Sites? If GSB would have that ratio, it would be absolute unusable.. So comparing these two is absolutely wrong...

mrexcess 7mo ago

These statistics would be a lot better if they were compared directly to the same measurements taken from dedicated cloud SWGs/SSEs like Zscaler. My somewhat subjective sense is that the whole industry is in a bit of a rough patch, the miss rate seems to be noticeably climbing all across the board.

pothamk 7mo ago

[flagged]

tadfisher 7mo ago
If you are not a bot, I suggest changing your voice so that you are distinguishable from one. You're not wrong, just like you weren't wrong about "one thing that trips people up about asyncio" yesterday, but I noticed the slop-speak immediately. I'm sure others have as well.
- niwtsol 7mo ago
  Multiple comments that start w/ "what's interesting about" by this user and very similar formatting kind of answers that question on human vs bot. Weird internet we live in these days.

timnetworks 7mo ago

The most dangerous links recently have been from sharepoint.com, dropbox.com, etc. and nobody is going to block those.

deleted 7mo ago
[deleted]

varispeed 7mo ago

When Google will remove scams, phishing and other nonsense from their advertising? Especially the scareware stuff, where AI videos say someone might be listened to / hacked and here is the software that will help block it / find it whatnot. Then they collect personal data.

hedora 7mo ago

So, the false negative rate was 84%, but what was the false positive rate? They have a table "AUTOMATIC SCAN RESULTS (263 URLS)" that sort of presents this information. Of the 9 sites that were negatives, they say they incorrectly flagged 6 as phishing. With a false positive rate of 66%, it's not surprising they were able to drive down their false negative rate. Also, the test set of 254 phishing sites with 9 legitimate ones is a strange choice. (Or maybe they need to work on how they present data in tables; tl;dr the supporting text.)

decimalenough 7mo ago
The false positive rate was 66% for "automatic scan" and 100% (!) for "deep scan". In other words, you can get these numbers if your deep scan filter is isSuspicious() { return true; }.
- antonvs 7mo ago
  Brb, applying for YC funding for my new AI-based phishing detection system. (‘return true’ is just a very optimized neural network after all!)
- dataflow 7mo ago
  I think there might be a confusion here? The 100% seems like the true positive rate (correct detection), not the false positive rate?
  - decimalenough 7mo ago
    Nope, 9 of 9 legit sites were incorrectly flagged: > The tradeoff is that it flagged all 9 of the legitimate sites in our dataset as suspicious
    - dataflow 7mo ago
      Sorry, I think I had my wires crossed somewhere. Yeah, I see now. That's crazy/hilarious.

obblekk 7mo ago

Maybe I’m an outlier but I’d rather this than accidentally block legit sites. Otherwise this becomes just another tool for Google to wall in the subset of the internet they like.

caaqil 7mo ago

Yeah, maybe let's change the title to remove that 84% rate. It's meaningless because it's just 254 websites, given the scale of what Google Safe Browsing deals with. How is this serious? This is a marketing slop. If the title isn't enough indicator, the ending should be: > If you're interested in trying Muninn, it's available as a Chrome extension. We're in an early phase and would genuinely appreciate feedback from anyone willing to give it a shot. And if you run across phishing in the wild, consider submitting it to Yggdrasil so the data can help protect others.

bethekidyouwant 7mo ago

Their example is really dumb. Eventually, you get a fake Microsoft login page, but they clip out the address bar which clearly isn’t a Microsoft address so your auto complete password isn’t going to be put into the form and you’d have to be pretty dumb to type it in my hand or even to know your Microsoft password, it should be some random thing generated by Safari or whatever your password manager is. Not to mention two factor authentication.

eks391 7mo ago
Most people are "really dumb" by your standards then. Not only are most people not going to check the URL, but many people don't know how password managers work, and the only reason they use the browser password manager is because it is on by default, and it is saving their collection of 3 reused passwords they manually type at each site when it doesn't auto populate.

dsr_ 7mo ago

Almost all email phishing attempts we receive come from GMail.

kemotep 7mo ago

Default deny and only permitting what you explicitly allow stops 90% of this in a corporate environment. You don’t just leave all your ports open on the firewall and only close the ones exploited. You default deny and only allow the bare minimum you need in.

virken 7mo ago

I'm all for stopping phishing - and the tool sounds great - but I have to say the Web Store Extension listing is very concerning - even with a new company/offering - there's only 4 users - and 1 rating (a 5 of course) - I'd like to try - but seems phishy :-(

virken 7mo ago

I'm all for stopping phishing - and the tool sounds great - but the Web Store Extension listing is very concerning - even with a new company/offering - there's only 4 users - and 1 rating (a 5 of course) - I'd like to try - but seems phishy :-(

thayne 7mo ago

It would be interesting to see how many of the sites safe browsing does block are false positives.

nickphx 7mo ago

"If you're interested in trying Muninn, it's available as a Chrome extension. We're in an early phase " Domain is less than 4 months old.. Software is "early phase".. Already making misleading marketing claims of usefulness..

kopollo 7mo ago

Let me give you a simple detection algorithm. Apply OCR to the screenshot because they often use logos. Also, parse the text from the HTML and compare it to the URL. You can catch a lot of spam this way.You can also examine many parameters in the js html code.

throawayonthe 7mo ago

> ...full dataset of 263 URLs (254 phishing, 9 confirmed legitimate) > ... automatic scan is optimized for precision (keeping false alarms low... really? > When we ran the full dataset through the deep scan, ... it flagged all 9 of the legitimate sites in our dataset as suspicious lol

hulitu 7mo ago

The purpose of "Safe Browsing" is to send your URLs to Google.

nickphx 7mo ago

So I tested out the extension.. First the extension spammed me with "login required".. So I click the notification to be taken to a login page.. Great? Now I have to create an account and verify a link.. Now I can test how great this is against a "fresh" facebook phishing page being actively promoted via Facebook Ads.. hxxps://r7ouhcqzdgae76-fsc0fydmbecefrap.z03.azurefd.net/new2/?utm_medium=paid&utm_source=fb&utm_id=6900429311725&utm_content=6900429312725&utm_t erm=6900429314125&utm_campaign=6900429311725 The "extension" did a "scan". {"url":"https://r7ouhcqzdgae76-fsc0fydmbecefrap.z03.azurefd.net/new2/","mode":"light https://r7ouhcqzdgae76-fsc0fydmbecefrap.z03.azurefd.net/new2..."} response: {"classification":"clean"} great work? If I click "Deep scan".. I see a screenshot blob being sent over.. response: { "classification": "phish", "reasons": [ "Our system has previously flagged this webpage as malicious." ] } So if the site were already flagged, why does the "light" scan not show that?

luckman212 7mo ago

Is it possible to disable Safe Browsing AND also not have to manually click to confirm that "yes, I actually do want to keep the file I just downloaded, thank you" every. single. time.

burnte 7mo ago
No. I keep checking every year.
yogorenapan 7mo ago
The answer you probably don't want: Linux + Firefox. Or just copy link and wget.

RS-232 7mo ago

This seems like proof to me that Safe Browsing is about surveillance, not protection.

Anamon 7mo ago

Forget about checking Safe Browsing. Try reporting phishing sites to Google if you really want a reason to hate them. I report all (manually verified) phishing sites I get spam for to Safe Browsing, then automatically check the listing status daily, and re-report every week if it's still not blacklisted. Usually, takedown by hosters is faster. Phishing sites that stay up long enough often go for 3-4 months before Google finally deems it apt to block them. I think the entire Safe Browsing team consists of one overworked, underpaid intern. Compare this to Microsoft Defender. The time it takes for a site I report there to get blocked is usually measured in hours, not weeks. This is one thing MS actually seems to be running decently.