5 ms·

The topic here says "quantitatively". Does anyone have any actual statistics or quantitative data on the quality of Google search results? All of these thread

by wffurr 5y ago

The topic here says "quantitatively". Does anyone have any actual statistics or quantitative data on the quality of Google search results? All of these threads devolve into anecdotes and reminisces about the "good old days" and complaining about Pinterest. None of which is in the least quantitative. I'd be interested to see some actual data or research on the subject, if it exists. Or maybe it's not Google that's gotten worse but the web itself? Again, quantitative results, please, not anecdata.

Reuzel 5y ago
> Does anyone have any actual statistics or quantitative data on the quality of Google search results? Google has. They use this data expertly to improve search. Common sense and technological advancement tells us that, quantitatively, Google search has become better year over year, for all their relevant metrics/cost functions. And likely, exactly because it has become better for all its users in aggregate, it has to become a bit worse for a certain group of power users. There, we can only rely on anecdotes and personal experience, but these tell us it actually has gotten worse. Similarly, the web can become both worse and better. The really useful articles today are better researched, multi-modal, solid web of links, internet-first. Spam has also evolved. And "top 10 ways to do X"-McContent outranks better articles, because that is what the majority of Google users wants to see and clicks on. They truly have a better experience, while others' experiences suffer. It depends on what you measure.
- willhinsa 5y ago
  > Common sense and technological advancement tells us that, quantitatively, Google search has become better year over year, for all their relevant metrics/cost functions. lmfao. so you're telling me "quantitatively" that google search results have gotten better, without citing any data at all, but with an appeal to common sense and "technological advancement"? what if i told you that search is an adversarial problem, and that it's possible for google's tech to be getting better slower than the aggregate tech power used to game google search is getting better? is this not a patently obvious possibility? it's not some kind of gotcha impossibility for google's tech to get much worse over time, even if they weren't hamstringing themselves by lots and lots of user-hostile changes which benefit google's interests rather than their users.
  - Reuzel 5y ago
    > lmfao. so you're telling me "quantitatively" that google search results have gotten better, without citing any data at all, but with an appeal to common sense and "technological advancement"? Yes. If that sounds so unacceptable or strange to you, I suggest you try it. Works really well when reasoning about unavailable data, or researching a field with slow peer-review process. > what if i told you that search is an adversarial problem Then I get an adversarial reaction and a downvote from you. > it's possible for google's tech to be getting better slower than the aggregate tech power used to game google search is getting better? is this not a patently obvious possibility? Yes, that's plausible. Should be measurable quantitively too. Can you cite some data on this? :) We could compare to the available data on the quality of (HTML) e-mail spam filtering over the years, which all have kept up. Like pg said: Spam is solved, when skilled spammers start creating content which does not look, feel, or talk like spam. So webcontent-spammers still are on the first 2 pages of Google, but with content not classifiable as spam/content farm. > which benefit google's interests rather than their users. One of Google's interest is their user. But perhaps not the type of user you are. Studies have shown the value that the tech of Google is delivering its users per year. This value was in the thousandths, and this value has risen. Meanwhile, Google makes about tens of dollars per user per year, less for technical users which don't click ads or block these. > it's not some kind of gotcha impossibility for google's tech to get much worse over time It really is, no way to mince it. Google search is funded by Google ad tech. Google ad tech has improved ML by a ton. To say Google tech is getting worse, is to totally overlook deep learning revolution, word2vec, transformers, BERT, etc. etc. etc. To state that, is to reveal the truth that you are ignorant of major technological advances in the past decade, only looking at the issue from the viewpoint of a single atypical Google-search user. What would you even do with quantitive SEQ data? Do you really think it is possible that Google runs an implementation test of a new ranking model, and deploys it, while all measurements, human labeling, and user tests show it is doing worse? Of course not! Only if you think you are smarter, could you think that Google search changed and has gotten worse. If Google does ML like the rest of the industry, all model changes move up their designed levers, or these changes are not committed. So if Google was unable to improve search, then Google would have looked exactly like 2008 Google. The fact that it does not, shows either you or Google is wrong. If I had to make a bet...
- bsanr2 5y ago
  I don't think this is necessarily true. Your standard desktop computer has gotten much easier for a casual user to navigate over time, but not any less robust for power users. This is because powerful tools for customization and building are still exposed to power users. Google has slowly stripped away many of these tools. One has to wonder why, and my guess would be that it's because they would expose either the unethical ways Google deals with results, or the failure of its search model.
- hnaccount141 5y ago
  This presumes that the metrics they optimize for are intended to represent usefulness to actual users and not, say, ad revenue. Even if they do intend to optimize for usefulness, this doesn't mean that they have metrics that accurately represent that. I also think you're underestimating average users. Anecdotally I've heard my parents complain repeatedly about the incoherent, auto-generated, affiliate link spam that plagues product searches.
  - Reuzel 5y ago
    > This presumes that the metrics they optimize for are intended to represent usefulness to actual users and not, say, ad revenue. Even if they do intend to optimize for usefulness, this doesn't mean that they have metrics that accurately represent that. They have multiple levers, of which user search quality is a big set. There is always trade-offs and a balance that must be found, which aligns with company vision and strategy. Having these levers allows business-decision makers to direct focus top-down (on a certain set of users, on producing great ad numbers, etc.). It is clearly hard and important to design these levels and find the right balance, given a rapidly changing company and user-base. So a lot of expertise and power is invested to measure the right things, and to find the right balance (an incorrect/risky balance should also be adjustable with other levers). So for me: either Google is trying really hard, but essentially failing. Or they have the best of the world, with all the right context, designing these levers. While hard and sometimes wrong, I do not expect to contribute anything which may improve their lever settings. If someone does know, Google would like to hire them. So while true, that accurately measuring things with proxies, is really hard, and sometimes done wrong at companies. I do not think Google gets this wrong, or at least, gets this to be the best of breed. If their metrics still cause long-term search engine quality loss, would show them to not know what they are doing. I think they do know very well, better than me at least. I would agree too that the balance of levers right now is in line with Google's strong market position. Search engine quality could take a small hit, if justified with extra adsense income. But when search engine quality noticeably start going down, then all other metrics will suffer. You should have teams with sole focus on improving quality. Other teams will have to realize that favoring their lever over the search-engine-quality lever must lead to worse outcomes for Google in general. About product searches, I myself was not able to do this satisfactory 15 years back. It improved. But need to stop viewing things as a single lever, a single metric. To say search has become "worse" in general, is to exactly fall into the trap of not accurately measuring and losing too much nuance/details for competing objectives.
mda 5y ago
My thoughts exactly, until we have actual dat but not "feelings" this whole argument is moot.
necovek 5y ago
To be able to quantitatively compare Google-of-old with Google-of-today on the web content today, we'd have to have access to both. And we'd have to assume that Google-of-old's algorithms have not been figured-out and abused by the content that's there today. And certainly the content has changed: I wonder if you can look into archive.org by-date and run a search engine on it for comparison (I'd like to test if proportion of commercial content has grown up). Without access to both Googles, the best you can do is compare across different search engines: special-cased ones like search.marginalia.nu can net you a quantitative feel for what exists out there that's less likely to be content marketing, but I am not sure if you can figure out where those pages rank in Google search results for the same terms programmatically? You can also prepare for the future: record some data today, and compare in 10 years time.
dehrmann 5y ago
It'd be cool to have a tool that runs locally on my Firefox history and builds metrics like how often I have to refine my search query.
corny 5y ago
"Quantitative" doesn't mean "empirical". I read the question to mean has google search returned _fewer_ results. Not fewer good results, but fewer results full stop. Perhaps the asker made a typo and meant "qualitatively".
skinkestek 5y ago
> Does anyone have any actual statistics or quantitative data on the quality of Google search results? It is pretty quantitative yes, but it happened at different times for different people. But it used to be that when you searched for something you got pages containing the thing you searched for, and if you couldn't find it at first then you could do a search on the page and find out some enterprising scammer had included your keywords in white text on white background. Today Google and Bing has teamed up with the scammers so they don't need to use such hacks anymore. Google and Bing will include the results even if they don't contain said keywords at all. Uncharitable? Yes. Do I hope Google and Bing engineers read this and fix it or do I hope some Russian enterprise launch a better engine? I actually hope Google find back to its roots! I don't hate you guys but you really don't make it easy for us in between using all the oxygen in the room and annoying me all the time with useless time wasting results.

by wffurr 5y ago

Reuzel 5y ago

> Does anyone have any actual statistics or quantitative data on the quality of Google search results? Google has. They use this data expertly to improve search. Common sense and technological advancement tells us that, quantitatively, Google search has become better year over year, for all their relevant metrics/cost functions. And likely, exactly because it has become better for all its users in aggregate, it has to become a bit worse for a certain group of power users. There, we can only rely on anecdotes and personal experience, but these tell us it actually has gotten worse. Similarly, the web can become both worse and better. The really useful articles today are better researched, multi-modal, solid web of links, internet-first. Spam has also evolved. And "top 10 ways to do X"-McContent outranks better articles, because that is what the majority of Google users wants to see and clicks on. They truly have a better experience, while others' experiences suffer. It depends on what you measure.

willhinsa 5y ago
> Common sense and technological advancement tells us that, quantitatively, Google search has become better year over year, for all their relevant metrics/cost functions. lmfao. so you're telling me "quantitatively" that google search results have gotten better, without citing any data at all, but with an appeal to common sense and "technological advancement"? what if i told you that search is an adversarial problem, and that it's possible for google's tech to be getting better slower than the aggregate tech power used to game google search is getting better? is this not a patently obvious possibility? it's not some kind of gotcha impossibility for google's tech to get much worse over time, even if they weren't hamstringing themselves by lots and lots of user-hostile changes which benefit google's interests rather than their users.
- Reuzel 5y ago
  > lmfao. so you're telling me "quantitatively" that google search results have gotten better, without citing any data at all, but with an appeal to common sense and "technological advancement"? Yes. If that sounds so unacceptable or strange to you, I suggest you try it. Works really well when reasoning about unavailable data, or researching a field with slow peer-review process. > what if i told you that search is an adversarial problem Then I get an adversarial reaction and a downvote from you. > it's possible for google's tech to be getting better slower than the aggregate tech power used to game google search is getting better? is this not a patently obvious possibility? Yes, that's plausible. Should be measurable quantitively too. Can you cite some data on this? :) We could compare to the available data on the quality of (HTML) e-mail spam filtering over the years, which all have kept up. Like pg said: Spam is solved, when skilled spammers start creating content which does not look, feel, or talk like spam. So webcontent-spammers still are on the first 2 pages of Google, but with content not classifiable as spam/content farm. > which benefit google's interests rather than their users. One of Google's interest is their user. But perhaps not the type of user you are. Studies have shown the value that the tech of Google is delivering its users per year. This value was in the thousandths, and this value has risen. Meanwhile, Google makes about tens of dollars per user per year, less for technical users which don't click ads or block these. > it's not some kind of gotcha impossibility for google's tech to get much worse over time It really is, no way to mince it. Google search is funded by Google ad tech. Google ad tech has improved ML by a ton. To say Google tech is getting worse, is to totally overlook deep learning revolution, word2vec, transformers, BERT, etc. etc. etc. To state that, is to reveal the truth that you are ignorant of major technological advances in the past decade, only looking at the issue from the viewpoint of a single atypical Google-search user. What would you even do with quantitive SEQ data? Do you really think it is possible that Google runs an implementation test of a new ranking model, and deploys it, while all measurements, human labeling, and user tests show it is doing worse? Of course not! Only if you think you are smarter, could you think that Google search changed and has gotten worse. If Google does ML like the rest of the industry, all model changes move up their designed levers, or these changes are not committed. So if Google was unable to improve search, then Google would have looked exactly like 2008 Google. The fact that it does not, shows either you or Google is wrong. If I had to make a bet...
bsanr2 5y ago
I don't think this is necessarily true. Your standard desktop computer has gotten much easier for a casual user to navigate over time, but not any less robust for power users. This is because powerful tools for customization and building are still exposed to power users. Google has slowly stripped away many of these tools. One has to wonder why, and my guess would be that it's because they would expose either the unethical ways Google deals with results, or the failure of its search model.
hnaccount141 5y ago
This presumes that the metrics they optimize for are intended to represent usefulness to actual users and not, say, ad revenue. Even if they do intend to optimize for usefulness, this doesn't mean that they have metrics that accurately represent that. I also think you're underestimating average users. Anecdotally I've heard my parents complain repeatedly about the incoherent, auto-generated, affiliate link spam that plagues product searches.
- Reuzel 5y ago
  > This presumes that the metrics they optimize for are intended to represent usefulness to actual users and not, say, ad revenue. Even if they do intend to optimize for usefulness, this doesn't mean that they have metrics that accurately represent that. They have multiple levers, of which user search quality is a big set. There is always trade-offs and a balance that must be found, which aligns with company vision and strategy. Having these levers allows business-decision makers to direct focus top-down (on a certain set of users, on producing great ad numbers, etc.). It is clearly hard and important to design these levels and find the right balance, given a rapidly changing company and user-base. So a lot of expertise and power is invested to measure the right things, and to find the right balance (an incorrect/risky balance should also be adjustable with other levers). So for me: either Google is trying really hard, but essentially failing. Or they have the best of the world, with all the right context, designing these levers. While hard and sometimes wrong, I do not expect to contribute anything which may improve their lever settings. If someone does know, Google would like to hire them. So while true, that accurately measuring things with proxies, is really hard, and sometimes done wrong at companies. I do not think Google gets this wrong, or at least, gets this to be the best of breed. If their metrics still cause long-term search engine quality loss, would show them to not know what they are doing. I think they do know very well, better than me at least. I would agree too that the balance of levers right now is in line with Google's strong market position. Search engine quality could take a small hit, if justified with extra adsense income. But when search engine quality noticeably start going down, then all other metrics will suffer. You should have teams with sole focus on improving quality. Other teams will have to realize that favoring their lever over the search-engine-quality lever must lead to worse outcomes for Google in general. About product searches, I myself was not able to do this satisfactory 15 years back. It improved. But need to stop viewing things as a single lever, a single metric. To say search has become "worse" in general, is to exactly fall into the trap of not accurately measuring and losing too much nuance/details for competing objectives.

mda 5y ago

My thoughts exactly, until we have actual dat but not "feelings" this whole argument is moot.

necovek 5y ago

To be able to quantitatively compare Google-of-old with Google-of-today on the web content today, we'd have to have access to both. And we'd have to assume that Google-of-old's algorithms have not been figured-out and abused by the content that's there today. And certainly the content has changed: I wonder if you can look into archive.org by-date and run a search engine on it for comparison (I'd like to test if proportion of commercial content has grown up). Without access to both Googles, the best you can do is compare across different search engines: special-cased ones like search.marginalia.nu can net you a quantitative feel for what exists out there that's less likely to be content marketing, but I am not sure if you can figure out where those pages rank in Google search results for the same terms programmatically? You can also prepare for the future: record some data today, and compare in 10 years time.

dehrmann 5y ago

It'd be cool to have a tool that runs locally on my Firefox history and builds metrics like how often I have to refine my search query.

corny 5y ago

"Quantitative" doesn't mean "empirical". I read the question to mean has google search returned _fewer_ results. Not fewer good results, but fewer results full stop. Perhaps the asker made a typo and meant "qualitatively".

skinkestek 5y ago

> Does anyone have any actual statistics or quantitative data on the quality of Google search results? It is pretty quantitative yes, but it happened at different times for different people. But it used to be that when you searched for something you got pages containing the thing you searched for, and if you couldn't find it at first then you could do a search on the page and find out some enterprising scammer had included your keywords in white text on white background. Today Google and Bing has teamed up with the scammers so they don't need to use such hacks anymore. Google and Bing will include the results even if they don't contain said keywords at all. Uncharitable? Yes. Do I hope Google and Bing engineers read this and fix it or do I hope some Russian enterprise launch a better engine? I actually hope Google find back to its roots! I don't hate you guys but you really don't make it easy for us in between using all the oxygen in the room and annoying me all the time with useless time wasting results.