6 ms·
Nice catch. A long time ago the services on the backend were killed by a special URL. And someone found it, and it wasn't filtered by the front end. And of cour
by ChuckMcM 10y ago
Nice catch. A long time ago the services on the backend were killed by a special URL. And someone found it, and it wasn't filtered by the front end. And of course someone tried to use it, but it never returns since it kills the service, but their client retried ... it was a lot of "what the heck is happening" going on until SRE figured it out and then they immediately patched the front end and the anomalies stopped. It is too bad the person who caused it didn't file for a bug bounty like this person did, they probably would have had something to show for their efforts besides "hey look at this funny thing you can do, oh wait it doesn't do it any more."
- carvalho 10y agoOn Quora someone asked what the longest search query time was. I was able to craft a query that took multiple seconds to complete. It used wildcards and undocumented iteration allowing one to stuff thausands of queries into a single query. Turns out it is someone's job to measure result response times, and he/she came into the thread to kindly ask us to stop messing up their statistics.
- logicallee 10y agoIt's hard to believe, but years ago, back when Google had what was called "stop words" (like 'the', that it ordinarily ignored) I was able to make Google perform a search that took over 30 seconds. The reason stop words take such a long time is that millions of sites have words like "the" on them, so doing a join on all those simply takes a long time. My method to find a long string consisting entirely of stop words, was to just download a project gutenberg of the complete works of shakespeare, and find the longest string consisting of just stop words in there, then search for it as a literal quote. The longest one I found was: "From what it is to a". Let me see how long Google takes to do it now :) 2.04 seconds! Nice :) - http://i.imgur.com/IhPTpr6.png http://i.imgur.com/IhPTpr6.png that took 30+ seconds 'back in the day'.
- sova 10y agoIs it really technically correct to say that Google was performing web-wide joins on data? Isn't it all about clever indexing?
- logicallee 10y agoThere's nothing to index. How could it have found my Shakespeare quote via an index? It consisted entirely of words 'from what it is to a' but produced only the Shakespeare quote. I don't see how it could have indexed anything.... it must have done a join. (Which makes sense given the 30+ seconds I had to sit and wait before it returned its answer, while also reporting the time it took to produce it. What else could it have been doing?) By the way I believe I wanted to know whether it would return the Shakespeare quote at all. If you mean that it might have cached the results of the query, I doubt anyone else queried that exact phrase, other than me.
- Benjammer 10y ago>There's nothing to index Huh? What do you mean? Google indexes HTML web page content from the entire public internet using web crawlers...
- logicallee 10y agoI'm confused. By "clever indexing" I thought they meant, in the database sense of the word. The reason my search took 30 seconds is because it started by getting a list of every site with "from" on it, every site with "what" on it, and so on, intereseecting them all. That's how it ended up finding my quote. how else do you think it did it? ----- edit: to find the string "from what it is to a" which occurs only hidden in the middle of shaespeare's texts -- what do you think they do? In my opinion they combine the list of sites that have every word - starting with the least common ones. It's easier if you search for something that has a few uncommon words. Then you start with a small list, and have to combine it with other small lists. When every word in the phrase has billions of sites (there are billions of pages that have the word "to" on them, same for "from", "what", "it", "is", "a"), you have to combine them all. Then you have to do a string search within the resulting set, since I put it in quotation marks. There is no easy strategy. Hence the long search time. what else could they be doing?
- vochinch 10y agoNice! Do you still have the link to the Quora question or an example of the query?
- rshaban 10y agolinked in https://news.ycombinator.com/item?id=14372977 https://news.ycombinator.com/item?id=14372977 https://www.quora.com/What-is-the-slowest-Google-query https://www.quora.com/What-is-the-slowest-Google-query
- dsacco 10y agoDo you have a link to that? I'd be interested in reading it his response and I can't see it by searching Quora.
- ahiknsr 10y agohttps://www.quora.com/What-is-the-slowest-Google-query/answer/Kevin-Lacker/comment/536273 https://www.quora.com/What-is-the-slowest-Google-query/answe...
- ClassyJacket 10y ago>and he/she came into the thread to kindly ask us to stop messing up their statistics. Shouldn't someone with a job in statistics know how to account for outliers?
- annnnd 10y agoAnd also, wouldn't they be interested in getting those queries so they could either fix their performance or block them?
- developer2 10y agoThe query was already published on Quora. I assume that Google did patch the issue. They were simply requesting that it not be posted in a public forum, resulting in the potential for denial of service attacks. It wouldn't have been a statistician making a fuss about their pretty graphs being ruined. The problem was the very real performance impact the queries were having on the service, as thousands of visitors copy/pasted from Quora to see for themselves. This is why companies like Google have bounties for such things. "Please submit bugs and performance issues privately so we can patch them before you disclose the details publicly and hurt our services - we'll even pay you for your discretion!"
- developer2 10y agoThis particular issue was posted on Quora, where anyone could pick it up and participate in what is essentially a denial of service attack (whether or not performed intentionally). It wasn't submitted as a private bug report to Google so they could fix the issue. It was spread in a public forum. I think it's fair for Google to politely ask "a few of your own tests to validate an issue you will submit as a bug report is fine, but please don't disclose to the public until we patch it." When you operate at the scale of Google, everything is expected to be airtight; outliers should not be possible. It wouldn't surprise me if their monitoring systems are built without the ability to "massage" (ie: manipulate) statistics, as it is a terrible practice. I don't think a statistician who relies on ignoring outliers would last long working for Google. They're not doing their job if the only thing they care about is silencing warnings to make pretty graphs that falsely show everything is running smoothly. Their job is to work with the truth - not manufacture little white lies to appease management.
- hoschicz 10y agoThis is his answer: "I work on search at Google, and I have to say, very clever answers! Now, please stop. :-p" I don't think that he did it because it's his job to stop random people on the Internet from running slow queries. I think she was just surprised how creative people are and found it funny.
- kw71 10y agoSomething to show for effort. This matters a bit. One of my young-adulthood colleagues went on to be an early googler who is influential in relevant policies. We built our relationship sharing bugs and analysis techniques. Quite a few years ago some scoundrels whose trust I gained proudly showed me how they were using youtube links to drop malware. Since my old mate worked there, I mentioned it and they were quite interested. We hadn't shared anything in a while, both of us demonstrating loyalty to our employers and not talking about work details. I said that it would be really cool to have a one dollar check from Google for a bug report. I probably offered to send something cool from my workplace too. They said, "We don't pay for bugs" Fifty cents? "We don't pay for bugs!" I felt like I was simply after a piece of paper and the evildoers were a mildly useful source, but I could easily do without them and the souvenir would have been treasured. I was unreasonably miffed that I couldn't get that piece of paper, though. So I reviewed the links I'd collected and passed some general information but withheld details that would be obviously unique to these attackers. They expressed disappointment with me the next time we spoke. It turns out that what I gave wasn't specific enough to easily identify the lame cross site exploit, despite my actual intent to lead them to the bug. Interesting they have a bounty program now.