5 ms·

I needed some real-world MS Word & Excel documents years ago to test some parsing code against. So I started crawling Google results for 'filetype:.doc .xls' st

by roel_v 11y ago

I needed some real-world MS Word & Excel documents years ago to test some parsing code against. So I started crawling Google results for 'filetype:.doc .xls' style queries. Left it running for a weekend, then the whole Monday was wasted as I was sucked into looking through the results - some stuff in there was certainly not meant for public disclosure...

Ntrails 11y ago
Great. There goes any chance of me actually doing work this week. ;p
elorant 11y ago
Pardon me for asking, but how did you crawl Google for a whole weekend. From what I know, Google blocks you if you request too many search queries in a short period of time. Did you use proxies?
- roel_v 11y ago
  Maybe now, I don't remember I had do anything sneaky back then (8-10 years ago)
  - wanderingstan 11y ago
    In 2004 I attempted to do some automatic crawling of Google for my masters thesis and was astonished to get an unfriendly server response saying it was disallowed and "don't even bother asking for an exception for research, it won't be granted." So at least 11 years ago it was blocked. (I didn't know about spoofing a user agent back then, so it might not have been as easy as that to get around it.)
frik 11y ago
Did you really crawl Google? That has to be a long time ago. But speaking about searching on Google as a user: Google's Advanced search used to be a great tool, until around 2007/08. For some reason it never received an upgrade and several things are broken or don't work any more or were removed (e.g. '+' which is now a keyword for Google+, the '"' does mean the same; e.g some filetypes are blocked, some show only a few results).
- DanBC 11y ago
  The original Google has gone forever. But then, so has the www that the original Google worked so well for. I suspect original Google would be horrible on today's web. I miss 1998 and I mourn for what could have been.
  - jl6 11y ago
    Is there any truth to my suspicion that the web of hyperlinks (on which the famed algorithm relied) is significantly weaker and reaches fewer corners these days? Certainly feels like content is migrating to the walled gardens and there are fewer and fewer personal websites injecting edges into the open graph.
    - teddyh 11y ago
      Last November I speculated why Google would let HTTP/2 get standardized without specifying the use of SRV records: “This is going to bite them big time in the end, because Google got large by indexing the Geocities-style web, where everybody did have their own web page on a very distributed set of web hosts. What Google is doing is only contributing to the centralization of the Web, the conversion of the Web into Facebook, which will, in turn, kill Google, since they then will have nothing to index. They sort of saw this coming, but their idea of a fix was Google+ – trying to make sure that they were the ones on top. I think they are still hoping for this, which is why they won’t allow a decentralized web by using SRV records in HTTP/2.” https://news.ycombinator.com/item?id=8550133 https://news.ycombinator.com/item?id=8550133
    - cmdrfred 11y ago
      It feels the same way to me. To a large percent of users the internet is Facebook rather than the largest compendium of human knowledge in existence, but lucky for us that use it for the latter reason that value of such a thing will always be evident.
- roel_v 11y ago
  Come to think of it, we moved offices 3 times since then, must've been 8-10 years ago. I don't think I had to do any special trickery, I spend only an afternoon or so writing and testing the code. I didn't realize such a thing would be impossible now - what a shame. I downloaded several gigabytes iirc - a big amount at the time.
iyn 11y ago
> some stuff in there was certainly not meant for public disclosure... Care to say more? I'm not asking about specifics (but won't mind reading about them either), I'm just curious about the type of information. Internal documents of corporations? .mil? Something else?
- tajen 11y ago
  Yes OP, make us laugh with ancient stories of the ".doc" web! You've said to much when you mentioned "some documents"...
- roel_v 11y ago
  Meh, nothing that would make the newspapers, not even good enough remembering details about 10 years later - sorry to get your hopes up :) Internal company meeting notes, reports on legal proceedings, I remember some correspondence on a divorce, checklists and procedures for business processes, test grades and student notes from schools, that sort of stuff. Excel sheets with budgets and timekeeping sheets, just typical office stuff - no .mil or at least nothing interesting, I would have remembered if it had been anything juicy. Maybe Google results didn't show those, don't know. It's not rocket science - go to Google.com, type "[some keyword] filetype:doc" (or pds, or xls, whatever), skip to page 10 or so, then start looking. I had to click just twice before I found (5 minutes ago) meeting minutes from some British council meeting, marked 'confidential and not for distribution', complete with names and dates/times. Go to your 'private files' folder on your machine (you know, where you store your job applications, that invitation you made once for your sister's birthday party, that sort of junk), look at the documents you'd least want somebody else to see, identify some keywords that are in those documents but not usually in others, use those as keywords in Google. Here's a fun one: "site:gallery.mailchimp.com filetype:pdf coupon". Nothing shocking, but I still don't think Mailchimp's customers expect their email list attachments to show up like this... (disclaimer: pure speculation, maybe these are meant to be publicly available, or maybe they're not even real customers and just test data, I read it for the articles, etc.)

by roel_v 11y ago

Ntrails 11y ago

Great. There goes any chance of me actually doing work this week. ;p

elorant 11y ago

Pardon me for asking, but how did you crawl Google for a whole weekend. From what I know, Google blocks you if you request too many search queries in a short period of time. Did you use proxies?

roel_v 11y ago
Maybe now, I don't remember I had do anything sneaky back then (8-10 years ago)
- wanderingstan 11y ago
  In 2004 I attempted to do some automatic crawling of Google for my masters thesis and was astonished to get an unfriendly server response saying it was disallowed and "don't even bother asking for an exception for research, it won't be granted." So at least 11 years ago it was blocked. (I didn't know about spoofing a user agent back then, so it might not have been as easy as that to get around it.)

frik 11y ago

Did you really crawl Google? That has to be a long time ago. But speaking about searching on Google as a user: Google's Advanced search used to be a great tool, until around 2007/08. For some reason it never received an upgrade and several things are broken or don't work any more or were removed (e.g. '+' which is now a keyword for Google+, the '"' does mean the same; e.g some filetypes are blocked, some show only a few results).

DanBC 11y ago
The original Google has gone forever. But then, so has the www that the original Google worked so well for. I suspect original Google would be horrible on today's web. I miss 1998 and I mourn for what could have been.
- jl6 11y ago
  Is there any truth to my suspicion that the web of hyperlinks (on which the famed algorithm relied) is significantly weaker and reaches fewer corners these days? Certainly feels like content is migrating to the walled gardens and there are fewer and fewer personal websites injecting edges into the open graph.
  - teddyh 11y ago
    Last November I speculated why Google would let HTTP/2 get standardized without specifying the use of SRV records: “This is going to bite them big time in the end, because Google got large by indexing the Geocities-style web, where everybody did have their own web page on a very distributed set of web hosts. What Google is doing is only contributing to the centralization of the Web, the conversion of the Web into Facebook, which will, in turn, kill Google, since they then will have nothing to index. They sort of saw this coming, but their idea of a fix was Google+ – trying to make sure that they were the ones on top. I think they are still hoping for this, which is why they won’t allow a decentralized web by using SRV records in HTTP/2.” https://news.ycombinator.com/item?id=8550133 https://news.ycombinator.com/item?id=8550133
  - cmdrfred 11y ago
    It feels the same way to me. To a large percent of users the internet is Facebook rather than the largest compendium of human knowledge in existence, but lucky for us that use it for the latter reason that value of such a thing will always be evident.
roel_v 11y ago
Come to think of it, we moved offices 3 times since then, must've been 8-10 years ago. I don't think I had to do any special trickery, I spend only an afternoon or so writing and testing the code. I didn't realize such a thing would be impossible now - what a shame. I downloaded several gigabytes iirc - a big amount at the time.

iyn 11y ago

> some stuff in there was certainly not meant for public disclosure... Care to say more? I'm not asking about specifics (but won't mind reading about them either), I'm just curious about the type of information. Internal documents of corporations? .mil? Something else?

tajen 11y ago
Yes OP, make us laugh with ancient stories of the ".doc" web! You've said to much when you mentioned "some documents"...
roel_v 11y ago
Meh, nothing that would make the newspapers, not even good enough remembering details about 10 years later - sorry to get your hopes up :) Internal company meeting notes, reports on legal proceedings, I remember some correspondence on a divorce, checklists and procedures for business processes, test grades and student notes from schools, that sort of stuff. Excel sheets with budgets and timekeeping sheets, just typical office stuff - no .mil or at least nothing interesting, I would have remembered if it had been anything juicy. Maybe Google results didn't show those, don't know. It's not rocket science - go to Google.com, type "[some keyword] filetype:doc" (or pds, or xls, whatever), skip to page 10 or so, then start looking. I had to click just twice before I found (5 minutes ago) meeting minutes from some British council meeting, marked 'confidential and not for distribution', complete with names and dates/times. Go to your 'private files' folder on your machine (you know, where you store your job applications, that invitation you made once for your sister's birthday party, that sort of junk), look at the documents you'd least want somebody else to see, identify some keywords that are in those documents but not usually in others, use those as keywords in Google. Here's a fun one: "site:gallery.mailchimp.com filetype:pdf coupon". Nothing shocking, but I still don't think Mailchimp's customers expect their email list attachments to show up like this... (disclaimer: pure speculation, maybe these are meant to be publicly available, or maybe they're not even real customers and just test data, I read it for the articles, etc.)