4 ms·

Wasn't the first Google scraper written in Java? I saw an old Usenet post by Page from about 1997 asking about user agents in Java. Perhaps that was the second

Wasn't the first Google scraper written in Java? I saw an old Usenet post by Page from about 1997 asking about user agents in Java. Perhaps that was the second scraper.

adamnemecek 8y ago
> Perhaps that was the second scraper. I believe that is the case yes. However correct me if m wrong.
- secure 8y ago
  Yes, Larry Page’s first crawler was in Java, and Scott Hassan rewrote it in Python. From https://www.vanityfair.com/news/2018/07/valley-of-genius-excerpt-google https://www.vanityfair.com/news/2018/07/valley-of-genius-exc...: > Scott Hassan: In the fall of ’95, for some reason, I started hanging out with Larry in his office. . . . At the time, Larry was trying to download a hundred pages simultaneously. And I was fixing some of the bugs that he was having with Java itself, and this went on for weeks, if not months. And I remember thinking, Wow, this is insane!, because I was spending a lot of time fixing this underlying tool. And so one weekend, I just took all his code, I took his whole entire thing, and threw it all out, and rewrote the thing that he’s been working on for months very quickly—over a weekend—because I was just sick and tired of it. I knew I could get the thing working if I used a language I knew very well, called Python. I wrote it in such a way that it could download 32,000 pages simultaneously. So Larry went from barely downloading a 100, to doing 32,000 [pages] simultaneously on a single machine.
  - cutler 8y ago
    32,000 pages simultaneously - in 1995? Async has only recently been added to Python, no?
    - boyter 8y ago
      Using threads maybe? Id be curious to see how you would achieve that in Python easily back in 1995.
    - mathgenius 8y ago
      Using select: https://en.wikipedia.org/wiki/Select_(Unix) https://en.wikipedia.org/wiki/Select_(Unix)
      - gcb0 8y ago
        is that a fact or are you guessing?
      - adamnemecek 8y ago
        select doesnt scale well.
      - scandox 8y ago
        Doesn't select have a maximum of 1024 file descriptors it can handle at any one time? Or some such?
        __sr__ 8y ago
        By default, yes. But it can be increased. See limit(1) and ulimit(1).

Wasn't the first Google scraper written in Java? I saw an old Usenet post by Page from about 1997 asking about user agents in Java. Perhaps that was the second scraper.

adamnemecek 8y ago

> Perhaps that was the second scraper. I believe that is the case yes. However correct me if m wrong.

secure 8y ago
Yes, Larry Page’s first crawler was in Java, and Scott Hassan rewrote it in Python. From https://www.vanityfair.com/news/2018/07/valley-of-genius-excerpt-google https://www.vanityfair.com/news/2018/07/valley-of-genius-exc...: > Scott Hassan: In the fall of ’95, for some reason, I started hanging out with Larry in his office. . . . At the time, Larry was trying to download a hundred pages simultaneously. And I was fixing some of the bugs that he was having with Java itself, and this went on for weeks, if not months. And I remember thinking, Wow, this is insane!, because I was spending a lot of time fixing this underlying tool. And so one weekend, I just took all his code, I took his whole entire thing, and threw it all out, and rewrote the thing that he’s been working on for months very quickly—over a weekend—because I was just sick and tired of it. I knew I could get the thing working if I used a language I knew very well, called Python. I wrote it in such a way that it could download 32,000 pages simultaneously. So Larry went from barely downloading a 100, to doing 32,000 [pages] simultaneously on a single machine.
- cutler 8y ago
  32,000 pages simultaneously - in 1995? Async has only recently been added to Python, no?
  - boyter 8y ago
    Using threads maybe? Id be curious to see how you would achieve that in Python easily back in 1995.
  - mathgenius 8y ago
    Using select: https://en.wikipedia.org/wiki/Select_(Unix) https://en.wikipedia.org/wiki/Select_(Unix)
    - gcb0 8y ago
      is that a fact or are you guessing?
    - adamnemecek 8y ago
      select doesnt scale well.
    - scandox 8y ago
      Doesn't select have a maximum of 1024 file descriptors it can handle at any one time? Or some such?
      - __sr__ 8y ago
        By default, yes. But it can be increased. See limit(1) and ulimit(1).