3 ms·
Yes, Larry Page’s first crawler was in Java, and Scott Hassan rewrote it in Python. From https://www.vanityfair.com/news/2018/07/valley-of-genius-excerpt-googl
by secure 8y ago
Yes, Larry Page’s first crawler was in Java, and Scott Hassan rewrote it in Python.
From https://www.vanityfair.com/news/2018/07/valley-of-genius-excerpt-google https://www.vanityfair.com/news/2018/07/valley-of-genius-exc...:
> Scott Hassan: In the fall of ’95, for some reason, I started hanging out with Larry in his office. . . . At the time, Larry was trying to download a hundred pages simultaneously. And I was fixing some of the bugs that he was having with Java itself, and this went on for weeks, if not months. And I remember thinking, Wow, this is insane!, because I was spending a lot of time fixing this underlying tool. And so one weekend, I just took all his code, I took his whole entire thing, and threw it all out, and rewrote the thing that he’s been working on for months very quickly—over a weekend—because I was just sick and tired of it. I knew I could get the thing working if I used a language I knew very well, called Python. I wrote it in such a way that it could download 32,000 pages simultaneously. So Larry went from barely downloading a 100, to doing 32,000 [pages] simultaneously on a single machine.
- cutler 8y ago32,000 pages simultaneously - in 1995? Async has only recently been added to Python, no?
- boyter 8y agoUsing threads maybe? Id be curious to see how you would achieve that in Python easily back in 1995.
- mathgenius 8y agoUsing select: https://en.wikipedia.org/wiki/Select_(Unix) https://en.wikipedia.org/wiki/Select_(Unix)
- gcb0 8y agois that a fact or are you guessing?
- adamnemecek 8y agoselect doesnt scale well.
- scandox 8y agoDoesn't select have a maximum of 1024 file descriptors it can handle at any one time? Or some such?
- __sr__ 8y agoBy default, yes. But it can be increased. See limit(1) and ulimit(1).