5 ms·

I am trying to be charitable to the Java implementation, but I am struggling to understand the complexity there. Java can iterate over files, it can iterate ove

I am trying to be charitable to the Java implementation, but I am struggling to understand the complexity there. Java can iterate over files, it can iterate over strings, the rules to permissively identify phone numbers are not great, but both implementations would suffer from that complexity. Unless it was full-on FactoryFactoryPPIExtractor<PhoneNumber> style coding.

Arnavion 3y ago
"find all phone numbers on their website" sounds like it had to act like a web crawler, not parse a bunch of files, so perhaps the complexity was because of that.
- seadan83 3y ago
  I like how y'all are thinking the complexity was in the web crawling. It was actually just a simple act of crawling over source files! Amazon was looking to remove their customer support phone numbers from all web pages (to save money, they wanted people to stop calling them; back in the early 2010s) I share the wonder that it took several days (perhaps that is the anecdote taking a life of its own, perhaps it was actually several hours. Or maybe it was was several days interspersed with on-call support, meetings, and other work running in parallel). Though.. the file parsing libraries back in the early 2010s in Java are somewhat basic, probably using Java 5. Which does make me wonder, how long would that exercise take in Java 8 or later compared to Java 5?
  - qohen 3y ago
    Steve Yegge discusses this in his "The Five Essential Phone-Screen Questions" blog post[0] from 2004 and states that his team-member (at Amazon) produced the list of phone-numbers in an hour: Here are some facts for you to ponder: 1. Our Contact Reduction team really did have exactly this problem in 2003. This isn't a made-up example. 2. Someone on our team produced the list within an hour, and the list supported more than just the 2 formats above. 3. About 25% to 35% of all software development engineer candidates, independent of experience level, cannot solve this problem, even given the entire interview hour and lots of hints. [0] https://sites.google.com/site/steveyegge2/five-essential-phone-screen-questions https://sites.google.com/site/steveyegge2/five-essential-pho...
- fbdab103 3y ago
  I was considering that, but web crawling via shell definitely crosses the line of things I would not want to attempt. Although, I guess you could outsource that complexity with `wget --mirror`, bringing you right back to file-crawling.
  - PaulHoule 3y ago
    My take is this. There is a very simple way to write a breadth-first webcrawler that works in stages and saves the frontier in an ordinary text file, this kind of webcrawler can be implemented in almost any programming language, particularly shell. That is, you write a script that loops over the urls in the frontier, fetches them, then writes new urls it find in the next frontier, you can use sort, uniq and such to manage the frontier. That simple breadth first webcrawler has the big advantage that it does not get stuck in web traps (say a calendar that has a button to go the next year which could easily scan up to the year 9595, if man is still alive). Or rather, after a certain number of passes (maybe N=10-15] all the real URLs have been crawled and the remaining URLs are web traps. In a language like Java it is tempting to write a much more complex crawl manager that has the potential to crawl faster with threads but takes more care to not crash target sites, get caught in web traps, etc. The first web crawler I wrote (1999) had a highly complex crawl control system, future web crawlers I wrote got simpler and simpler because the simple breadth first crawler works very well for small projects.

Arnavion 3y ago

"find all phone numbers on their website" sounds like it had to act like a web crawler, not parse a bunch of files, so perhaps the complexity was because of that.

seadan83 3y ago
I like how y'all are thinking the complexity was in the web crawling. It was actually just a simple act of crawling over source files! Amazon was looking to remove their customer support phone numbers from all web pages (to save money, they wanted people to stop calling them; back in the early 2010s) I share the wonder that it took several days (perhaps that is the anecdote taking a life of its own, perhaps it was actually several hours. Or maybe it was was several days interspersed with on-call support, meetings, and other work running in parallel). Though.. the file parsing libraries back in the early 2010s in Java are somewhat basic, probably using Java 5. Which does make me wonder, how long would that exercise take in Java 8 or later compared to Java 5?
- qohen 3y ago
  Steve Yegge discusses this in his "The Five Essential Phone-Screen Questions" blog post[0] from 2004 and states that his team-member (at Amazon) produced the list of phone-numbers in an hour: Here are some facts for you to ponder: 1. Our Contact Reduction team really did have exactly this problem in 2003. This isn't a made-up example. 2. Someone on our team produced the list within an hour, and the list supported more than just the 2 formats above. 3. About 25% to 35% of all software development engineer candidates, independent of experience level, cannot solve this problem, even given the entire interview hour and lots of hints. [0] https://sites.google.com/site/steveyegge2/five-essential-phone-screen-questions https://sites.google.com/site/steveyegge2/five-essential-pho...
fbdab103 3y ago
I was considering that, but web crawling via shell definitely crosses the line of things I would not want to attempt. Although, I guess you could outsource that complexity with `wget --mirror`, bringing you right back to file-crawling.
- PaulHoule 3y ago
  My take is this. There is a very simple way to write a breadth-first webcrawler that works in stages and saves the frontier in an ordinary text file, this kind of webcrawler can be implemented in almost any programming language, particularly shell. That is, you write a script that loops over the urls in the frontier, fetches them, then writes new urls it find in the next frontier, you can use sort, uniq and such to manage the frontier. That simple breadth first webcrawler has the big advantage that it does not get stuck in web traps (say a calendar that has a button to go the next year which could easily scan up to the year 9595, if man is still alive). Or rather, after a certain number of passes (maybe N=10-15] all the real URLs have been crawled and the remaining URLs are web traps. In a language like Java it is tempting to write a much more complex crawl manager that has the potential to crawl faster with threads but takes more care to not crash target sites, get caught in web traps, etc. The first web crawler I wrote (1999) had a highly complex crawl control system, future web crawlers I wrote got simpler and simpler because the simple breadth first crawler works very well for small projects.