Reading through the article and comments here I find it odd that a company like FB can't get a crawler to respect robots.txt or 429 status codes. Even I would

Reading through the article and comments here I find it odd that a company like FB can't get a crawler to respect robots.txt or 429 status codes. Even I would stop and think "maybe I should put in some guards against large amounts of traffic" when writing a crawler, and I'm certainly not one of those brilliant minds who manage to pass their interview process.