4 ms·
Isn't this just like a 50 line python script to request all the anchor links and if any response is not 200 store in a file somewhere. then recursively do the s
by hasty_pudding 3y ago
Isn't this just like a 50 line python script to request all the anchor links and if any response is not 200 store in a file somewhere. then recursively do the same thing on wherever the working links lead ?
- haswell 3y agoA large number of useful apps/services could be described as “just a 50 line Python script”, but the devil is usually in the details. Spending a reasonable amount of money to not worry about those details is why a lot of us are employed.
- hasty_pudding 3y agoThis isnt one of what you described though. I've literally done this before. This is an extremely simple product that could be written in a morning by someone competent.
- haswell 3y ago> by someone competent Which is why there will always be a market for products like this. The point is that someone will pay a developer (someone competent) for software that just solves the problem for them. Plenty of people run websites and have no idea how to even begin something like this. The curse of knowledge we have in the bubble we're in makes these things appear simpler than they are to most people.
- sph 3y agoIt's a bit rude to assume I am incompetent. I am sorry, but you vastly underestimate the problem. Just read the other comments and see how many corner cases are there. It took me 3 days to implement a crawler than says OK on 200, and ERROR on 404, and 6 months to deal with exceptions to the rule, and apparently thanks to the feedback Ive received, there are many other exceptions to deal with. Sorry, this is a bit like that old Dropbox comment. The thing is, if your website is easily covered by a 50-line Python script, you do not need Bernard.
- hasty_pudding 3y agohttp is just a very simple request response protocol. maybe you're right though