3 ms·
Modern crawlers can easily follow javascript or some basic POSTs. So, apart from the insane state-changing GET requests in the code, unless you are spidering yo
by pierlu 7y ago
Modern crawlers can easily follow javascript or some basic POSTs.
So, apart from the insane state-changing GET requests in the code, unless you are spidering your own application, I would not give a crawler admin credential. Anyway a very nice anecdote of the two-thousands decade. Never ever forget the basics.
- deleted 7y ago[deleted]