The crawler actually DID use the API. I'm using crawler as it crawled the API. It would get the MAX ID from the API, then request every ID between the last one

The crawler actually DID use the API. I'm using crawler as it crawled the API. It would get the MAX ID from the API, then request every ID between the last one it found and that one. One of the things that would cause a problem early on is that a request would bring back a "null" response for an ID that should exist, so it would pause and retry after 5 seconds, and do that 5 times. Sometimes it would still not have a response, so any reply to that response would then crash on a referential integrity check. So I had to remove the integrity so that that reply would still come in, even when I couldn't request the parent for whatever reason (caching?). It was little things like that that would happen randomly at 3am, and kill the client for hours until I woke up and saw there was an exception in the log, and no threads had downloaded in hours. Heaven forbid I was on vacation when it went down. No SSH key to remote in. I would have to hard reboot the droplet from the DO admin panel.