It's certainly an interesting approach! Depending on the volume, it could have the benefits you described - but it may get a bit old when the volumes increase.

It's certainly an interesting approach! Depending on the volume, it could have the benefits you described - but it may get a bit old when the volumes increase. I'd engineer the crawler to talk to a persistent message queue, and load the database from there. That gives you a lot of flexibility to move loads around, instrument the queue and you're not reinventing things, either.