My only advice would be have a backup plan (redundancy) . I.e. Design basic version that works and dont get blocked as bot, then design another one. This will s

My only advice would be have a backup plan (redundancy) . I.e. Design basic version that works and dont get blocked as bot, then design another one. This will save you from situation like described below where your original method stopped working, but your client wants data now (because yjey pay for it). And be nice, dont take data you dont need. Keep it easy on servers.