> But if you are trying web scraping at scale, reverse engineering should always be your first choice. Not only it enables you a faster solution, it is more eth

> But if you are trying web scraping at scale, reverse engineering should always be your first choice. Not only it enables you a faster solution, it is more ethical (IMO) as you are minimizing your impact to it's resources. Rendering full website resources is always my last choice. I find my time is by far the most limited resource. I am usually scraping huge corporations at scale and don't care/doubt I will impact their resources. If they would open their APIs I would use those. That being said, I often end up reverse engineering to preserve my own resources. I can and do run thousands of instances of chrome but it isn't cheap. Also, related to IPs, carrier grade NAT has been a blessing ;)