4 ms·
I work in a company that does a lot of web scraping, mostly from companies who's ToS says no scraping. Instead of violating the ToS we have business contracts
by danpalmer 9y ago
I work in a company that does a lot of web scraping, mostly from companies who's ToS says no scraping.
Instead of violating the ToS we have business contracts with each company, that give us the permission to scrape. We use this as a way to take control of integrations, and put the ball in our court, as most of these companies have little to no technical expertise or resource. By doing this we can create an integration as quickly as we want, instead of waiting months or even never managing to get one if it were to be done through an API.
Scraping can be a powerful tool in this respect, make sure you have permission, but ToS saying no scraping doesn't necessarily mean you can't get special permission.
- nurettin 9y agoA customer asked us to integrate their website with our mobile application. They had no IT department to do the integration, and they didn't want to open their database to the web, so I came up with this solution; ssh into the machine, select using a db commandline tool, return data back from stdout in csv format, parse data at our side. It's surprisingly fast and secure.
- caleblloyd 9y agoSSH tunneling supports forwarding remote hosts to local ports. Then you could use a proper database connector in your app. For example access MySQL through an SSH tunnel: https://stackoverflow.com/questions/18373366/mysql-connection-over-ssh-tunnel-how-to-specify-other-mysql-server https://stackoverflow.com/questions/18373366/mysql-connectio...
- nurettin 9y agoI forgot to mention that it was an SQLite database.
- chx 9y agoThey could reverse SSH even so they don't have a port open.