4 ms·

How do you deal with the usual CF, akamai and other fingerprinting and blocking you? Or is that the customer's job to figure out?

omneity 1y ago
Thank you for the question! It depends on the scale you're operating at. 1. For individual use (or company use but each user is on their device) typically the traffic is drown out in regular user activity since we use the same browser and no particular measure is needed, it just works. We have options for power users. 2. For large scale use, we offer tailored solutions depending on the anti-bot measures encountered. Part of it is to emulate #1. 3. We don't deal with "blackhat bots", so we don't offer support to work around legitimate anti-bot measures such as social spambots etc.
- lyu07282 1y ago
  If you don't put significant effort into it, any headless browser from cloud IP ranges will be banned by large parts of the internet. This isn't just about spam bots, you can't even read news articles in many cases. You will have some competition from residential proxies and other custom automation solutions that take care of all of that for their customers.
  - omneity 1y ago
    Thanks, that's so true! We learned this the hard way building Monitoro[0] and large data scraping pipelines in the past, so we had the opportunity to build up the required muscle. One thing to note, there are different "tiers" of websites, each requiring different counter-measures. Not everyone is pursuing the high competition websites, and most importantly as we learned in several cases scraping is fully consensual or within the rights of the user. For example: * Many of our users scrape their own websites to send notifications to their discord community. It's a super easy way to create alerts without code. * Sometimes users are locked in their own providers, for example some companies have years of job posting information in their ATS they cannot get out. We do help with that. * Public data websites who are underutilized precisely because the data is difficult to access. We help make that data operational and actionable. We had for example a sailor setup alerts on buoys to stay safe in high waters. A random example[1] 0: https://monitoro.co https://monitoro.co 1: https://wavenet.cefas.co.uk/details/312/EXT https://wavenet.cefas.co.uk/details/312/EXT

omneity 1y ago

Thank you for the question! It depends on the scale you're operating at. 1. For individual use (or company use but each user is on their device) typically the traffic is drown out in regular user activity since we use the same browser and no particular measure is needed, it just works. We have options for power users. 2. For large scale use, we offer tailored solutions depending on the anti-bot measures encountered. Part of it is to emulate #1. 3. We don't deal with "blackhat bots", so we don't offer support to work around legitimate anti-bot measures such as social spambots etc.

lyu07282 1y ago
If you don't put significant effort into it, any headless browser from cloud IP ranges will be banned by large parts of the internet. This isn't just about spam bots, you can't even read news articles in many cases. You will have some competition from residential proxies and other custom automation solutions that take care of all of that for their customers.
- omneity 1y ago
  Thanks, that's so true! We learned this the hard way building Monitoro[0] and large data scraping pipelines in the past, so we had the opportunity to build up the required muscle. One thing to note, there are different "tiers" of websites, each requiring different counter-measures. Not everyone is pursuing the high competition websites, and most importantly as we learned in several cases scraping is fully consensual or within the rights of the user. For example: * Many of our users scrape their own websites to send notifications to their discord community. It's a super easy way to create alerts without code. * Sometimes users are locked in their own providers, for example some companies have years of job posting information in their ATS they cannot get out. We do help with that. * Public data websites who are underutilized precisely because the data is difficult to access. We help make that data operational and actionable. We had for example a sailor setup alerts on buoys to stay safe in high waters. A random example[1] 0: https://monitoro.co https://monitoro.co 1: https://wavenet.cefas.co.uk/details/312/EXT https://wavenet.cefas.co.uk/details/312/EXT