6 ms·
Abusing Ahrefs backlink checker
- cantSpellSober 3y agoYou can make the text a bit more legible with this userstyle: body { --fg: white; font-family: sans-serif; }
- skilled 3y agoAhrefs was using recaptcha until recently, I wonder if that was abused in the same way or whatever the reason for them switching up was.
- supriyo-biswas 3y agoRecaptcha could also be bypassed in the same way FWIW.
- moneywoes 3y agoIs there any captcha that can’t? I presume LinkedIn captcha
- supriyo-biswas 3y agoThat's Arkose Labs and 2Captcha, Anti-captcha and Nopecha have bypasses for them.
- marban 3y agoDo Ahrefs et al operate their own crawling infrastructure or is there another third party vendor for this?
- gkbrk 3y agoAhrefs operates their own crawler, and gets really valuable data from it. Source: My server logs.
- nubinetwork 3y agoThey're really bad at purging outdated search results, I often see them trying to GET pages that haven't existed in 5+ years.
- Kye 3y agoIt isn't just a search engine. It's also a link monitor. Someone somewhere is linking those pages. It might not be a current user, but they still crawl it in case the person with those bad links signs up.
- skilled 3y agoThey do. And they have their own search engine, too - Yep.com They have some info on their Data[0] page. I always saw them as - if Google Search had a public back-end UI, then Ahrefs would be it. [0]: https://ahrefs.com/big-data https://ahrefs.com/big-data
- smarx007 3y agoThey do: https://ocaml.org/success-stories/peta-byte-scale-web-crawler https://ocaml.org/success-stories/peta-byte-scale-web-crawle...
- gnfargbl 3y agoThe main point of this article is that you can pay humans to solve CAPTCHAs, via a company which acts as middle-man. The company in question (Kolotibablo) has a website at https://kolostories.com/ https://kolostories.com/ which argues that paid CAPTCHA-solving is a good economic option for the people who do it. Maybe it is? I can't decide.
- arthur_sav 3y agoI wonder if it's cheaper to just pay for a subscription and get access to their API - given that solving captcha cost $1-2.
- slig 3y ago$1 - $2 per thousand queries.
- arthur_sav 3y agooh, good observation
- smarx007 3y agoLooks like API access is only available at $1k/mo.: https://ahrefs.com/pricing https://ahrefs.com/pricing
- moneywoes 3y agoHmm is this still cheaper than purchasing the service?
- mdaniel 3y agoI wonder if there's some architectural reason for using `["Status", {}]` shapes instead of a more traditional response shape [ "Ok", { "data": { It seems to place a lot of the error detection burden upon the caller, since the response can be empty `[]`, it can have a status without a body (which, I guess may be fine for their api?) `["OkHarHar"]`, and then the usual promises-made-promises-kept of the expected keys being in the `response[1]` location. One can see this heartache sort of play out in the subsequent python code in the blog post although they just return None in the simple `len() != 2` case
- akiselev 3y agoWith pattern matching it'd look like: match (response) { ["Ok", { data }] => { ... }, ["Error", { error }] => { ...handle known error... } _ => { ...handle unknown error... }, } It could also just be auto-generated serialization/deserialization code like in Rust.
- petercooper 3y agoI guess that explains why Turnstile has been as useful as an ashtray on a motorbike for stopping automated signups for me.