19 ms·

Ask HN: Someone is proxy-mirroring my website, can I do anything?

• dutchbrit 4y ago
  Warning: Don't visit the proxy mirror at work, I was redirected to xcams/adult content.
  • geoah 4y ago
    weird, wasn't the case here.
  • agotterer 4y ago
    The mirror is injecting their own ads. My guess is it was just a malicious ad forcing the redirect. It could still happen, but it doesn’t appear to be the main intention of the mirror site.
• satya71 4y ago
  We had a similar situation, though it was just a snapshot. We found out they were hosting using S3, and filed a DMCA request with AWS. It was taken down and hasn’t returned.
• simonz05 4y ago
  Another option would be to use a service like Cloudflare, which offers protections against scraping and other malicious behavior. This can help prevent the proxy-mirror site from being able to access your site's content. https://blog.cloudflare.com/introducing-scrapeshield-discover-defend-dete/ https://blog.cloudflare.com/introducing-scrapeshield-discove...
  • stanislavb 4y ago
    I have Cloudflare at the front already. The issue is that they are not actively scraping the content but rather mirroring it on demand.
    • fragmede 4y ago
      Then where they're coming from should be exceedingly visible in logs (via splunk or whatever), so deny those requests.
    • maybe_pablo 4y ago
      Just an idea but maybe you could cause big loads on their servers by requesting in parallel a large amount of urls where you actively serve a gzipped massive html file that is full of links to your website. EDIT: or building up on what user zhouyisu says above you can generate your perfect match IP blacklist by calling urls via the abusing site that automatically puts any caller into the blacklist.
      • is_true 4y ago
        I'm not sure how easy would be to serve a "zip bomb" without getting into trouble, but it would be neat
    • DoctorOW 4y ago
      If you're on Cloudflare and they're stripping JS wouldn't a JS challenge be appropriate? https://developers.cloudflare.com/fundamentals/get-started/concepts/cloudflare-challenges/#js-challenge https://developers.cloudflare.com/fundamentals/get-started/c...
• trinovantes 4y ago
  They are probably using some public cloud service so simply banning all IPs from cloud ASNs [1] will usually be enough. Downside is you're also banning any users using VPNs [1] https://github.com/brianhama/bad-asn-list https://github.com/brianhama/bad-asn-list
  • stanislavb 4y ago
    Thanks, that seems like something I could work on if I can't find a better solution. Cheers.
  • gary_0 4y ago
    Another resource that can be used to check for abusive client IPs is https://github.com/firehol/firehol https://github.com/firehol/firehol
• sdrinf 4y ago
  If the host (DediPath) is not respecting DMCA notices, one other thing you can do is adding the requester's IP address to every page, eg as a div class. If the responses are live proxied, this will surface the cloner's front-facing IP address, and you can block that (and their ASN) specifically.
  • filleokus 4y ago
    Oh, I like this idea. Would be pretty easy to automate it by setting up some script scraping the IP revealed on their site, adding it to the block list as they rotate around. Clever.
    • _siis 4y ago
      Wouldn't they be able to do the same preventing you from scraping the site? They may have many IPs to work with, but you may not?
  • MildlySerious 4y ago
    To extend on this, I wouldn't use clear text for this. Create a HMAC of the IP and add it somewhere in the page, makes it harder to realize what's happening and for the adversary to work around it.
  • stanislavb 4y ago
    I'm not sure I can understand your advice.
    • ktpsns 4y ago
      I think it works the following: Assuming the proxy has a different IP pointing to it's client, by inserting the IP it uses to connect to the original server into the HTTP reply (HTML/body code), it can be exposed to the OP. However, since he seems to have access logs and seems to understand the proxy requests pretty well, I wonder how it actually helps.
    • pornel 4y ago
      Add a comment (or attribute or JS with a string literal) to your HTML that contains IP address of whoever requested the page. Obscure it somehow so it's not obvious that the HTML contains the IP address. Then check source code of the copy, and you'll see who requested it. You can then go after that IP. BTW: if they're removing/replacing domain name of your site, try obscuring it with HTML entities. This may dodge simple find'n'replace.
• zhouyisu 4y ago
  Maybe some logic honeypot would be good, such as a infinite content paging list with some random trigger hidden at pages with non-sense titles. When one IP hits these triggers, it is automatically banned. Bots will trigger it by walking through all pages, but real human would not click in since the paging is non-sense and titles are non-sense.
  • stanislavb 4y ago
    Yeah, but I don't want to ban bots. Also, they are not actively crawling anything, but rather mirroring the content on demand. At least that's my observation so far... thanks anyways.
• colesantiago 4y ago
  If you have a trademark, domain takedown always works.
  • 2000UltraDeluxe 4y ago
    That isn't neccesary true. While I definitely agree that the OP should pursue that path, many providers are in other jurisdictions.
• politelemon 4y ago
  Add a link rel="canonical" to your pages as well, it should give engines a hint that your domain is the legit one. https://webmasters.stackexchange.com/questions/56326/canonical-urls-with-multiple-domains https://webmasters.stackexchange.com/questions/56326/canonic... I noticed that the other domain is hotlinking your images. So you can disable image hotlinking, by only allowing certain domains as the referers. If you block hotlinked images then the other domain will not look as good. Remember to do it for SVGs too. https://ubiq.co/tech-blog/prevent-image-hotlinking-nginx/ https://ubiq.co/tech-blog/prevent-image-hotlinking-nginx/ Finally I also see they are using a CDN called Statically to host some assets off your domain. You can block their scrapers by user agent listed here: http://statically.io/docs/whitelisting-statically/ http://statically.io/docs/whitelisting-statically/
  • stanislavb 4y ago
    I think they are replacing all mentions of saashub.com with their domain. Also, I'm not using statically.io, that's something they are prepending in front of all images. Automatically.
    • CGamesPlay 4y ago
      But Statically isn't forwarding the User-Agent of the visitor, and they publish the list of User-Agents that they use, which you can block.
    • politelemon 4y ago
      It's adding the CDN for some of the images but not all of them, so you'd have to cover both
    • matt_heimer 4y ago
      Sometimes the replacement is done with simple pattern matching. Try different forms of encoding you domain to see if you can get through their replacement.
• cheri9 4y ago
  [dead]
• pornel 4y ago
  If they are stripping all JS, you can make the page work only with JS enabled :/
• pornel 4y ago
  Don't block their IPs, but rather return them subtly wrong content that isn't broken at the first glance. Insert typos, replace important terms, inject nonsense technobabble, make URLs point to wrong pages, inject off-topic SEO-spammy keywords that search engines will see as the SEO spam they are.
• musabg 4y ago
  1. Create fake url endpoint. And go to that endpoint in the adversary's website, when your server gets request, flag the ip. Do this nonstop with a script. 2. Create fake html elements and put unique strings inside. And you can search that string in search engines for finding similar fake sites on different domains. 3. Create fake html element and put all request details in encrypted format. Visit adversary's website and look for that element and flag that ip OR flag the headers. 4. Buy proxy databases, and when any user requests your webpage, check if its a proxy. 5. Instead of banning them, return fake content (fake titles and fake images etc) if proxy is detected OR the ip is flagged. 6. Don't ban the flagged ip's. She/He's gonna find another one. Make them angry and their user's angry so they give up on you. 7. Maybe write some bad words to the user on random places in the HTML when you detect flagged ip's :D So the user's will leave the site and this will reduce the SEO point of the adversary. Will be downranked. 8. Enable image hotlinking protection. Increase the cost of proxying for them. 9. Use @document CSS to hide the stuff when the URL is different. 10. Send abuse mail request to the hosting site. 11. Send abuse mail request to the domain provider. 12. Look for the flagged IPs and try to find the proxy provider. If you find, send mail to them too. Edit: More ideas sparkled in my mind when I was in toilet: 1. Create fake big css files (10MB etc). And repeatedly download that from the adversary's website. This should cost them too much money on proxies. 2. When you detect proxy, return too big fake HTML files (10GB) etc. That could crash their server if they load the HTML into the memory when parsing.
  • mkoryak 4y ago
    I like how you think. These are all great ideas! Reminds me of a time some real estate website hotlinked a ton of images from my website. After I asked them to stop and they ignored me I added an nginx rewrite rule to send them a bunch of pictures of houses that were on fire. For some reason they stopped using my website as their image host after that.
    • egberts1 4y ago
      What a sure-fire way to toast them! Kudos!
    • 4y ago
• marginalia_nu 4y ago
  Look at your traffic logs and see if you can't fingerprint the scraper. Should be relatively easy since they're mirroring your entire site. Then instead of blocking the fingerprint, poison the data. Introduce errors that are hard to detect. Maybe corrupt the URLs, or use the incorrect description or category. Be creative, but make it kind of shit. It's easy to work around blocks. Working around poisoned data is much harder.
  • buro9 4y ago
    This... there are definitely aspects of the proxy that they aren't configuring or are unaware of. i.e. ssl_cipher, http_x_requested_with, http_accept... and the order of all headers supplied... the casing of all headers supplied... TLS client HELO. It is relatively easy, if you have enough signals, to essentially create a fingerprint that they won't understand how it works. Yet it will be effective at blocking it regardless of the IP. Once you add enough of these together it will be hard for them to get around it without being obvious as they do so. Super aggressive... those same fingerprints will reveal legit browser traffic and the fingerprints for things like Google-bot... so you could go towards a whitelist rather than blocklist. But this is a place you'd have to actively manage as new variations arise constantly.
    • RockRobotRock 4y ago
      This is some really cool anti-scraping inside baseball. Is it safe to say that Cloudflare uses these techniques for weeding out bots?
      • buro9 4y ago
        It's safe to say that if you have enough signals from every possible layer (of which the above a barely a few) that it becomes trivial to build a model that can identify the majority of bots. However, then you're left with the really hard problem of when real browsers are used. But hey, you went a long way before you had to actually look at traffic patterns and in the meantime you've significantly raised the costs for those operating the bots. It's also worth noting that if you really get enough signals, that bot writers cannot control them all. Everyone can rewrite a HTTP header, but can you pick the right HTTP headers in the right order with the right TLS cipher and TLS HELO to appear to be the same as Chrome on Windows? Good luck.
• heartbeats 4y ago
  What about steganography? If you change subtle details about spelling, spacing, formatting, etc by the source IP, then you can look at one of their pages and figure out which IP it was scraped from. Then, just add goatse to all pages requested by that IP. Alternatively, replace every other sentence with GPT-generated nonsense. EDIT: it should be quite easy to use JS to fingerprint the scraper. The downside is that you will also block all NoScript users.
• 55555 4y ago
  Anyone have any idea what this does? It's embedded in the copycat site's source: earlierindians.com/39faf03aa687eeefffbe787537b56e15/invoke.js
  • mhasbini 4y ago
    Deobfuscated https://gist.github.com/mhasbini/97b471911866e7c7016fa6f27e7f10c7 https://gist.github.com/mhasbini/97b471911866e7c7016fa6f27e7...
    • 55555 4y ago
      Cool bot detection!
• NicoJuicy 4y ago
  Quick and easy first: 1) Add a watermark to your images when they proxy to you. Stolen image from {url} 2) Add a js script when the url differs from yours and display a message + redirect
• JW_00000 4y ago
  What about a slightly alternative approach, where instead of trying to block the abuser, you try to make it clear to end users what the real website is? E.g. in your logo image, include the real domain name "saashub.com". Have some introduction text on your home page "Here at saashub.com, we compare SaaS products ...." When your images are hotlinked, replace them with text like "This is a fraudulent website, find us at saashub.com". Anything that can make it obvious to end users that they're on the wrong website when they visit the abuser's URL. By the way, I've also reported the abuser as a phishing/fraud website through https://safebrowsing.google.com/safebrowsing/report_phish/?url=https%3A%2F%2Fsukuns.us.to https://safebrowsing.google.com/safebrowsing/report_phish/?u...
  • riz_ 4y ago
    Not sure if this would help since: > 4) Use absolute URLs everywhere - they are rewriting everything www.saashub.com to their domain name.
    • lukevp 4y ago
      Embed the welcome text in an image then!
    • antifa 4y ago
      Try some things like sa(zero-width-space)ssh<b></b>ub.com
• halifaxbeard 4y ago
  Setup Cloudflare on the domain and turn on “bot fight mode”. If the TLS ciphers the client proposes for negotiation doesn’t align with the client’s User-Agent they get a CAPTCHA. I would suspect that whoever is doing this proxy-mirroring isn’t smart enough to ensure the TLS ciphers align with the User-Agent they’re passing through.
  • nezirus 4y ago
    I would agree with the above, as an easier version of TLS fingerprinting. One could also ise nginx/haproxy to extract enough TLS info, and detect requests xoming through proxy Magic string: JA3 fingerprint
  • supriyo-biswas 4y ago
    On the free tier, does bot fight mode do anything other than simply detect bots based on JavaScript detections?
  • strictnein 4y ago
    This is the correct first step.
  • deleted 4y ago
    [deleted]
• amelius 4y ago
  Generate your pages from Javascript.
• MohammadAZ 4y ago
  I'm not going to focus on the problem here. I just want to say that I like the idea behind your website, a good source of market research. Bookmarked it.
• henryackerman 4y ago
  I wonder what they get out of this. Injecting ads perhaps?
  • askiiart 4y ago
    I saw another comment saying that the site sometimes redirects to adult content sites.
  • chillfox 4y ago
    Yep, they injects ads
  • kjs3 4y ago
    Landing place for a phishing campaign. See: watering hole attack.
• arnorhs 4y ago
  I would just contact cloudflare via discord. They will know what your best recourse is
• deleted 4y ago
  [deleted]
• adql 4y ago
  When someone did it to us we replaced the served content with ads for our site.
  • jansan 4y ago
    That sounds like the most elegant approach so far.
• coding123 4y ago
  Their domain is likely to be considered hostile, and drop in Google results.
• gildas 4y ago
  Block all requests having "https://sukuns.us.to https://sukuns.us.to" as "Referer" HTTP header.
  • pastacacioepepe 4y ago
    Requests are proxied so the proxy can rewrite the Referer HTTP header at will, AFAIK.
    • gildas 4y ago
      It looks like they're also downloading images directly from your domain, I see https://www.saashub.com/images/app/service_logos/129/k2q4pxz97dmy/medium.png https://www.saashub.com/images/app/service_logos/129/k2q4pxz... for example in my debugger. Edit: you could maybe add a <meta> tag to define a CSP in but I guess they will remove it [1]. [1] https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP
• batch12 4y ago
  Have you looked at filtering the traffic by ASN? You may be able to identify the provider your adversary likes to use and apply some of the controls musabg suggested to any traffic sourcing from these networks. I have a website doing this to one of my domains. I have let it slide for now since I get value out of users that use their site too, but I have thought about packing their content with advertisements to turn the tables a bit.
• than3 4y ago
  Have you considered enabling HSTS on the webserver with dynamic endpoints and rate limiting after detecting the flagged IP, literally to a crawl? I seem to recall someone doing something similar at one point hosting files and setting up resources that get pulled down only on flagged IPs such as a 300kb gzip encoded file that tries to expand to 100TB.
• sagarpatil 4y ago
  What’s the motivation for someone to proxy mirror a site? Does copied content even rank in Google? How are they driving the traffic to it?
  • nigma1337 4y ago
    Motivation is money, the proxy site is serving their own ads. According to OP, it ranks pretty well on bing.
• zupa-hu 4y ago
  You can overload their servers as they pass on any URL to you. Just make sure their resource usage is significantly more than yours. Eg. serving a gzipped response that contains millions copies of the same URL to your site. Ideally, make it not fit in their memory. Or, simply large enough to take really long to compute. Put it under a URL only you know, then start DoS-ing it. Of course that requires you to be able to serve a prepared gzipped resonse, depends on your stack.
• Fire-Dragon-DoL 4y ago
  Could you prepend full urls to your website on assets and trigger cors requests on all assets? That would make it really annoying to proxy
• santah 4y ago
  Same thing happened to me and my service (https://next-episode.net https://next-episode.net) almost 2 years ago. I wrote a HN post about it as well: https://news.ycombinator.com/item?id=26105890 https://news.ycombinator.com/item?id=26105890, but to spare you all the irrelevant details and digging in the comments for updates - here is what worked for me - you can block all their IPs, even though they may have A LOT and can change them on each call: 1) I prepared a fake URL that no legitimate user will ever visit (like website_proxying_mine.com/search?search=proxy_mirroring_hacker_tag) 2) I loaded that URL like 30 thousand times 3) from my logs, I extracted all IPs that searched for "proxy_mirroring_hacker_tag" (which, from memory, was something like 4 or 5k unique IPs) 4) I blocked all of them After doing the above, the offending domains were showing errors for 2-3 days and then they switched to something else and left me alone. I still go back and check them every few months or so ... P.S. My advice is to remove their URL from your post here. This will not help with search engines picking up their domain and ranking it with your content ...
  • deleted 4y ago
    [deleted]
  • rexreed 4y ago
    For 2) you mean you loaded it from the adversary's proxy site, just to clarify?
    • deleted 4y ago
      [deleted]
    • santah 4y ago
      Yes, constructed the honeypot URL using the proxy site and called it (thousands of times) so I can get them to fetch it from my server through their IP so I can log it.
      • WirelessGigabit 4y ago
        They literally proxy your website? I thought they'd cache it... that makes more sense now in your statement that you hit their website with a specially formatted url. Since they pass that through to you you can filter on that. Also: since you say 4k-5k IPs... any of them from cloud providers? And specific location?
• pera 4y ago
  *.us.to are FreeDNS subdomains, I would contact them. Additionally you could do a whois and contact the ISP.
• deleted 4y ago
  [deleted]
• nstart 4y ago
  Sending a mail to the hosting provider is helpful. Also, if you are looking at blocking IP you can try blocking an entire ASN temporarily to see how that works. It’s one thing for someone to destroy a server and reimage it on the same service. It’s another thing to destroy it and bring it up on a fresh provider. Currently the attacker is using dedipath for example. Block the ASN while waiting for their abuse team to respond.
• alexfromapex 4y ago
  There are ways you can fix this yourself but like all things it's way easier to just get a managed solution. CloudFlare or similar should give the necessary tools to block these types of sites.
• reaktivo 4y ago
  Redirect via meta tag <meta http-equiv="refresh" content="time; URL=new_url" />
  • Benanov 4y ago
    won't survive a grep
    • chris_wot 4y ago
      Still never easy, they need to do a small amount of work. Shot across the bows, so to speak.
• jFriedensreich 4y ago
  there are infinite mitigations and it will always boil down to how much they want to do this vs how much you want to prevent them. in the end they could render in a remote controlled browser and use cdn or aws ip adresses en mass, i would consider highjacking their users in subtle ways like replacing pictures or text with obscenities or legal disclaimers. unfortunately their motivation is ad selling to other dodgy companies so unlikely you can mitigate that way. i would also invest in getting the seo in order and having them removed from google if possible. lastly there are solutions like cloudflare turnstile that impact normal users not as much as in days of captchas
  • highwaylights 4y ago
    Maybe op only needs to do enough to undermine their website, rather than drive them away. it’s possible the combination of blocked image hotlinks, watermarking the domain inside the images, and CSS trickery that messes up the page on the proxy (along with whatever other steps that can be thought of to make it look wrong or erroneous on the proxied site) could get op bumped to #1 on search on enough links that it no longer matters. Given the other site isn’t generating original content it’s unlikely to ever get its google juice back. On a side note - does Google have an option for this? I’m sure they must have encountered this before and it helps the quality of their results too to block obviously fudged content.
    • jFriedensreich 4y ago
      I would try googles phishing report (as others here have reported and allready done)https://safebrowsing.google.com/safebrowsing/report_phish/ https://safebrowsing.google.com/safebrowsing/report_phish/ even the example here is not targeted at stealing user dat per se
• ycommentator 4y ago
  Lots of great ideas here. A slight variation or emphasis on some: Specifically aim to advertise your own site on the other one. While you can anyway. Free advertising to their (should be your) audience, in return for what they're doing... Seems fair!
• NorwegianDude 4y ago
  Add the worst content imaginable to the page, but don't make it visible by default. If the site strips JS, then use CSS to only show the terrible content when it's shown on that domain. You can use css to check the current domain based on e.g. links. Extra points if you can cause legal trouble for whoever runs the site. If you're hosting rather large files, then you can also hide content by default that will never be loaded on your site, but will load on the other site. Add a large file to your site, then reference that file a few thousand times with query params to ensure cache busting, and then make the browser load it all using CSS when it detects that it runs on the other site.
• ycommentator 4y ago
  My networking knowledge isn't great, so apologies if this is wrong. But if it's not wrong, it could help. FIND THE IP FOR THE DOMAIN PS > ping sukuns.us.to Pinging sukuns.us.to [45.86.61.166] with 32 bytes of data: Reply from 45.86.61.166: bytes=32 time=319ms TTL=39 ... REVERSE DNS TO FIND HOST https://dnschecker.org/ip-whois-lookup.php?query=45.86.61.166 Apparently it's "Dedipath". And that WHOIS lookup gives an abuse email address: "Abuse contact for '45.86.60.0 - 45.86.61.255' is 'abuse@dedipath.com'" So you could try emailing that address. They may take the site down, or hopefully more than that...
  • mxuribe 4y ago
    This is not a bad idea, though i would guess that if these guys change IPs, then it will be annoying to spend your time sneding emails, etc. But, then i thought: why not automate this with some simple scripts? You have al;ready outlined your recipe, so simply automate the steps...But the more i thought of the automation around this, you need to be creful not to turn into a "spammer of sorts, constantly sending emails...certainly, you wouild be sending legitimate emails, but if they change their IPs more often, that might trigger your automatiomn more often, somewhat turning you into a mild "spammer", right? :-) I'm not suggesting you abandon your apporoach, but simply to remember to not overdo it with big scale of emails sent out. ;-)
    • ycommentator 4y ago
      Aha, some more good ideas there! But you're right, there's tradeoffs and dependencies and uncertainties throughout, so it's not easy to even guess in advance what would work or be worthwhile. Plus as you say there could be negative consequences from a kind of arms-race, with the solution becoming a problem in itself. It's not the same thing, but I'm reminded now of email in the past, when you would usually get an undeliverable message if something went wrong. But later that was almost entirely stopped - because of spam. Massive volumes of spam was sent from forged addresses, and much of it led to those replies. So that made things worse by doubling the volume, plus the innocents whose addresses had been forged got deluges of confusing undeliverable messages! I think you're right in that changing IPs would be easy for them. But, changing hosts would be significantly more work and hassle. So if the abuse reporting worked, that could have much more of an impact...
• 3nt0py__ 4y ago
  well... the only reasonable thing to do is to find a hosting that accepts monero as a payment, rent a baremetal server with ipmi acces, cript the hardisk with lusk and veracrypt, scan 0.0.0.0/0 for unpached dns servers and start ddossing the mirror site
• notorandit 4y ago
• lessname 4y ago
  If they copy the Javascript too, you could a code that checks if the domain matches and if not show a blank page (or something like that)
• signaru 4y ago
  Some "less technical" suggestions after being inspired by other creative suggestions here: Put brandings/personalizations/signatures in your pages that are not easy to remove to remove automatically. Include your site URL if possible. The idea is that if a visitor sees these on a different site, it becomes obvious that the content doesn't belong there. Write an article page about these things happening, specifically mentioning the mirroring site URLs, and see if they will also blindly mirror it.
• tyingq 4y ago
  Other posts mentioned ways to detect, like: - "bait urls" that other crawlers won't touch - trigger by request volume and filter out legit crawlers - find something unique about the headers in their requests that you can identify them with One additional suggestion is to not block them, but rather, serve up different content to them. Like have a big pool of fake pages and randomly return that content. If they get a 200/OK and some content they are less likely to check that anything is wrong. Another idea is to serve them something that you can then report as some type of violation to Google, or something (think SafeSearch) that gets their site filtered.
• wpietri 4y ago
  One strategy tip: don't play cat and mouse. As you've demonstrated, if you change one thing, they will figure it out and change one thing. Not only does that not work, but you are training them that it's worth trying to beat your latest change. Instead, plot a few different changes and throw them in all at once. Preferably in a way where they will have to solve all of the changes at the same time to figure out what happened and get things working again. Also, favor changes that are harder to detect. E.g., pure IP blocks are easier to detect than tarpitting and returning fake/corrupted content. The longer their feedback loops, the more likely it is that they'll just give up and go be a parasite somewhere else.
  • DamnInteresting 4y ago
    > pure IP blocks are easier to detect than tarpitting and returning fake/corrupted content I recently had to employ such a strategy against some extremely aggressive card testers (criminals with lists of stolen credit cards who automate stuffing card info into a donation form to test which cards are still working). Instead of blocking their IPs, I started feeding them randomly generated false responses with a statistically accurate "success" rate. They ran tens of thousands of card tests over many days, and 99% of the data they collected was bogus. It amuses me to know that I polluted their data and wasted so much of their time and effort. Jerks.
    • wpietri 4y ago
      This warms my heart and it's a great example of lengthening the feedback loop.
  • yonixw 4y ago
    I love it, also add a randomness, there is nothing more frustrating than a problem that only reproducers sometimes!
    • wpietri 4y ago
      Excellent idea!
• antonyh 4y ago
  One thing you can do is add a canonical to each page, which will help solve the the Bing/Google issue until they realise it's there. Do it before they add one. You're already using Cloudflare, you could try talking to their support or just turning up settings to make it more strict for bots.
• Ra8 4y ago
  You could maybe put your website behind a captcha? Google's recaptcha works behind the scene, so it won't affect normal users.
• rglover 4y ago
  You might be able to do an origin filter on the headers for requests to your backend (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-For https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-...). Check and make sure the x-forwarded-for header is what you expect and if not, block the request at middleware level.
• cph123 4y ago
  I used to do this to other websites (we won't go into why) - one thing that may help you is to always return your HTML responses gzipped, regardless of whether the client asked for them or not, so ignore Accept-Encoding preferences. This makes it harder for their server to rewrite your URLs on demand, and most clients will accept gzipped responses.
• 0xbkt 4y ago
  You can be able to identify those requests by inspecting the TLS cipher. Cloudflare Workers has that value in `request.cf.tlsCipher`[0]. Keep in mind the collateral damage it may have, though. [0] https://developers.cloudflare.com/workers/runtime-apis/request/#incomingrequestcfproperties https://developers.cloudflare.com/workers/runtime-apis/reque...
• nickphx 4y ago
  What about blocking by ssl fingerprinting? Established browsers have known fingerprints derived from how the ssl request is made, supported connection options, etc.
• tekno45 4y ago
  Block everyone else except them and start hosting disney content. Then give the mouse a ring
• thenickdude 4y ago
  Maybe they're also proxying URLs like the HTML verification files that search engines have you upload to claim the domain as your own? You may be able to claim their domain out from under them and then mess with search settings (e.g. In Google Search Console you can remove URLs from search results).
• T3RMINATED 4y ago
• deleted 4y ago
  [deleted]
• amurf 4y ago
  Instead of rendering server side, render client side. If they block JS, they get nothing. In JS script check for hostname and if it matches their hostname, don’t render anything. Potential downsides: SEO.
• modeless 4y ago
  HN probably won't like this but if they are blocking all JS you can make all content invisible with CSS and use JS to unhide it before page load finishes. Temporarily of course until these guys go away. The nice thing about this is it can be made arbitrarily complex. For example you can make the page actually blank and fetch all the normal, real content with JS after validating the user's browser as much as you like on both client and server. That's what Cloudflare's bot shield stuff does. Since JS is Turing complete there is no shortcut that the proxy can take to avoid running your real JS if you obfuscate it enough. They would have to solve the halting problem. What a determined adversary would do is run your code in a sandbox that spoofs the URL, so then your job becomes detecting the sandbox. But it's unlikely they would escalate to this point when there are so many other sites on the internet to copy.
  • SpeedilyDamage 4y ago
    HN contains multitudes, I love this response. At the very least you collect info about their sophistication level; will they adapt to adversity or will the bail/move on?
    • modeless 4y ago
      I say that because I know there are a lot of people on HN who browse with JS off and rail against sites that require it. But sometimes you need it.
  • ww520 4y ago
    I personally like to use JavaScript on websites and indifferent to the no-js movement. I think the assumption of HN being no-js is because of the vocal few.
  • jeroenhd 4y ago
    I very much would leave the website if I'm opening the site for the first time and it doesn't even render partially, but I recognise people like me are the minority of potential visitors. Unless you have a tiny target audience that includes people who tend to disable Javascript, this seems like a fine solution. I use JS to hide email from the most basic of scrapers most of my sites and so far it has worked wonders. However, long term this seems like a solution that will be difficult to maintain. Either the proxy will start stripping CSS as well (s/hidden/visible) or it'll require constantly playing along in a cat-and-mouse game that you don't stand to gain anything in by winning. I would add random, page-like endpoints that only you know and request them through their proxy (through VPNs/TOR/you name it). Ban the entire /48 (IPv6) or /24 (IPv4) or send them into a tar pit (iptables -A INPUT -p tcp -m tcp -dport 80 -j TARPIT for the IP address you target) to exhaust their resources.
• mkoryak 4y ago
  > They are stripping all JS Are they now? Add a `visibility: hidden` to random elements on the page, and show them with javascript. OR Are they removing _all_ js? Have you checked whether they remove `<body onclick="some javascript code that injects some other code">` ? You can try to do script injection _into your own site_ to see if their mirroring software is smart enough to deal with all the different xss vectors. Bonus points: if they remove your <body onhover=>` attribute, add a style like body { display: none} body[onhover='the js code that they will remove'] {display: block}
  • SeriousM 4y ago
    Just try some of the polimorphic xss tricks hacker try in order to get JavaScript into a page. Portswigger has a wonderful page of an extensive xss list.
• LinuxBender 4y ago
  I tried to look up their site then realized I block "us.to" locally. Since you have their site linked in this thread they are likely seeing the HN thread as a referrer in their access logs and reading this. I expect this to turn into an ongoing battle as a result, but maybe this could be a fun learning exercise for everyone here. The current IP 45.86.61.166 is likely a compromised host [1] which tells me you are dealing with one of the gangs that create watering holes for phishing attacks and plan to use your content to lure people in. They probably have several thousand compromised hosts to play with. Since others mentioned you could change the content on your site, I would suggest adding the EICAR string [2] throughout the proxied content as well so that people using anti-malware software might block it. They are probably parking multiple phishing sites on the same compromised hosts [3]. This would also be a game of whack-a-mole but if you can find a bunch of their watering hole sites and get the certificate fingerprints and domains into a text file, give them to ZeroSSL and see if they can mass revoke them. Not many browsers validate this but it might get another set of eyes on the gang abusing their free certs. If you have a lot of spare time on your hands, you could automate scripting the gathering of the compromised proxy hosts they are using and submit the IP, server name, domain name to the hosting provider with the subject "Host: ${IP}, ${Hostname}, compromised for phishing watering hole attacks". Only do this if you can automate it as many server providers have so many of these complaints they end up in a low priority bucket. Use the abuse@, legal@ and security@ aliases for the hosting company along with whatever they have on their abuse contact page. Send these emails from a domain you do not care about as it will get flagged as spam. Another option would be to draft a very easy to understand email that explains what is occurring and give that to Google and Bing. Even better would be if we could get the eyes of Tavis Ormandy from Google's vulnerability research team to think of ways to break this type of plagiarized content. Perhaps ping him on Twitter and see if he is up to the challenge of solving this in a generalized way to defeat the watering holes. I can think of a few other things that would trip up their proxies but no point in mentioning it here since the attackers are reading this. [1] - https://www.shodan.io/host/45.86.61.166 https://www.shodan.io/host/45.86.61.166 [2] - https://www.eicar.org/download-anti-malware-testfile/ https://www.eicar.org/download-anti-malware-testfile/ [3] - https://urlscan.io/result/af93fb90-f676-4300-838f-adc5d16b4735/related/ https://urlscan.io/result/af93fb90-f676-4300-838f-adc5d16b47...
  • 4y ago
• btbuildem 4y ago
  Lots of good suggestions here, let me throw one more in the pot -- could you do an equivalent of a "ghost ban"? Instead of blocking their IPs, detect if the traffic is coming from the abuser's IPs, and serve different content -- blank, irrelevant, offensive, copyright violations, etc.
• acomjean 4y ago
  DMCA takedown? They're serving your copyrighted content. Seems like what it was made for.
• werid 4y ago
  .us.to subdomains are sourced from (dynamic) dns provider, FreeDNS: https://freedns.afraid.org/ https://freedns.afraid.org/
• SergeAx 4y ago
  Does it hurt you in any way? If not - I would just leave it alone. Google can tell a copy from original. I tried to search some arbitrary text from your website - there is no trace of copycat in Google SERP. What striked me, though, is that a copycat website is waaaay faster than your original. If I were in your shoes, I would invest my time and effort into speeding up the site. Unlike hunting some script kiddies, that will bring palpable benefits.
  • zmmmmm 4y ago
    Bing is directing searches for their service to the fake web site, which is then serving up porn after a few seconds delay. I'd say it is hurting them.
• scarmig 4y ago
  This is a game of cat and mouse; although engineering approaches are fun, it's primarily an organizational/legal challenge, not a red/blue team exercise. The first line of defense is contacting the relevant authorities. This means search engines, the hosting provider, and the owner of the domain (who may not be the abuser). Be polite and provide relevant evidence. Make it easy for them to act on it. There'll be some turnaround time and it's not always successful, but it's the best way to get a meaningful resolution to the issue. What about in the meantime? If all the source IPs are from one ASN, just temporarily block all IPs originating from that ASN. There'll be some collateral damage, but most of your users won't be affected.
• someweirdperson 4y ago
  If they are serving all files, that should work for systems that check if you are the owner by asking to serve a file as a response to a challenge. The copy is using ZeroSSL. This seems to use a similar mechanism like letsencrypt to verify certs. Maybe, you could get their certificate by serving the response to their challenge from your server. Not idea how to proceed from there. Or activating the google webmaster tools. Maybe there's some setting "remove from index" or "upload sitemap" that could reduce its visibility on google.
  • watusername 4y ago
    That's actually a good idea. Apparently it's possible to revoke the certificates via the ACME API, even when you are using another ACME account: https://letsencrypt.org/docs/revoking/#using-a-different-authorized-account https://letsencrypt.org/docs/revoking/#using-a-different-aut...
• napolux 4y ago
  Want to have some fun? Happened to me back in the days of blogging. Posted an image of me mocking them on my blog. Sure enough they published it and they didn't notice for a while. They stopped it soon after :)
• frankzander 4y ago
  Had the same problem. They used a scraper which runs on Amazon AWS ... so I blocked all Amazon AWS IPs (google for the list of IPs ... and than for a script which creates you NGINX rules for all IPs). Works quite well.
• psyfi 4y ago
  Fail2ban might help Even with IP rotation, a proxy website would probably generate more traffic than normal from these few IPs, tweak fail2ban vars so you to make it less likely to trigger on false positives (larger number of requests / larger amount of time) but block the violating IPs for long period, few days for example. I hope it helps
• VanTheBrand 4y ago
  This will change if they switch hosting but here’s a list of all the ip prefixes for their current hosting provider. https://bgp.he.net/AS35913#_prefixes https://bgp.he.net/AS35913#_prefixes The IPs they switch between may all be from this pool.
• MR4D 4y ago
  I just checked the offending site - it’s full of malware. I think if you report that aspect then you might get faster resolution from the search engines.
• mortehu 4y ago
  Can you use the fact they they're proxying to prove to Bing and Google webmaster tools that you own their domain, and delist it? The verification is done by serving a file provided by Bing/Google.
  • shhsshs 4y ago
    If they're proxying /.well-known/acme-challenge/, you should be able to get a TLS certificate in their name through Lets Encrypt.
• shahidkarimi 4y ago
  What an idea. I will do same with some popular websites.
• someweirdperson 4y ago
  If direct (non proxied) access from the search engine spiders can be identified serve the real robots.txt. Otherwise disable crawling. Also, switch meta noindex like this.
• 5350-uiop-1130 4y ago
  add some CSS to mess with their URLs a[href*="sukuns"] { font-size: 500px!important; color: LimeGreen!important; } pretty much destroys the page. i guess eventually they would give up in the specificity battle. probably more stuff you could do with CSS to mess with them.
• deleted 4y ago
  [deleted]
• jamal-kumar 4y ago
  Believe it or not ICANN actually takes abuse reports seriously: https://www.icann.org/resources/pages/abuse-2014-01-29-en https://www.icann.org/resources/pages/abuse-2014-01-29-en
• khiqxj 4y ago
  i cant see your site. the mirror/proxy is useful and doing its job. Access denied Error code 1020 You do not have access to www.saashub.com. The site owner may have set restrictions that prevent you from accessing the site. Error details Caret icon Was this page helpful? Performance & security by Cloudflare External link
• cweagans 4y ago
  Just block all of the OVH IP ranges? https://ipinfo.io/AS16276 https://ipinfo.io/AS16276
• tacone 4y ago
  Some years ago I let expire my blog domain, only to find out that somebody bought it and was serving a mirror of my content plus scam ads. I reported them to their DNS provider and they were gone in 2-3 weeks.
• naijagoal 4y ago
  [dead]
• m1sta_ 4y ago
  Dynamically generate all the content in the browser to a canvas element. No HTML to steal. More simply you could just make all the HTML links broken unless some obfusticated or server-backed algorithm is run on them. Think google search results.
• __oh_es 4y ago
  Facebook had multiple approaches to keep users seeing ads (ie your message) on their site despite ad blockers. Could you mix a message amongst some content mixed up with some elements? Hopefully would not affect rankings too much, but could at least reach users. https://www.bbc.co.uk/news/technology-46508234.amp https://www.bbc.co.uk/news/technology-46508234.amp Base64 encoding images with watermarks may also be worth a shout. Love the zip bombing. Long shot but I wonder if its possible to execute some script on their server.
• oliv__ 4y ago
  This made me wonder whether something similar was happening to my domain? How would one go about finding out?
  • ycommentator 4y ago
    Maybe as simple as a Google / Bing (Yandex / Baidu?) search using some text likely to be unique to your site ("enclose the text in quotes" like that). If your site had been copied but wasn't indexed by the big search engines, that would presumably limit the options to make money from it?
• balls187 4y ago
  Um, so I clicked the second link, and was redirected to a not-safe-for-work website. Luckily, I am at home, and my children are at school. I have no idea what happened, or why I got redirected, but I can certainly suggest not taking up the idea to serve disgusting content (given I clicked a link that someone on HN posted, I shouldn't be subjected to that).
  • deleted 4y ago
    [deleted]
• meltedcapacitor 4y ago
  Use them a free CDN? User page from your domain actually downloads content through them but with your ads. (Maybe for the continents which are not your primary market.) (Less economical if they're not caching anything.)
• achillean 4y ago
  The following can be done for free without an API key or Shodan account: 1. Grab the list of IPs that you've already identified and feed them through nrich (https://gitlab.com/shodan-public/nrich https://gitlab.com/shodan-public/nrich): "nrich bad-ips.txt" 2. See if all of the offending IPs share a common open port/ service/ provider/ hostname/ etc. Your regular visitors probably connect from IPs that don't have any open ports exposed to the Internet (or just 7547). 3. If the IPs share a fingerprint then you could lazily enrich client IPs using https://internetdb.shodan.io https://internetdb.shodan.io and block them in near real-time. You could also do the IP enrichment before returning content but then you're adding some latency (<40ms) to every page load which isn't ideal.
• Doorstep2077 4y ago
• deleted 4y ago
  [deleted]
• ddalex 4y ago
  Once upon a time, I served all the static content base64-xored with a session key on the backend, and decrypted the content on the front-end with JS.
• worthless-trash 4y ago
  > 3) Add some JavaScript to redirect to the original domain-name - They are stripping all JS. Make your site only work with JS.. Easy.
• lofaszvanitt 4y ago
  Kill their ad ranking by inserting explicit words into the content body only porn sites use (like get the categories/some video titles from pornhub). Google only shows results from adult sites when the search string also has explicit words in it.
• darthrupert 4y ago
  Implement a reverse swearword filter for their IP.
• alam2000 4y ago
• codeonline 4y ago
  When you detect their ip.. - serve images upside down - serve images blurry More examples here from long long ago.. http://www.ex-parrot.com/~pete/upside-down-ternet.html http://www.ex-parrot.com/~pete/upside-down-ternet.html
• JensRantil 4y ago
  Depending on the nature of access patterns you might be able to automatically block the IPs by tuning the parameters on fail2ban (if you have a server) to block the proxy IPs.