9 ms·

I have some experience on this field. Around two years ago i was a DevOp for the company running Dagbladet, Norways #2 newspaper. One of the things I did was ke

by Ueland 5y ago

I have some experience on this field. Around two years ago i was a DevOp for the company running Dagbladet, Norways #2 newspaper. One of the things I did was keep an eye on mysterious traffic. I managed to find a huge spam network that set up a proxy service that delivered normal content, but injected "you can win an iPhone!" spam to all users visiting them. Since I was in the position of being able to monitor their proxy traffic towards many sites I managed. I could easily document their behaviour. In the same time, I wrote a crawler that visited their sites over a long, long time. I learned that they kept injecting hidden links to other sites in their network, so I did let my bot look at those also. By this time, I also got a journalist with me that started to look at the money flow to try and find the organisation behind it. My bot found in excess of 100K domains being used for this operation, targeting all of westeren Europe. All the 100K sites contained proxied content and was hidden behind Cloudflare, but thanks to the position I had, I managed to find their backend anyways. We reported the sites to both CF and Google, and to my knowledge, not a single site were removed before the people behind it took it down. Oh, and the journalist? He did find a Dutch company that was not happy to see neither him or the photographer :)

pepy 5y ago
Do you want to get to the bottom of this? A friend of mine is a top Dutch lawyer with an interest in these things.
- Ueland 5y ago
  This was two years ago and the network is now (to my knowledge) gone.
  - pepy 5y ago
    alright, thanks for the clarification
avian 5y ago
> We reported the sites to both CF and Google, and to my knowledge, not a single site were removed before the people behind it took it down. As someone that tried reporting spam sites because they were using content scrapped from my website, I'm not surprised. Cloudflare has a policy that they will not stop providing their IP hiding/reverse proxy services to anyone, regardless of complaints. The best they do is forward your complaint to the owner of the website, who is free to ignore it. They say "we're not a hosting provider" as if that's an excuse that they can't refuse to offer their service. I'm sure many spam websites would go away if they couldn't hide behind Cloudflare.
- FeepingCreature 5y ago
  That seems like the sort of thing that should require a judge's order.
  - trangus_1985 5y ago
    Cloudflare is not a public institution. It troubles me that they get to define, draw, and then maintain that line. However, I do agree - privacy unveiling like that should require a judge's order.
    - studentrob 5y ago
      Yes, it is akin to revealing the IP of a user on a social media site.
    - stavros 5y ago
      But they don't, that's explicitly their stance. There is no line. They host everyone equally. To do the opposite would require drawing a line.
      - account42 5y ago
        > They host everyone equally. Everyone except those that are too right wing.
ultimoo 5y ago
> By this time, I also got a journalist with me that started to look at the money flow to try and find the organisation behind it. Very curious to know what you found!
- Ueland 5y ago
  We did publish a whole series about the network and companies we found in the process, unfortunately in Norwegian only and soft-paywalled: https://www.dagbladet.no/nyheter/sonjas-52-oppdagelse-avslorte-det-norske-nettangrepet/70782498 https://www.dagbladet.no/nyheter/sonjas-52-oppdagelse-avslor...
  - IG_Semmelweiss 5y ago
    did the article resonate with norwegians? I assume the report probably answered so many questions of the populace on google's malfunction, even if nothing came out of it what was the feedback to the article ?
dylan604 5y ago
That sounds like a hell of an investigation, and now my curiosity is running. 100k domains sounds like an huge amount of logistics on their side to keep it all running. It would be interesting to read about how a spam company manages that kind of infrastructure compared to a "legit" company. Legit company will always have internal struggles between dev/sales/marketing, so things just take longer and are much more draining to accomplish. I'd imaginge spam org just needs to have bare minimum stuff up to satisfy whatever need it is they have knowing that humans won't necessarily be perusing those domains, yet it's 100K domains. I could almost see something like this running more smoothly. I can also see it being run by small number of people that let things lapse and it's just barely hanging together. So many questions...
- tluyben2 5y ago
  It is not very difficult to manage: a company of mine was bought by a squatter (I found out after dealing with a broker for the sale; I had to integrate it with their 'tech team' and walked away after) and for many years already, this all has been fairly easy to automate. The registars have apis, cloud flare has apis. There was 1 tech guy keeping it all up and running and he didn't have to do anything. It would register and provision with content automatically. There is really almost no work involved besides keeping money in the registrar account and the costs are only the domains probably, maybe they have a little hetzner load balanced setup with 2 machines but that's likely it.
lifeisstillgood 5y ago
Can I just clarify? There is / are organisations that a) scrape legitimate sites for content, b) host that content on their own 100K domains, c) sit behind cloudflare, d) do some seo??? e) when someone finds their site they then inject an ad or similar rubbish f) do this enough that they make money off the ad / competition / porn ? That seems like a problem that the ”original-source” metatag was supposed to stop?
- tyingq 5y ago
  Canonical urls help with noting your own purposeful duplicated content. But that meta tag goes on the duplicated content. So it doesn't help with scrapers, who strip that out.
  - lifeisstillgood 5y ago
    But I thought that it was useful for google - who could find two caches with same content, one of which was 2018 one of which 2020 and both say "this is canonical". At that point the 2018 version is real and the other rejected. Then again, you could just do it with publication dates ...
    - tyingq 5y ago
      I don't know why, but Google seems unable to figure out (or just doesn't care) "who published it first". I've seen it be confused many times.
tikiman163 5y ago
The reason you found so many domains is that they intentionally take down thier spam sites and reload them under a new domain every few hours. They do this so they can't be taken down by people reporting them as spam. They literally setup the next domain while the current one starts being used so they can do a live swap to the next one without interruptions to thier spam operations. This is typically done in an effort to spread Trojan malware to anybody running computers with out of date operating systems and browsers. Windows getting people off of Internet Explorer has been a huge hit for them as it reduces the number of possible vulnerabilities someone might have when they get sent to one of these Trojan spam sites.

by Ueland 5y ago

pepy 5y ago

Do you want to get to the bottom of this? A friend of mine is a top Dutch lawyer with an interest in these things.

Ueland 5y ago
This was two years ago and the network is now (to my knowledge) gone.
- pepy 5y ago
  alright, thanks for the clarification

avian 5y ago

> We reported the sites to both CF and Google, and to my knowledge, not a single site were removed before the people behind it took it down. As someone that tried reporting spam sites because they were using content scrapped from my website, I'm not surprised. Cloudflare has a policy that they will not stop providing their IP hiding/reverse proxy services to anyone, regardless of complaints. The best they do is forward your complaint to the owner of the website, who is free to ignore it. They say "we're not a hosting provider" as if that's an excuse that they can't refuse to offer their service. I'm sure many spam websites would go away if they couldn't hide behind Cloudflare.

FeepingCreature 5y ago
That seems like the sort of thing that should require a judge's order.
- trangus_1985 5y ago
  Cloudflare is not a public institution. It troubles me that they get to define, draw, and then maintain that line. However, I do agree - privacy unveiling like that should require a judge's order.
  - studentrob 5y ago
    Yes, it is akin to revealing the IP of a user on a social media site.
  - stavros 5y ago
    But they don't, that's explicitly their stance. There is no line. They host everyone equally. To do the opposite would require drawing a line.
    - account42 5y ago
      > They host everyone equally. Everyone except those that are too right wing.

ultimoo 5y ago

> By this time, I also got a journalist with me that started to look at the money flow to try and find the organisation behind it. Very curious to know what you found!

Ueland 5y ago
We did publish a whole series about the network and companies we found in the process, unfortunately in Norwegian only and soft-paywalled: https://www.dagbladet.no/nyheter/sonjas-52-oppdagelse-avslorte-det-norske-nettangrepet/70782498 https://www.dagbladet.no/nyheter/sonjas-52-oppdagelse-avslor...
- IG_Semmelweiss 5y ago
  did the article resonate with norwegians? I assume the report probably answered so many questions of the populace on google's malfunction, even if nothing came out of it what was the feedback to the article ?

dylan604 5y ago

That sounds like a hell of an investigation, and now my curiosity is running. 100k domains sounds like an huge amount of logistics on their side to keep it all running. It would be interesting to read about how a spam company manages that kind of infrastructure compared to a "legit" company. Legit company will always have internal struggles between dev/sales/marketing, so things just take longer and are much more draining to accomplish. I'd imaginge spam org just needs to have bare minimum stuff up to satisfy whatever need it is they have knowing that humans won't necessarily be perusing those domains, yet it's 100K domains. I could almost see something like this running more smoothly. I can also see it being run by small number of people that let things lapse and it's just barely hanging together. So many questions...

tluyben2 5y ago
It is not very difficult to manage: a company of mine was bought by a squatter (I found out after dealing with a broker for the sale; I had to integrate it with their 'tech team' and walked away after) and for many years already, this all has been fairly easy to automate. The registars have apis, cloud flare has apis. There was 1 tech guy keeping it all up and running and he didn't have to do anything. It would register and provision with content automatically. There is really almost no work involved besides keeping money in the registrar account and the costs are only the domains probably, maybe they have a little hetzner load balanced setup with 2 machines but that's likely it.

lifeisstillgood 5y ago

Can I just clarify? There is / are organisations that a) scrape legitimate sites for content, b) host that content on their own 100K domains, c) sit behind cloudflare, d) do some seo??? e) when someone finds their site they then inject an ad or similar rubbish f) do this enough that they make money off the ad / competition / porn ? That seems like a problem that the ”original-source” metatag was supposed to stop?

tyingq 5y ago
Canonical urls help with noting your own purposeful duplicated content. But that meta tag goes on the duplicated content. So it doesn't help with scrapers, who strip that out.
- lifeisstillgood 5y ago
  But I thought that it was useful for google - who could find two caches with same content, one of which was 2018 one of which 2020 and both say "this is canonical". At that point the 2018 version is real and the other rejected. Then again, you could just do it with publication dates ...
  - tyingq 5y ago
    I don't know why, but Google seems unable to figure out (or just doesn't care) "who published it first". I've seen it be confused many times.

tikiman163 5y ago

The reason you found so many domains is that they intentionally take down thier spam sites and reload them under a new domain every few hours. They do this so they can't be taken down by people reporting them as spam. They literally setup the next domain while the current one starts being used so they can do a live swap to the next one without interruptions to thier spam operations. This is typically done in an effort to spread Trojan malware to anybody running computers with out of date operating systems and browsers. Windows getting people off of Internet Explorer has been a huge hit for them as it reduces the number of possible vulnerabilities someone might have when they get sent to one of these Trojan spam sites.