5 ms·

So sad this is happening. Github Actions is a big quality and workflow improvement, I hope they get it under control without changing its model.

ilkkao 6y ago
Detecting mining jobs sounds like an interesting engineering challenge. Kernel has an OOM-killer, could it have a mining algorithm detector and killer too? Something that is hard to bypass. Detecting them while they are running would be the best approach I guess. Not after the time limit when the damage is already done.
- sahkopoyta 6y ago
  AFAIK, I don't think OOM-killer would help too much. Calculating hashes is not very memory intensive job
- luto 6y ago
  We, a medium-sized hosting provider, just do it using the process name. Works much better than you'd expect, since most miners are using very similar software. No interesting engineering challenge here, sadly.
  - plasma 6y ago
    Wonder if you could check the repeated use of cryptographic functions in a process as a sign.
    - ant6n 6y ago
      Do miners heavily rely on crush instructions? Then it could perhaps be possible to reduce the performance of these instructions 100-fold, making mining useless but other code still run.
    - nostoc 6y ago
      As a bonus, you'll also see ransomwares, disk encryption software, http servers, browsers... And what if the miner uses it's own crypto lib, and doesn't rely on the OS crypto API?
  - londons_explore 6y ago
    Ssssh.... If you tell everyone that's what you do, people will just rename the binary... The trick to protections like this is to not tell anyone how they work, and to run them only occasionally. Ie. once a week, ban half of users who are running xmrig.exe. Also include users who signed up with the same email address, phone number or IP address as the detected users and who have a consistently high CPU use - these are probably successful bypasses of your simple process name based filter. That way bad actors have a very hard time figuring out exactly what your protections are or how they work. If they were to get an immediate ban as soon as they fired up xmrig.exe, then they'd quickly think to rename it or recompile it or run it under wine or a host of other ideas. Yet having a random selection of their accounts banned seemingly at random means they learn nothing. Obviously you need a process for users accidentally caught in the net to get their accounts reactivated, and if you're a service like githuib you should probably let the user have a grace period to do that before killing their entire business...
    - luto 6y ago
      Good luck finding that post and linking it to the provider. That set aside, except for DDoS defense, we've run into no problems at all when talking very openly about how we operate. Seems naive at first glance, but we've had a good 10+ year run (so far!). Works for us, might not for others.
      - faeyanpiraat 6y ago
        Delayed banning seems to work well for multiplayer game anti cheat systems (like VAC). These problems are essentially the same, some of the know-how can be easily adapted between them.
      - thaumasiotes 6y ago
        > Good luck finding that post and linking it to the provider. No need to link it to the provider. If one provider does things that way, you want to block their method. (And, of course, the odds are overwhelming that the other providers are doing the same thing.)
      - 6y ago
  - jfoster 6y ago
    You could also just address it through your rate table by appropriately pricing for intense workloads, couldn't you?
    - luto 6y ago
      We're offering shared hosting for web apps and other hacker/toy projects. Intense compute workloads are just not what our product is built for, so we let other providers handle that part of the market.
      - jfoster 6y ago
        That's what I mean. So make it very expensive to run them on your service and the coin miners will go elsewhere.
        luto 6y ago
        We're aiming to make our service accessible to everyone, which involves customers choosing their own price. So there isn't a fixed price we could raise. Additionally, there is a free trial month, which adversaries typically make use of, instead of paying for the service. Hiking he price without deep additional changes is not in our interest and wouldn't change anything in our case.
- raverbashing 6y ago
  CPU throttling is probably your best bet I'd say. Watch for things pegging the CPU Most of build actions is not 100% CPU bound But yeah, abusers are going to abuse
  - NullPrefix 6y ago
    > Most of build actions is not 100% CPU bound Unless you compile large C/C++ projects on a low core count VM.
- xnbya 6y ago
  RandomX is quite easily detectable due to the unusual use of floating point operations. Proof of concept detectors have already been created, for example https://github.com/tevador/randomx-sniffer https://github.com/tevador/randomx-sniffer
- Tenoke 6y ago
  Google Colab supposedly has great miner detection for example. I Googled out of curiosity recently and couldn't find any clearnet mentions of anyone bypassing it in the recent past and the old ways don't work. Though it must be possible it at minimum doesn't seem to be easy.
  - williamstein 6y ago
    Google Cloud is similar (perhaps the same). I run sites there that involve arbitrary code execution by our users, and periodically Google support contacts us to report cryptomining violations. I don't know exactly what Google's heuristics are for triggering this, but their reports to us typically include the ip address of a mining pool that was contacted by one of our virtual machines. My impression is that mining typically involves a network connection.
    - slig 6y ago
      Aren't you afraid of running such service on Google infrastructure and getting the ban hammer from Google?
      - Tenoke 6y ago
        Does it even matter to GCloud (a paid service) if I use 100% of the GPU for Deep Learning or for mining? They operate on profit either way.
        slig 6y ago
        It does, I think, for the same reason GPU manufactories segment GPUs for gaming and for Deep Learning.
        williamstein 6y ago
        They also care because in practice such use is often done by attackers unbeknownst to the person paying for the resources. That would much rather make the person paying aware ASAP of such usage, rather than have them be on the hook for a large bill at the end of the month. Google does have a complicated process to allow cryptomining on their infrastructure. They also flag all kinds of other suspicious activity, eg once somebody was launching DOS attack from cocalc on “the country of Turkey” and that got some of cocalc shutoff immediately.
rurban 6y ago
What's there to get under control? It cannot get out of control. GH Action have a timeout of 60min already. A PR with about 5 jobs running for 5 hrs is nothing to gain for the culprit. The repo owner certainly finds out soon enough, and reports it at GitHub to block him. GitHub doesn't even need to start a mass scan for such losers. Azure is such a huge server farm, nobody should care about a few 1hr miners, who get eventually thrown out. But the easiest mitigation would be up block outbound traffic to the miners IPs. These are well-known.

ilkkao 6y ago

Detecting mining jobs sounds like an interesting engineering challenge. Kernel has an OOM-killer, could it have a mining algorithm detector and killer too? Something that is hard to bypass. Detecting them while they are running would be the best approach I guess. Not after the time limit when the damage is already done.

sahkopoyta 6y ago
AFAIK, I don't think OOM-killer would help too much. Calculating hashes is not very memory intensive job
luto 6y ago
We, a medium-sized hosting provider, just do it using the process name. Works much better than you'd expect, since most miners are using very similar software. No interesting engineering challenge here, sadly.
- plasma 6y ago
  Wonder if you could check the repeated use of cryptographic functions in a process as a sign.
  - ant6n 6y ago
    Do miners heavily rely on crush instructions? Then it could perhaps be possible to reduce the performance of these instructions 100-fold, making mining useless but other code still run.
  - nostoc 6y ago
    As a bonus, you'll also see ransomwares, disk encryption software, http servers, browsers... And what if the miner uses it's own crypto lib, and doesn't rely on the OS crypto API?
- londons_explore 6y ago
  Ssssh.... If you tell everyone that's what you do, people will just rename the binary... The trick to protections like this is to not tell anyone how they work, and to run them only occasionally. Ie. once a week, ban half of users who are running xmrig.exe. Also include users who signed up with the same email address, phone number or IP address as the detected users and who have a consistently high CPU use - these are probably successful bypasses of your simple process name based filter. That way bad actors have a very hard time figuring out exactly what your protections are or how they work. If they were to get an immediate ban as soon as they fired up xmrig.exe, then they'd quickly think to rename it or recompile it or run it under wine or a host of other ideas. Yet having a random selection of their accounts banned seemingly at random means they learn nothing. Obviously you need a process for users accidentally caught in the net to get their accounts reactivated, and if you're a service like githuib you should probably let the user have a grace period to do that before killing their entire business...
  - luto 6y ago
    Good luck finding that post and linking it to the provider. That set aside, except for DDoS defense, we've run into no problems at all when talking very openly about how we operate. Seems naive at first glance, but we've had a good 10+ year run (so far!). Works for us, might not for others.
    - faeyanpiraat 6y ago
      Delayed banning seems to work well for multiplayer game anti cheat systems (like VAC). These problems are essentially the same, some of the know-how can be easily adapted between them.
    - thaumasiotes 6y ago
      > Good luck finding that post and linking it to the provider. No need to link it to the provider. If one provider does things that way, you want to block their method. (And, of course, the odds are overwhelming that the other providers are doing the same thing.)
    - 6y ago
- jfoster 6y ago
  You could also just address it through your rate table by appropriately pricing for intense workloads, couldn't you?
  - luto 6y ago
    We're offering shared hosting for web apps and other hacker/toy projects. Intense compute workloads are just not what our product is built for, so we let other providers handle that part of the market.
    - jfoster 6y ago
      That's what I mean. So make it very expensive to run them on your service and the coin miners will go elsewhere.
      - luto 6y ago
        We're aiming to make our service accessible to everyone, which involves customers choosing their own price. So there isn't a fixed price we could raise. Additionally, there is a free trial month, which adversaries typically make use of, instead of paying for the service. Hiking he price without deep additional changes is not in our interest and wouldn't change anything in our case.
raverbashing 6y ago
CPU throttling is probably your best bet I'd say. Watch for things pegging the CPU Most of build actions is not 100% CPU bound But yeah, abusers are going to abuse
- NullPrefix 6y ago
  > Most of build actions is not 100% CPU bound Unless you compile large C/C++ projects on a low core count VM.
xnbya 6y ago
RandomX is quite easily detectable due to the unusual use of floating point operations. Proof of concept detectors have already been created, for example https://github.com/tevador/randomx-sniffer https://github.com/tevador/randomx-sniffer
Tenoke 6y ago
Google Colab supposedly has great miner detection for example. I Googled out of curiosity recently and couldn't find any clearnet mentions of anyone bypassing it in the recent past and the old ways don't work. Though it must be possible it at minimum doesn't seem to be easy.
- williamstein 6y ago
  Google Cloud is similar (perhaps the same). I run sites there that involve arbitrary code execution by our users, and periodically Google support contacts us to report cryptomining violations. I don't know exactly what Google's heuristics are for triggering this, but their reports to us typically include the ip address of a mining pool that was contacted by one of our virtual machines. My impression is that mining typically involves a network connection.
  - slig 6y ago
    Aren't you afraid of running such service on Google infrastructure and getting the ban hammer from Google?
    - Tenoke 6y ago
      Does it even matter to GCloud (a paid service) if I use 100% of the GPU for Deep Learning or for mining? They operate on profit either way.
      - slig 6y ago
        It does, I think, for the same reason GPU manufactories segment GPUs for gaming and for Deep Learning.
      - williamstein 6y ago
        They also care because in practice such use is often done by attackers unbeknownst to the person paying for the resources. That would much rather make the person paying aware ASAP of such usage, rather than have them be on the hook for a large bill at the end of the month. Google does have a complicated process to allow cryptomining on their infrastructure. They also flag all kinds of other suspicious activity, eg once somebody was launching DOS attack from cocalc on “the country of Turkey” and that got some of cocalc shutoff immediately.

rurban 6y ago

What's there to get under control? It cannot get out of control. GH Action have a timeout of 60min already. A PR with about 5 jobs running for 5 hrs is nothing to gain for the culprit. The repo owner certainly finds out soon enough, and reports it at GitHub to block him. GitHub doesn't even need to start a mass scan for such losers. Azure is such a huge server farm, nobody should care about a few 1hr miners, who get eventually thrown out. But the easiest mitigation would be up block outbound traffic to the miners IPs. These are well-known.