23 ms·
GitHub investigating crypto-mining campaign abusing its server infrastructure
- woutr_be 6y agoWould anyone be able to estimate how much money the attackers would’ve made doing this?
- politelemon 6y agoDepending on when you ask this question, anywhere between $.02 and $200,000
- woutr_be 6y agoThe article says they’ve been doing this since November 2020. But I was mostly referring to how much crytpo they could really mine, since I assume the process has limited CPU, and a limited run time.
- thysultan 6y agodepends on the coin, but they're making 100% irregardless cause the cpu time is "free"
- junon 6y agoNot much at all. I don't know why they bother.
- richardfey 6y agoA wild guess: 1 cent every 20$ burned away on GitHub side
- NullPrefix 6y agoSounds like a VC funded startup
- toomanybeersies 6y agoLooking at one of the CI runs, they're mining between 2-4 cents of TRTL per 6 hour CI step, and running 98 steps per run. So around $2-4 per CI run. GH hosted CI runs use Azure DS2 v2 instances. I'm not sure how GH is billed by Microsoft for compute, but the 3 year reserved pricing is $0.0405/hour. So all up, that's around $2-4 of TRTL for $23.81 of compute. Plus there's the real and opportunity cost of needing to assign engineers to try and prevent this from happening.
- exdsq 6y agoIf I were them I'd mine unpopular coins hoping that they go up in value during a general bull run in the future. That way you're quite likely to end up with a decent number of coins.
- m00dy 6y agoUnpopular coins mostly dont have their own blockchain.
- vmception 6y agoThat... actually is true now. Anything with its own blockchain and also not popular is basically worthless The mining rewards over a few months would still be just a fraction of a few million dollar marketcap network But there are quite a few valuable ones you could get a decent take on, Tellor comes to mind.
- speedgoose 6y agoThe attacker would probably make much more money selling his time and expertise once in a while instead of doing bullshit like this.
- photon-torpedo 6y agoAnother comment from ahiknsr points out one repo affected: https://github.com/Bhargav1912/test_1/pull/2/files https://github.com/Bhargav1912/test_1/pull/2/files Following the files used reveals the pool and wallet used. The pool lets you look up the stats for this wallet. For this wallet, it says: > Total Paid: 163000 TRTL - 42.16$ But I guess the culprit is rotating wallets. Judging by the hash rate, the pool estimates a weekly income of 700$. It could be a multiple of this if the culprit uses multiple wallets at once.
- DelightOne 6y agoSo sad this is happening. Github Actions is a big quality and workflow improvement, I hope they get it under control without changing its model.
- ilkkao 6y agoDetecting mining jobs sounds like an interesting engineering challenge. Kernel has an OOM-killer, could it have a mining algorithm detector and killer too? Something that is hard to bypass. Detecting them while they are running would be the best approach I guess. Not after the time limit when the damage is already done.
- sahkopoyta 6y agoAFAIK, I don't think OOM-killer would help too much. Calculating hashes is not very memory intensive job
- luto 6y agoWe, a medium-sized hosting provider, just do it using the process name. Works much better than you'd expect, since most miners are using very similar software. No interesting engineering challenge here, sadly.
- plasma 6y agoWonder if you could check the repeated use of cryptographic functions in a process as a sign.
- ant6n 6y agoDo miners heavily rely on crush instructions? Then it could perhaps be possible to reduce the performance of these instructions 100-fold, making mining useless but other code still run.
- nostoc 6y agoAs a bonus, you'll also see ransomwares, disk encryption software, http servers, browsers... And what if the miner uses it's own crypto lib, and doesn't rely on the OS crypto API?
- mike_d 6y agoThis is pretty much par for the course if you allow others to run arbitrary code. I'd bet that Github deals with many cases of people trying to do this, and the only thing different here is a security researcher found it and talked to a reporter.
- junon 6y agoWhen I worked at ZEIT (Vercel) this was a huge issue we'd face. There was one person on GitHub that was shamelessly encouraging others to set up a docker image that ran crypto mining processes. This is inevitable, and I'm sure every CI system has faced this issue.
- faeyanpiraat 6y agoIs this against the TOS though?
- the_mitsuhiko 6y agoYes.
- vmception 6y ago:’D Very, most explicitly mention mining and an accurate description of it But its funny because ToS is not a deterrent and I thought you were being sarcastic, instead of inquisitive
- junon 6y agoYes. But that doesn't stop people.
- nodesocket 6y agoAre they mining with CPU’s or did GitHub actions add GPU’s recently? If just CPU’s that’s gotta be such a gigantic waste of CPU power for such little crypto.
- tinus_hn 6y agoIt doesn’t matter if the resource you are wasting is free.
- ForHackernews 6y ago"This will encourage Github to place its data centres near unused hydroelectric dams!" ~ Cryptokiddies, when you try to talk to them about waste.
- foepys 6y agoThis is a real talking point of crypto currency enthusiasts for people that didn't know and assume satire. They think that consuming more and more power results in more renewable energy being build. But because PoW algorithms create a static load, they siphon the energy away and when renewables cannot deliver enough power (e.g. no wind, no sun), coal and gas will be burned so blackouts don't happen.
- gtirloni 6y agoGitHub Actions' instances don't have a GPU [0]: "GitHub hosts Linux and Windows runners on Standard_DS2_v2 [1] virtual machines in Microsoft Azure" The only way to have access to a GPU is to use your own self-hosted runner. 0 - https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners https://docs.github.com/en/actions/using-github-hosted-runne... 1 - https://docs.microsoft.com/en-us/azure/virtual-machines/dv2-dsv2-series https://docs.microsoft.com/en-us/azure/virtual-machines/dv2-...
- ahiknsr 6y agoThey are using the CPU. This (https://github.com/Bhargav1912/test_1/pull/2/files https://github.com/Bhargav1912/test_1/pull/2/files) is one the PRs abusing the GitHub actions. It looks like they are mining turtlecoin.
- reader_mode 6y agoI was mostly ambivalent about crypto - it seemed like a lot of speculation but I liked some scenarios it enabled (mostly scenarios where traditional payment was not practical for w/e reason). But seeing the HW situation, energy burn, scammers, infra hijacking, etc. there are so many negatives I'm more and more in favour of making it illegal.
- loceng 6y agoThe SEC has always been in place supposedly to protect the average citizen from being able to make bad investments - however I don't understand how they dropped the ball so hard with crypto-"currencies;" regulatory capture by VC-finance industrial complex? Is it that they are global and decentralized that made them just not take a stance on them, also the same reason it obviously would easily and quickly gain popularity as a global, decentralized MLM-Ponzi structure?
- tarsinge 6y agoThe problem is what is illegal then? Numbers? Cryptography? Because nobody buy sell or own crypto technically, there just happen to be transactions on a distributed network, with some people knowing private keys to generate new transactions. It could be possible to regulate exchanges dealing with fiat, but I feel like the regulable surface area is thin. Also I hope Proof of stake take over so that a big part of these issues will be non existent in a few years.
- thefounder 6y agoCrypto without fiat exchange is worthless..am I wrong? Like I would start minting my own illegal currency.
- quickthrowman 6y ago> Crypto without fiat exchange is worthless..am I wrong? You’re not wrong, Walter.
- joaomacp 6y ago
- qqii 6y agoI asked this question previously[0] but why are random PRs ran by CIs by default without user intervention? It would make more sense that by default PRs aren't ran by default and an explicit whitelist or configuration option needs to be set. [0] in a discussion about the dangers of running your own CI server. I thought it were sensible if there was a permission config file which specified who can trigger which actions, where the CI uses the permissions of the last ran commit.
- hnnameblah365 6y agoI was surprised that Github Actions allows you to write scripts that anon users could trigger by opening PRs. But then it turns out this is how a lot of open source CI flows work. So maybe Github overlooked that they would become the biggest player overnight and be an obvious target? It's really strange to me. You would think anyone offering free anything on the internet knows there are people out there who will automate abusing your compute, storage, IPs, any resources you leave exposed. I've seen someone abuse a password reset form cuz they could advertise using that email addr '+' trick. (realuser+buy_our_scam@example.com). On a platform that didn't even have a million users. If it's free resources, there's no scam too unlikely.
- oefrha 6y agoWhat do you mean "by default"? By default workflows simply aren't run, not even when you push to master. You need to explicitly include the 'pull_request' or 'pull_request_target' event in the 'on' section of your workflow in order to build PRs. And you can implement a whitelist with reasonable ease if you want to. However, it turns out — who would have guessed — people do want to build random PRs, because 99% of PRs aren't jerks trying to mine crypto. --- Edit: To expand on the 'pull_request_target' event, it was introduced later and defends against running arbitrary unsanctioned workflows. It could have been introduced earlier, and maybe made more prominent, but there's a tradeoff.
- de6u99er 6y agoCryptomining is a waste of energy and compute resources. Imagine such an effort being put into protein folding, or finding a cure against rare diseases instead. What makes it worse is that a tax, higher energy or hardware costs, or increased transaction fees only speed up crypotocurrency inflation relative to other currencies. It's hard for me to comprehend, that others don't see the pyramid scheme behind it.
- ryebit 6y agoEthereum is working to address a lot of those points. When it switches to Proof of Stake (beacon chain is already live), energy costs will drop significantly (talking < 100 watts / node)... Just doing minimum to process blocks, no wasteful mining. That means network has to pay less in block reward under PoS, so inflation drops to 1% or less. Eip-1559 upgrade this summer will also start "burning" a portion of fees. Estimates have this around 0.5% - 1.5% deflation. Combine those features together, and Ethereum should have long term issuance at around 0% indefinitely, while still paying for validators to secure it. It's a feedback mechanism too... If usage goes down, burn drops, supply inflates slightly, stimulating use, increasing burn again. The idea is to make an elastic self-securing system. I don't understand where the pyramid scheme is in that... The money being paid out to validators will by nature barely cover costs (otherwise more people will validate, reducing margins to match). Since anyone can join, and the network punishes correlated misbehavior, that incentivizes decentralization and wide disbursement of block rewards. --- In the case of topic here, spare CPU cycles will no longer be valuable to exploit under PoS. The valuble qualities will be availability and uptime... Which won't be had by exploring in-broswer mining scripts, or CI exploits.
- macspoofing 6y ago>Cryptomining is a waste of energy and compute resources. I have my objections to crypto currencies, but this is not one of them. Most everything we do as humans is a 'waste' if you want to get philosophical. Blockchain is an interesting technology. It may or may not go anywhere, but there is value in thousands of people 'playing' with it to see what's possible. That is, there is value in exploring the solution space of the Blockchain because there is possibility in finding some local or global efficiency minima. Think of it as a R&D investment that may or may not pay off. Still too early to tell. Besides, the energy cost of the Blockchains is not and will not be the determining factor in fight against climate change or any other environmental concern. So the last thing you want is government bureaucracies cracking down on it from that angle. Let people explore and play and see where it goes. Having said that, if crypto mining does start to impact other infrastructure, regulators may have to look at it. And there's precedent for that kind of action. For example, regulators will typically work against silver speculation because silver is also an industrial product and if the price inflates to much, it will have deleterious effects on other industries and the wider economy. >It's hard for me to comprehend, that others don't see the pyramid scheme behind it. There is that aspect of it and most recognize it. When crypto intersects money and acts as an investment, it should be regulated because there are a lot of nefarious individuals who are using crypto currencies to swindle money from people. For example, my next door neighbor tried to enlist me to buy into some crypto scheme recently. It took me 5 mins of googling around to see how incredibly risky that 'sure-bet' is [1]. I'm pretty sure he sunk tens of thousands of his money into it and I'm pretty sure he doesn't quite understand what it is that these scammers are offering him (basically he's trading in crypto, that he had to buy, that is popular and therefore relatively liquid - i.e. something you can actually sell - for crypto that is not and has a good chance of collapsing in the near to mid future). Regulations will help in this area. [1]https://behindmlm.com/mlm-reviews/hypercapital-review-hypercash-resurrection-ponzi-scheme/ https://behindmlm.com/mlm-reviews/hypercapital-review-hyperc...
- smarx007 6y agoI think it could be solved by requiring maintainers to kick off the CI job for every PR coming from a first-time contributor manually. I would not be annoyed by this provided that regular contributors don’t have to go through this. I think GH can further reduce the impact of this by not applying the limitation to first-time contributors who have lots of followers, forks, PR reviews, commits in the last year. Also, edits to the CI script could be made suspect. I never had a first-time contributor on my projects start by making changes to the CI pipeline. I am pretty sure smart folks at GH thought of this and just deliberating whether to introduce such a breaking flow to maintainers.
- hrpnk 6y agoHaving the maintainer trigger the CI job moves the problem elsewhere. Attackers will create new repositories (or take over existing ones) and create the PRs, for which they will trigger the CI jobs. The more apparent problem is that CI jobs can execute arbitrary code and are not limited wrt. their execution time. If limited, it would render them useless when used for cryptomining.
- foepys 6y agoEven just running for 10 minutes is enough. It's not fast, it's not efficient but it's free and thus worth it if you can automate it.
- tomasreimers 6y agoI think this is close but possibly insufficient. What's to stop attackers from making a one-off harmless edit ("forgot a comma in readme") and then, once they're whitelisted, deploying a malicious executable to the CI pipeline. I think the root issue is that people without write access to your repo can queue arbitrary compute on your dime by simply creating a PR and changing the GitHub workflow files (the definition for GitHub actions). This is even a bigger issue for companies with self-hosted runners who can't use those for public repos as an attacker could file a PR with malicious code and compromise a machine (https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#self-hosted-runner-security-with-public-repositories https://docs.github.com/en/actions/hosting-your-own-runners/...) One possible solution here is that a maintainer needs to okay any change that changes a workflow file before it runs. Not sure if that introduces other problems...
- ddevault 6y agoWe have to deal with this from time to time on sourcehut, too, mostly recently last night.
- knorker 6y agoYou can no longer have a freebie compute tier on ANYTHING, because the cryptocriminals swoop in and abuse 100% of it. This is why we can't have nice things.
- williamstein 6y agoAs somebody who has been creating or supporting websites for mathematicians for two decades that support running arbitrary code (e.g., Pari/Magma calculator, Sage notebook, Sage Cell server, CoCalc), this is very much the case. In fact, https://sagecell.sagemath.org/ https://sagecell.sagemath.org/ finally had to be locked down much more in the last few days due to abuse by cryptominers.
- sydd 6y agoYep, I know that at least 2 of large online CI providers are facing the same issue with their free tier.
- deleted 6y ago[deleted]
- sodality2 6y agoGoogle, AWS, oracle, etc all have free tiers
- knorker 6y agoAnd entire departments fighting it, and $100B in the bank to chalk it down to cost of doing business. But yes, it's not completely gone yet.
- gonzo41 6y agoWouldn't it be great if Crypto was just a giant honey pot. With the intent of creating cyber criminals who think they can get rich with crypto money but instead just end up putting tons of effort into mining coins rather than doing other simpler crimes. I think if half the effort spent here was spent learning finance, the bad actors would have more money.
- GordonS 6y agoI seem to recall this was a problem early on Azure DevOps - now Microsoft owns GitHub, I would have thought they'd have knowledge to bring here. Hope they manage to sort it out - and also hope these arseholes doing the damage don't ruin GitHub Actions for the rest of us.
- damsta 6y agoWe are one step closer to having GitHub Actions limited or disabled entirely on free accounts. Really soon it is going to be available on paid plans and for open-source after an approval.
- smarx007 6y agoApproval will do nothing. It's not projects who are abusing actions, but PR submitters! This is actually a much bigger risk to the PR model popularised by GH than GH would like to admit. Now the git-send-email model does not look so wrong anymore.
- natfriedman 6y agoThis is a cat and mouse game. We add code to detect and disable abuse – sometimes in very clever ways – and then the abusers come up with a new way of circumventing that detection. In order to prevent miners from creating long queues for legitimate free users of GitHub Actions, we have to stay on top of this all the time. So the miners are not just stealing CPU time, they are also stealing engineer time. Because without mitigations the miners will consume all available CPU, and because devising abuse countermeasures is, for whatever reason, a very powerful nerd snipe (including for me!). The sad thing is that it's displacing time that would be spent improving Actions in other ways.
- anoojb 6y agoThanks Nat. What if we made new GitHub Actions temporarily only available to users with a verified second factor? Could temporarily reduce the population of abusers while we figure out a more sustainable strategy?
- chatmasta 6y agoA TOTP code response is trivial to implement on the client. So if you wanted this to be meaningful, you would need to force users to use SMS 2FA, which is widely considered insecure. Not a great solution IMO.
- deleted 6y ago[deleted]
- sorbits 6y agoDo I understand correctly that the attacker forks a repository with GitHub Actions enabled, modifies the action, submits a PR, which makes GitHub run the altered action? If so, I wonder if there is a legit need for running modified GitHub actions from non-collaborators? Could also subject modified actions coming in via pull requests (from non-collaborators) to heavy resources constraints and timeouts.
- natfriedman 6y ago
- lmeyerov 6y agoThis has been a frustrating thing for us on GitHub , not because of crypto, but for stealing credentials. It's hard to make it so only approved people can run actions and lock down what exposed actions' runtimes can do. Until it is default-deny for ~all capabilities with say RBAC for enabling, which are basic security principles, they're pretty scary for public repos. It's even scary for private ones as you might want say a contractor or intern to be limited. That they have people publicly dedicated to whackamole response, but seemingly not to security fundamentals (or if they do, not following them / empowered to), is frustrating. Look at the GHA permissions panel and then think like an attacker or defender, it's scary. The environment variable stuff awhile back was understandable.. but not the big issue. As basically any user can submit a PR that can make current actions that run code (ex: run tests) to run something else by editing the action or code, that means any public user can burn repo $, play in repo-exposed runners (ex: get into corp sandboxes), and if using continuous deployment or service integrations , into their production systems, wherever they publish packages, etc. GitHub is api exposed and git can have commits deleted, so they can even cover most of their tracks. This is SolarWinds all over again, but now a bot can run it automatically on ~everyone! GHA is both one of my favorite things about GitHub but also been scary. Maybe now that they have access to some. of the best security engineers and researchers in the world, they can fix this...
- smarx007 6y agoFrom https://github.com/../../settings/secrets/actions https://github.com/../../settings/secrets/actions "Secrets are not passed to workflows that are triggered by a pull request from a fork."
- lmeyerov 6y agoYeah and afaict actions on fork PR's are off by default. This is akin to fixing a sinking ship by plugging your fingers into holes as they appear, not how you build secure systems: default deny on all logical+physical capabilities, central authorization policies for grants, defense in depth for layering those.
- 6y ago
- sneak 6y agoThis, to me, isn't news. Any platform that permits free arbitrary code execution (so, Netlify, Cloudflare Pages, GitHub Actions, DockerHub automated builds, CircleCI, et al) that can access the network can be assumed to have been abused in this manner.
- lanstin 6y agoI think blocking network access is the simplest fix. Just make the payor specify a whitelist. Email them the blocked attempts. You can run the white list checker on a c3a.large for a pretty big CI.
- amasad 6y agoWe’ve been dealing with mining at Replit since the beginning and have gotten really good at handling it. Early on they nearly bankrupt us. There was one month when our compute bill was 500% more than average. So we just had build tooling to catch it and handle it early. It’s a fun problem because it’s not only technical but also psychological. Mining is one vector of abuse but there are many others when you’re giving free compute to the world. Especially in the radically open way we’re doing it at Replit.
- aleksi 6y agoA similar thing happened to me four years ago: https://www.vice.com/en/article/nzkxgm/bitcoin-mining-github-open-source-bots https://www.vice.com/en/article/nzkxgm/bitcoin-mining-github...
- Johnyma22 6y agoThank you for all the free computing. <3 Etherpad team
- peanut_worm 6y agoGitHub actions is a very useful tool but it must be such a nightmare to deal with on GitHub's end.
- CynicusRex 6y agoObligatory mention: The Political Case for a Blanket Cryptocurrency Ban, https://www.stephendiehl.com/blog/banbitcoin.html https://www.stephendiehl.com/blog/banbitcoin.html.
- ada1981 6y agoAny word on how much has been mined this way?