10 ms·

Nocom – 2b2t Minecraft server exploit using Monte-Carlo localization

208 points by cyber_kinetist 5y ago

cyber_kinetist 5y ago
Here's the FitMC Youtube video that first announced the exploit: https://www.youtube.com/watch?v=elqAh3GWRpA https://www.youtube.com/watch?v=elqAh3GWRpA A 2b2t wiki page about the exploit: https://2b2t.miraheze.org/wiki/Nocom https://2b2t.miraheze.org/wiki/Nocom The teaser video: https://www.youtube.com/watch?v=3ayxeruAan8 https://www.youtube.com/watch?v=3ayxeruAan8
- 323 5y ago
  I second the first video, it's extremely well done, and as someone who never played minecraft I almost understood what happened.
- bastardoperator 5y ago
  "2b2t ,the oldest anarchy server in Minecraft" - FitMC (every video)
H8crilA 5y ago
This is from the same guy who coded Baritone: https://youtu.be/CZkLXWo4Fg4?t=78 https://youtu.be/CZkLXWo4Fg4?t=78 Extremely useful on anarchy servers, in fact the entire current "era" of anarchy is called "The Automation Period" mostly due to Baritone.
- leijurv 5y ago
  I actually helped convince Sato in Feb 2020 to call it "automation period"; some other ideas were "age of fragmentation" (referring to how group alleigances were becoming split, with players in more than one group at a time) or "age of the algorithm" (referring to the YouTube algorithm promoting the server).
leijurv 5y ago
Surprised to see this posted again, I wrote this, feel free to ask me anything!
- snowycat 5y ago
  Really impressive project! How did you learn all the things that you needed to make this? Are there any particular books or resources you would recommend? Also, how much did this entire project cost? I'd imagine processing that much data and storing it wasn't cheap.
  - leijurv 5y ago
    Mostly learned as I went! I had never before implemeneted Monte Carlo localization, or DBSCAN, or a disjoint union find; it was a lot of research and trial and error. I'd honestly recommend Wikipedia! I also read a lot of blog posts such as on data clustering in order to decide how to write an algorithm to tell what is and isn't a single contiguous base. The main expense was priority queue, which is a $20/account/month payment to 2b2t to be able to join the server faster. Probably nearing a thousand dollars on that. We lightened this by creating a proxy system where other builders could join through our headless, and it would add nocom exploit packets from their account to 2b2t, and peel off the responses on the way back. They got a more reliable experience with less disconnects, and we got some extra capacity for free (completely unbeknownst to them of course haha). By the end, all the overworld accounts were being paid for and used by other people, I only had to cover the 2 accounts in the nether. About $300 on a new 2TB nvme drive for postgres. An unknown amount on electricity. Other than that, all the big processing ran on an existing homelab server (thanks to fr1kin). About $150 on digitalocean for a droplet to run the headless minecraft with minimal ping to 2b2t ($10/month). Backups were just WAL rsyncs to postgres replicas ran (again on homelabs) by myself and a few others.
- ucarion 5y ago
  I'm curious what kind of tooling you developed as part of this process. It seems, from the linked YouTube video, like individuals were actively monitoring the data produced from the exfiltration attack. How did you go from the raw data to producing something that people could usefully monitor?
the_gipsy 5y ago
This is brilliant! Ah, gives me nostalgia of civcraft conspiracies...
- thegjp210 5y ago
  good to see some other civcraft alumni on HN. What an experience...
  - iFred 5y ago
    Probably the only server experience that really felt like you had an entire world and society to explore and consequences to deal with. I spent way too much time trying to figure out a way to scrounge up Ghasts and TNT to level Orion.
meibo 5y ago
What confuses me about this is that no one else found it? Being somewhat experienced at game networking and writing server code with this type of exploit(e.g. sending a packet to do something at the other end of the map and processing the result) in mind, this would be something I either would have checked for in the first place if performance would have allowed for it, or would have kept in the back of my mind in case I hear about a possible exploit that takes advantage of it.
- leijurv 5y ago
  One reason might be that this only existed in Paper, which is a downstream fork of Spigot, which is (sort of) a downstream fork of CraftBukkit, which is a set of patches to the official minecraft server. Another reason might be that this only does anything if there are many players on the server, and you hit a region loaded by someone else. Might be difficult to think of that. You'd never come across it if testing in singleplayer. But in essence I agree, it's surprising that no one else found and reported this years ago!
- Sesse__ 5y ago
  What confuses _me_ about this is that nothing seems to say what it _is_. It's like there's some giant practical joke I'm not a part of, and the README.md just throws off heatmaps and wants me to watch a 24-minute YouTube video. Is everybody supposed to be a Minecraft expert? Is a simple paragraph of what this is all about (at the very beginning of the file) too much? :-)
  - leijurv 5y ago
    There's a section called "The entire thing summed up in three sentences" in the linked GitHub README.
  - bobsmooth 5y ago
    "You could punch any block coordinate in the entire map, and 2b2t would either: -Tell you what block is placed there, if the chunk is currently loaded -Or ignore you, if the chunk is not currently loaded. 2b2t would immediately reveal whether or not any chunk coordinate that you ask about is currently loaded. If it is currently loaded, it tells you which specific block is there at the coordinate you asked about." This allowed the hackers to find every player's coordinates and where their bases were.
cityofdelusion 5y ago
If anyone else was lost like myself, here is the rundown. The linked github readme assumes prior knowledge of 2b2t. This is an exploit for a very old Minecraft server (called 2b2t) where griefing (destroying other creations) and hacking is basically allowed. This kind of environment means that groups of players are incentivized to hide their projects out in the middle of nowhere, hundreds of thousands of blocks away from the world origin coordinates. The exploit allows a client to ask the server if data chunks at arbitrary coordinates have player activity in them, opening the server up to a brute force search to locate well-hidden bases. Basically, a group got spy satellites in an anarchy Minecraft server where remaining hidden is critical to not being destroyed.
- kingcharles 5y ago
  Thank you. I had no idea what was going on. This is actually pretty neat once you explain it.
- Bluecobra 5y ago
  This is a pretty clever exploit. I played on this server off and on between 2013-2016. The server is/was an acquired taste. The biggest challenge was to escape the the hellscape around spawn, once you get past that it was pretty quiet, I don’t ever recall running into anyone else unless I went back to the nether hub near spawn. It was really fun exploring old builds, the griefed ones were like old ruins. I liked to read the signs that were left behind and leave my own. I used to have use sone map mod and would note all the neat things I came across. I admit that I also used an x-ray mod to detect things like chests, because why not? I wonder if any of my bases were ever discovered. Most of my ill gotten gains were hidden away in a bottom of a mineshaft in the middle of nowhere to help avoid detection. Once the queue started started to get long I stopped playing.
- rvnx 5y ago
  In Minecraft you can dig blocks. When you click to dig a block, it just calls the RPC method CPacketPlayerDigging with the coordinates of the block. The server replies "Block of type ABC is too far". So, of course, you now know that at (X,Y,Z) you have a block of type "ABC". This RPC has no rate-limiting, so yes, you can know the whole map. The fact that you can dig any block on the map, seems like a shortcut taken by developers of the server to save time or resources. I'd even say it's almost intended behaviour. Tech details: https://2b2t.miraheze.org/wiki/Nocom https://2b2t.miraheze.org/wiki/Nocom
TheJoeMan 5y ago
I learned some of the Monte Carlo techniques in my medical imaging classes, it’s very interesting the data one can fish out with a good-enough graph. I also find the adaptive location “pinging” quite interesting, reminds me of the tech behind the F-22 active radar.
bobsmooth 5y ago
My favourite part of this has to be the social engineering used to get the exploit patched in.
kregasaurusrex 5y ago
The code used in the exploit itself itself looks rather harmless, as it was created to be a mitigation against finding players based on previously loaded chunks. https://github.com/PaperMC/Paper/blob/ver/1.12.2/Spigot-Server-Patches/0196-Fix-block-break-desync.patch https://github.com/PaperMC/Paper/blob/ver/1.12.2/Spigot-Serv...
- leijurv 5y ago
  That is not correct, this patch is meant to prevent block break desync, which is when your client and the server disagree about what blocks are placed in the world. If the server discards/ignores anything your client sends, the server MUST reply telling the client to undo that action, otherwise desync occurs. (e.g. "i have broken this block" -> the client is too far to reach that block -> the server has to reply with "set that block back to air"). The original version of this patch https://github.com/PaperMC/Paper/blob/d934dcc1e3e8b1ac34acef1b72c690229ebb018e/Spigot-Server-Patches/0196-Fix-block-break-desync.patch https://github.com/PaperMC/Paper/blob/d934dcc1e3e8b1ac34acef... fixed that desync, but it allowed you to force the server to generate an unlimited amount of chunks, which is an expensive operation. You could only find players using this patch after it was modified to only reply if the chunk was already loaded. But at no point in time was this patch able to reveal if a chunk had been previously generated or not, that is a separate unrelated issue in which Spigot will reliably send a partial chunk on initial generation, but never for a previously generated chunk. Take a look at https://2b2t.miraheze.org/wiki/Nocom https://2b2t.miraheze.org/wiki/Nocom
  - kregasaurusrex 5y ago
    Thanks for the clarification!
w-ll 5y ago
Why would the `mineblock` packet not be clamped to at least the render distance or more realistic the chunk the sender is in and the 8 adjacent chunks. I know i'm learning about this in hindsight and it's obvious.
- leijurv 5y ago
  Yes, so Minecraft is exactly that. Any "I have mined this block" further than 6 blocks away from the player is ignored. The problem is that on a laggy server, an honest client can end up sending this, if the server lags out for more than 6 blocks of walking your player forward. So, Paper added a patch to undo these disallowed mines, setting them back to what they were. But this happened at any radius, and could be used to lag the server by making it generate any region. Then they patched it again and made it only reply if it were in a loaded chunk. The mistake was that it replies if the coordinate is in a chunk loaded by any player, not just by your player. :) See https://news.ycombinator.com/item?id=29620852 https://news.ycombinator.com/item?id=29620852
  - w-ll 5y ago
    Hey thanks for your reply. I get how the Paper patch allowed this new exploit, but Im still curious to why Paper never even had the range check. Even for lag compensation it shouldnt be more than a chunk or 2 range. Im actually reminded to the latest log4j exploit (main reason i aint actually gonna run Minecraft for a while) and even the Heartbleed exploit a few years back. Did yall assume the patch to Paper would just return if the chunk was player loaded, and they wouldn't add any other sane bounds checks? If the story from the FitMC guy is true, yall guessed/hoped that patch would be what it was. Giving yall a new leak of information without actually fixing the problem because if it was loaded you could still get all the block info in that chunk. Pretty awesome from where i'm sitting. \(ᵔᵕᵔ)/
    - leijurv 5y ago
      Well, Paper is trying to fix the result of vanilla Minecraft's range check. Taking their perspective, they wanted to fix the issue where <6 blocks worked, but >6 blocks caused desync. So it is natural to reply in the "else" block with "no you can't break that". They just didn't think of adding yet another range check with a looser limit on top of that. I wasn't around for when the patch was engineered into Paper, but from what I'm told, yes that was the idea. :)
howenterprisey 5y ago
https://2b2t.miraheze.org/w/index.php?title=Space_Valkyria_3_V2&diff=next&oldid=58707 https://2b2t.miraheze.org/w/index.php?title=Space_Valkyria_3... Wow... I love this edit so much.
- leijurv 5y ago
  Thanks!

Here's the FitMC Youtube video that first announced the exploit: https://www.youtube.com/watch?v=elqAh3GWRpA https://www.youtube.com/watch?v=elqAh3GWRpA A 2b2t wiki page about the exploit: https://2b2t.miraheze.org/wiki/Nocom https://2b2t.miraheze.org/wiki/Nocom The teaser video: https://www.youtube.com/watch?v=3ayxeruAan8 https://www.youtube.com/watch?v=3ayxeruAan8

323 5y ago
I second the first video, it's extremely well done, and as someone who never played minecraft I almost understood what happened.
bastardoperator 5y ago
"2b2t ,the oldest anarchy server in Minecraft" - FitMC (every video)

H8crilA 5y ago

This is from the same guy who coded Baritone: https://youtu.be/CZkLXWo4Fg4?t=78 https://youtu.be/CZkLXWo4Fg4?t=78 Extremely useful on anarchy servers, in fact the entire current "era" of anarchy is called "The Automation Period" mostly due to Baritone.

leijurv 5y ago
I actually helped convince Sato in Feb 2020 to call it "automation period"; some other ideas were "age of fragmentation" (referring to how group alleigances were becoming split, with players in more than one group at a time) or "age of the algorithm" (referring to the YouTube algorithm promoting the server).

leijurv 5y ago

Surprised to see this posted again, I wrote this, feel free to ask me anything!

snowycat 5y ago
Really impressive project! How did you learn all the things that you needed to make this? Are there any particular books or resources you would recommend? Also, how much did this entire project cost? I'd imagine processing that much data and storing it wasn't cheap.
- leijurv 5y ago
  Mostly learned as I went! I had never before implemeneted Monte Carlo localization, or DBSCAN, or a disjoint union find; it was a lot of research and trial and error. I'd honestly recommend Wikipedia! I also read a lot of blog posts such as on data clustering in order to decide how to write an algorithm to tell what is and isn't a single contiguous base. The main expense was priority queue, which is a $20/account/month payment to 2b2t to be able to join the server faster. Probably nearing a thousand dollars on that. We lightened this by creating a proxy system where other builders could join through our headless, and it would add nocom exploit packets from their account to 2b2t, and peel off the responses on the way back. They got a more reliable experience with less disconnects, and we got some extra capacity for free (completely unbeknownst to them of course haha). By the end, all the overworld accounts were being paid for and used by other people, I only had to cover the 2 accounts in the nether. About $300 on a new 2TB nvme drive for postgres. An unknown amount on electricity. Other than that, all the big processing ran on an existing homelab server (thanks to fr1kin). About $150 on digitalocean for a droplet to run the headless minecraft with minimal ping to 2b2t ($10/month). Backups were just WAL rsyncs to postgres replicas ran (again on homelabs) by myself and a few others.
ucarion 5y ago
I'm curious what kind of tooling you developed as part of this process. It seems, from the linked YouTube video, like individuals were actively monitoring the data produced from the exfiltration attack. How did you go from the raw data to producing something that people could usefully monitor?

the_gipsy 5y ago

This is brilliant! Ah, gives me nostalgia of civcraft conspiracies...

thegjp210 5y ago
good to see some other civcraft alumni on HN. What an experience...
- iFred 5y ago
  Probably the only server experience that really felt like you had an entire world and society to explore and consequences to deal with. I spent way too much time trying to figure out a way to scrounge up Ghasts and TNT to level Orion.

meibo 5y ago

What confuses me about this is that no one else found it? Being somewhat experienced at game networking and writing server code with this type of exploit(e.g. sending a packet to do something at the other end of the map and processing the result) in mind, this would be something I either would have checked for in the first place if performance would have allowed for it, or would have kept in the back of my mind in case I hear about a possible exploit that takes advantage of it.

leijurv 5y ago
One reason might be that this only existed in Paper, which is a downstream fork of Spigot, which is (sort of) a downstream fork of CraftBukkit, which is a set of patches to the official minecraft server. Another reason might be that this only does anything if there are many players on the server, and you hit a region loaded by someone else. Might be difficult to think of that. You'd never come across it if testing in singleplayer. But in essence I agree, it's surprising that no one else found and reported this years ago!
Sesse__ 5y ago
What confuses _me_ about this is that nothing seems to say what it _is_. It's like there's some giant practical joke I'm not a part of, and the README.md just throws off heatmaps and wants me to watch a 24-minute YouTube video. Is everybody supposed to be a Minecraft expert? Is a simple paragraph of what this is all about (at the very beginning of the file) too much? :-)
- leijurv 5y ago
  There's a section called "The entire thing summed up in three sentences" in the linked GitHub README.
- bobsmooth 5y ago
  "You could punch any block coordinate in the entire map, and 2b2t would either: -Tell you what block is placed there, if the chunk is currently loaded -Or ignore you, if the chunk is not currently loaded. 2b2t would immediately reveal whether or not any chunk coordinate that you ask about is currently loaded. If it is currently loaded, it tells you which specific block is there at the coordinate you asked about." This allowed the hackers to find every player's coordinates and where their bases were.

cityofdelusion 5y ago

If anyone else was lost like myself, here is the rundown. The linked github readme assumes prior knowledge of 2b2t. This is an exploit for a very old Minecraft server (called 2b2t) where griefing (destroying other creations) and hacking is basically allowed. This kind of environment means that groups of players are incentivized to hide their projects out in the middle of nowhere, hundreds of thousands of blocks away from the world origin coordinates. The exploit allows a client to ask the server if data chunks at arbitrary coordinates have player activity in them, opening the server up to a brute force search to locate well-hidden bases. Basically, a group got spy satellites in an anarchy Minecraft server where remaining hidden is critical to not being destroyed.

kingcharles 5y ago
Thank you. I had no idea what was going on. This is actually pretty neat once you explain it.
Bluecobra 5y ago
This is a pretty clever exploit. I played on this server off and on between 2013-2016. The server is/was an acquired taste. The biggest challenge was to escape the the hellscape around spawn, once you get past that it was pretty quiet, I don’t ever recall running into anyone else unless I went back to the nether hub near spawn. It was really fun exploring old builds, the griefed ones were like old ruins. I liked to read the signs that were left behind and leave my own. I used to have use sone map mod and would note all the neat things I came across. I admit that I also used an x-ray mod to detect things like chests, because why not? I wonder if any of my bases were ever discovered. Most of my ill gotten gains were hidden away in a bottom of a mineshaft in the middle of nowhere to help avoid detection. Once the queue started started to get long I stopped playing.
rvnx 5y ago
In Minecraft you can dig blocks. When you click to dig a block, it just calls the RPC method CPacketPlayerDigging with the coordinates of the block. The server replies "Block of type ABC is too far". So, of course, you now know that at (X,Y,Z) you have a block of type "ABC". This RPC has no rate-limiting, so yes, you can know the whole map. The fact that you can dig any block on the map, seems like a shortcut taken by developers of the server to save time or resources. I'd even say it's almost intended behaviour. Tech details: https://2b2t.miraheze.org/wiki/Nocom https://2b2t.miraheze.org/wiki/Nocom

TheJoeMan 5y ago

I learned some of the Monte Carlo techniques in my medical imaging classes, it’s very interesting the data one can fish out with a good-enough graph. I also find the adaptive location “pinging” quite interesting, reminds me of the tech behind the F-22 active radar.

bobsmooth 5y ago

My favourite part of this has to be the social engineering used to get the exploit patched in.

kregasaurusrex 5y ago

The code used in the exploit itself itself looks rather harmless, as it was created to be a mitigation against finding players based on previously loaded chunks. https://github.com/PaperMC/Paper/blob/ver/1.12.2/Spigot-Server-Patches/0196-Fix-block-break-desync.patch https://github.com/PaperMC/Paper/blob/ver/1.12.2/Spigot-Serv...

leijurv 5y ago
That is not correct, this patch is meant to prevent block break desync, which is when your client and the server disagree about what blocks are placed in the world. If the server discards/ignores anything your client sends, the server MUST reply telling the client to undo that action, otherwise desync occurs. (e.g. "i have broken this block" -> the client is too far to reach that block -> the server has to reply with "set that block back to air"). The original version of this patch https://github.com/PaperMC/Paper/blob/d934dcc1e3e8b1ac34acef1b72c690229ebb018e/Spigot-Server-Patches/0196-Fix-block-break-desync.patch https://github.com/PaperMC/Paper/blob/d934dcc1e3e8b1ac34acef... fixed that desync, but it allowed you to force the server to generate an unlimited amount of chunks, which is an expensive operation. You could only find players using this patch after it was modified to only reply if the chunk was already loaded. But at no point in time was this patch able to reveal if a chunk had been previously generated or not, that is a separate unrelated issue in which Spigot will reliably send a partial chunk on initial generation, but never for a previously generated chunk. Take a look at https://2b2t.miraheze.org/wiki/Nocom https://2b2t.miraheze.org/wiki/Nocom
- kregasaurusrex 5y ago
  Thanks for the clarification!

w-ll 5y ago

Why would the `mineblock` packet not be clamped to at least the render distance or more realistic the chunk the sender is in and the 8 adjacent chunks. I know i'm learning about this in hindsight and it's obvious.

leijurv 5y ago
Yes, so Minecraft is exactly that. Any "I have mined this block" further than 6 blocks away from the player is ignored. The problem is that on a laggy server, an honest client can end up sending this, if the server lags out for more than 6 blocks of walking your player forward. So, Paper added a patch to undo these disallowed mines, setting them back to what they were. But this happened at any radius, and could be used to lag the server by making it generate any region. Then they patched it again and made it only reply if it were in a loaded chunk. The mistake was that it replies if the coordinate is in a chunk loaded by any player, not just by your player. :) See https://news.ycombinator.com/item?id=29620852 https://news.ycombinator.com/item?id=29620852
- w-ll 5y ago
  Hey thanks for your reply. I get how the Paper patch allowed this new exploit, but Im still curious to why Paper never even had the range check. Even for lag compensation it shouldnt be more than a chunk or 2 range. Im actually reminded to the latest log4j exploit (main reason i aint actually gonna run Minecraft for a while) and even the Heartbleed exploit a few years back. Did yall assume the patch to Paper would just return if the chunk was player loaded, and they wouldn't add any other sane bounds checks? If the story from the FitMC guy is true, yall guessed/hoped that patch would be what it was. Giving yall a new leak of information without actually fixing the problem because if it was loaded you could still get all the block info in that chunk. Pretty awesome from where i'm sitting. \(ᵔᵕᵔ)/
  - leijurv 5y ago
    Well, Paper is trying to fix the result of vanilla Minecraft's range check. Taking their perspective, they wanted to fix the issue where <6 blocks worked, but >6 blocks caused desync. So it is natural to reply in the "else" block with "no you can't break that". They just didn't think of adding yet another range check with a looser limit on top of that. I wasn't around for when the patch was engineered into Paper, but from what I'm told, yes that was the idea. :)

howenterprisey 5y ago

https://2b2t.miraheze.org/w/index.php?title=Space_Valkyria_3_V2&diff=next&oldid=58707 https://2b2t.miraheze.org/w/index.php?title=Space_Valkyria_3... Wow... I love this edit so much.

leijurv 5y ago
Thanks!