7 ms·

Surprised to see this posted again, I wrote this, feel free to ask me anything!

snowycat 5y ago
Really impressive project! How did you learn all the things that you needed to make this? Are there any particular books or resources you would recommend? Also, how much did this entire project cost? I'd imagine processing that much data and storing it wasn't cheap.
- leijurv 5y ago
  Mostly learned as I went! I had never before implemeneted Monte Carlo localization, or DBSCAN, or a disjoint union find; it was a lot of research and trial and error. I'd honestly recommend Wikipedia! I also read a lot of blog posts such as on data clustering in order to decide how to write an algorithm to tell what is and isn't a single contiguous base. The main expense was priority queue, which is a $20/account/month payment to 2b2t to be able to join the server faster. Probably nearing a thousand dollars on that. We lightened this by creating a proxy system where other builders could join through our headless, and it would add nocom exploit packets from their account to 2b2t, and peel off the responses on the way back. They got a more reliable experience with less disconnects, and we got some extra capacity for free (completely unbeknownst to them of course haha). By the end, all the overworld accounts were being paid for and used by other people, I only had to cover the 2 accounts in the nether. About $300 on a new 2TB nvme drive for postgres. An unknown amount on electricity. Other than that, all the big processing ran on an existing homelab server (thanks to fr1kin). About $150 on digitalocean for a droplet to run the headless minecraft with minimal ping to 2b2t ($10/month). Backups were just WAL rsyncs to postgres replicas ran (again on homelabs) by myself and a few others.
ucarion 5y ago
I'm curious what kind of tooling you developed as part of this process. It seems, from the linked YouTube video, like individuals were actively monitoring the data produced from the exfiltration attack. How did you go from the raw data to producing something that people could usefully monitor?
- leijurv 5y ago
  Sure! For monitoring on a meta-level, we used Prometheus and Grafana. Here's what that looked like near the very end: https://www.dropbox.com/s/ktlvvtmppq64u8o/trimmed%20grafana%202.mov?dl=0 https://www.dropbox.com/s/ktlvvtmppq64u8o/trimmed%20grafana%... Since the ratelimit was heavily in place by then, the numbers are pretty low. For example, the headliner "blocks per second" was well over three thousand a few weeks before then. One of the most common queries for actual usage though, was "what bases in the last 24 hours (or whatever) have the most chests currently". We'd then travel to these locations to borrow items. Here's that query https://pastebin.com/fjhHGSYz https://pastebin.com/fjhHGSYz Roughly, it grabs chunks with chests in that timeframe, then uses a recursive Postgres CTE to traverse the https://en.wikipedia.org/wiki/Disjoint-set_data_structure https://en.wikipedia.org/wiki/Disjoint-set_data_structure that we used to implement DBSCAN, from the leaf to the root. Then it groups by the root, which in practice means grouping chunks together by which ones semantically have been determined to be part of the same base, then makes a report by quantity per base. There was also a web UI that showed all active tracks. You could click any player and see their travel history, the route they had taken to get to that location. As well as clicking any cluster and seeing the usernames associated with it. We used this a lot to grab what members were present at a given item stash or base, to make sure we weren't stepping on anyone's toes. In terms of what it took to make this happen, the stars of the show were probably the main Postgres, which was hand-tuned on top of a hand-tuned ZFS, on top of a NVME drive that was IOMMU passthrough'd to this VM for maximum performance. As well as https://github.com/OpenHFT/Chronicle-Map https://github.com/OpenHFT/Chronicle-Map, which really helped out. We needed a mapping from block coordinate to a bit of data about that location, such as the last timestamp we checked it, what Minecraft block was there last, whether the block is interesting, uninteresting, different from seed, etc. High frequency trading software was perfect for this use case, because it greatly reduced GC pressure on the JVM, since the data was stored off-heap. It ended up at around (roughly) fifty million entries, served hundreds of thousands of requests for block data per second, and requested historical data (from the Postgres) at about thirty thousands SELECTs per second (from a table with over ten billion rows). The reason why this needed to be so fast, is that the use case is downloading bases. Sometimes someone would log in to 2b2t at their base, after having not played for weeks at a time (so the data is in Postgres, not the map in RAM), we wanted to be able to pick up right where we left off on downloading what they had built. This means it needs to get back up to speed with their entire base, as fast as possible, in the seconds before they move away to other chunks, or log off the server. We are downloading bases one click at a time, and bases are huge. The area that is loaded in by a player is 144 by 144 by 256. That’s 5 million blocks. And there are about 250 players online on 2b2t at any given moment in time. That’s well over a billion blocks that we could click on, and learn which Minecraft block they are. But, which locations should we use up our clicks per second budget on? It focused on areas that are being changed rapidly, and areas that are different from the base vanilla terrain. Those are certainly bases, large Minecraft builds. We do an expanding paintbucket pattern that expands in 3 dimensions to whatever has been built there. However, we have to do this quickly, and we have to pick up where we left off. Someone might be flying around their base with an elytra, at very fast speed. We would love to scoop up the terrain that they’re flying over. If the base is particularly interesting, someone flying over a certain area could have had about twenty Minecraft accounts spread all over 2b2t’s map suddenly refocus all of their left clicks on that one area, to grab all the interesting neighbors that it wasn’t able to last time. We peaked at about three thousand block clicks per second, averaged over weeks. So this was a massive data loading and management problem.
fffobar 5y ago
Tracking was completely broken by any teleporation, right? Such as enderpearl loading or a respawn at the bed. Also, how many login sessions would it take to associate a track with an account?
- leijurv 5y ago
  Correct, loading a pearl and respawning would throw us off the trail. This is why our aim was to follow the initial travel to the base location, rather than a future pearl TP. The easiest advice in order to throw off nocom is to travel in a straight line in the overworld for perhaps 10k blocks, set a bed quickly, then continue another 10k blocks, then /kill to the bed, then go perpendicularly for a render distance or two before the chunks unload at your death location. Even with some hypothetical tracking improvements (that never actually were written), such as walking back the last 10 minutes whenever a track goes cold, that would still throw it off. Generally 2 or 3. 1 point of association was given whenever a track went cold, split up among however many players logged off within -6 seconds to +1 seconds of that timestamp (because of chunks unloading on a 5 second schedule). 1 point could be a false positive, but 2 or 3 would be reliably trustworthy.
ThouYS 5y ago
what's your take on university as opposed to learning by doing? how do they complement each other?
- leijurv 5y ago
  I am not done with university yet, so I don't speak confidently. But I'm afraid I have no special perspective, I just echo what I hear a lot: Learning by doing is probably the most important, but structured courses provide a theoretical and practical foundation that fills in a lot of important gaps that I would have had if I were only self taught.
  - kbenson 5y ago
    That echoes my own thoughts as someone that's been out of school for a long time and worked with people with and without degrees. There's some gaps without a degree, but not usually so much that they can't be overcome with a little time and effort, and I don't think it usually matters much. Connections from university though... I suspect that can be extremely useful, but didn't leverage that in my own time there, so can only guess as to how much (and also I didn't go to an ivy league school or anything where it could have made an even larger difference).
ajkjk 5y ago
Why did you do all this? I mean.. it's a ton of engineering, for what? For fun?
- leijurv 5y ago
  Yes absolutely, for fun! Of course, it's also fun to have a secret project that no one knows about, to get an edge. 2b2t is practically the only Minecraft server that allows any kind of hacking your client (without getting banned), while also having an active community of meaningful size, so that was the clear target.
  - phgn 5y ago
    I'm curious (without having context on 2b2t): How did you all think about the morality of the exploit? Like it's fun to overcome technical challenges and interesting to see / follow projects other people build in game, but aren't you destroying their work? Or is that accepted practice on the server?
    - leijurv 5y ago
      Yes, this is a very fair question to ask. Of course, on the one hand, I could just say "this was fair game". The "point" of choosing to play on this server, is that it is THE no-rules server. The people who stick around more than a few months are the ones who accept that anything will eventually be griefed. Sometimes people call this the "sandcastle mentality", which means that a player accepts the destructive nature of 2b2t, and learns to live with it. Its name comes from building sandcastles on beaches: no one builds a sandcastle believing or expecting that it will last forever. The waves will always wash it away eventually, or someone else will break it. The only thing standing between your base and "the waves" is that no one knows where it is. One way to think about it, is that in "normal" Minecraft servers, it's you versus everyone, but in an anarchy server that allows hacks, it's you versus the server software. There have been many hacks to locate other players, like following the trails that they leave in the world, that you can only see if you inspect some specific bits in chunk packets from the server. Perhaps they posted a screenshot that shows bedrock, which you can brute force their location from in a few hours/days on a GPU with OpenCL that simulates Minecraft terrain generation. Nocom was similarly "fair game", in that it only used the Minecraft protocol, and stayed within the "bounds" of talking to 2b2t with Minecraft packets and extracting information from that. All that being said, it's still plainly the case that people put a ton of time into their bases, and no matter how much moral sleight of hand and cope you apply, it's still kicking over someone's sandcastle. When it comes to actually traveling to these locations and raiding them, our aim in that campaign was pretty much solely to amass a large stash of items/blocks for ourselves for the future. I would pass on bases by-default, only handing them along to be raided if there was bad faith construction (e.g. swastikas), or if most of the builders were known to be naughty. The main campaign was against standalone item stashes. These do take some time to create (by having exploiting past item duplication glitches), but they don't really have any creative expression in the same way that a base does. I have no logical argument for why I say that the same amount of hours invested in artistically making a build is more morally valuable than duplicating building blocks, it's just what feels about right. During the time when I was involved, we passed up hundreds of bases just because there was something actively being built there, and essentially only hit places that were just rows of chests placed on the ground in the world.
deleted 5y ago
[deleted]
barneygale 5y ago
Hello! I'm the original owner of the 'BibleBot' account/bot. Glad to see you put it to good use :-)
- leijurv 5y ago
  Lol! Sorry for getting it locked, it is reversible if you still have the email. Mojang, it appears, didn't like spam login from a different cloudflare warp IP every few minutes...
  - barneygale 5y ago
    All is forgiven if you can share the trick to not getting AFK kicked? (first.last@gmail.com). I've wanted to bring the OG code back for years.
    - BbzzbB 5y ago
      Talking of OG, that Gmail account... looks rare.
      - Fnoord 5y ago
        Username is barneygale. You do the parsing.
        BbzzbB 5y ago
        I'm not clear what you are trying to hint towards. With your comment I figured he would be a Google developer or a long-time dev (hence likely to be exposed to Gmail early on to snatch first.last), but from what I see he was a few years away from starting college or his Github when it launched. I'm afraid that I am missing your point.
        crazysim 5y ago
        There's was a method of email obfuscation here. It's not literally firstname.lastname@gmail.com . That's just a template.
phgn 5y ago
This whole project is incredibly impressive, good job! How much time would you say went into this both from yourself and the others? Probably quite a lot over the years? (because again, it's very impressive)
- leijurv 5y ago
  Hard to estimate, probably a few hundred hours from me, and a comparable amount from others. As a random data point, I sent 53000 messages in the group chat developing it. ¯\_(ツ)_/¯
  - stickyricky 5y ago
    You're very lucky to have that. I've always wanted a circle of programmers to build stuff with. Its an incredible force multiplier and I'm sure you'll have fond memories of that camaraderie for life.
    - leijurv 5y ago
      :)
tommsy64 5y ago
I'm curious how you developed the "very cute headless" Minecraft client. Did you use the Forge build tools? How did you go about ripping out the rendering/keyboard stuff?
- leijurv 5y ago
  Yes, Forgegradle was used. Frankly, not much interesting here, just a whole lot of work going through all the Minecraft code, and using Java reflection where possible and JVM bytecode injection otherwise. There's a lot of access transforming, using theUnsafe to allocate classes without calling their constructors, etc. As a random example that I remember, one of the last things to patch was that we had it working, but when you got kicked from the server, it would crash. This is because it would try to create a currentScreen GUI for the "disconnected from server" message, which for some reason was long enough that it was considering whether to wrap the text to the next line, which needed something like a font size, or maybe the window width, which crashed since there is no window or font.
potatoman22 5y ago
Did you use some form of Bayesian statistics to choose which chunk to query?
- leijurv 5y ago
  Nothing so advanced, I'm afraid. The Monte Carlo system just creates a thousand points (2d position and velocity), and simulates them at 1 generation per second, culling points that are unlikely and reproducing points that are likely based on observations (clicking chunks and hearing back "loaded" or "unloaded").
  - Vetch 5y ago
    That sounds a lot like sequential monte carlo or a particle filter, which is a type of bayesian filter for inference. Really impressive what you did.
    - whymauri 5y ago
      The write up actually does reference a paper on particle filters :o
      - leijurv 5y ago
        Yeah, https://users.isy.liu.se/rt/schon/Publications/HolSG2006.pdf https://users.isy.liu.se/rt/schon/Publications/HolSG2006.pdf was great to refer to, just the right level of complexity for me.
gwern 5y ago
Why did it take them so long to discover the issue when you and another group were practically DoSing them every second for years on end (and they kept lowering the packet limit)? Didn't they ever wonder what it was all for or what the outgoing packets contained?
- leijurv 5y ago
  This is one of life's great mysteries. Given the hosting provider for 2b2t, we know pretty much for a fact that the packet spam from 2018 and 2019 (before I knew about the project ;)) would have cost tens of thousands of dollars in bandwidth overages per month. Yet, he never added a packet limiter until an unrelated exploit could cause other clients to crash due to spamming an offhand swap sound effect. It took the exploit going fully public, plus another 36 hours or so, for it to actually get patched. ¯\_(ツ)_/¯
  - goldenkey 5y ago
    How does monetization work? Was the server host making hundreds of thousands of dollars? That makes the extra cost negligible.
    - rebane2001 5y ago
      The server's main income is priority queue. By default, if the server is full (it almost always is) you are placed in the "normal queue" and have to wait for other people to leave. This normal queue is often hundreds of people long, leading to wait times several hours long. For $20 a month you can purchase the priority queue, which skips the wait. A lot of people pay for this as it is the only way to comfortably play on the server, so it is quite the income.
  - gwern 5y ago
    I see. Maybe they were just lazy, and since they didn't see anything obviously going wrong other than wasted bandwidth, let it go. Sort of like the NASA management in the Challenger report; as Feynman pointed out, the fact that the O-rings were eroding when they were not supposed to erode at all indicated that the models were simply wrong, and thus there was no guarantee of safety at all, as the "safety margins" were only margins if the models were right; but since no shuttle had visibly blown up yet... Cryptographic and anonymity and information leaks are particularly dangerous that way, because when they fail, they fail silently: a bad crypto scheme still encrypts and decrypts your emails, it just lets the enemy decrypt them too. I'm reminded of my work on darknet markets: DNMs often leaked their IP or other information because there are so many ways for Apache or Linux to do so by default, and when they do so, there's no visible error; the DNM will continue working exactly as intended, right up until the police kick in your door. Aside from being proactive about investigating anomalies, information leaks are very hard to defend against in general. Hard to see how you could reasonably block this bug without some fancy capability security implementation which would let you do something like require the client to produce a proof that it is in sight of a block and permitted to query its current status. (Then requesting arbitrary blocks would fail by default, and a patch which mistakenly gave everyone a global capability to let requests succeed would be impossible to write without knowing one was introducing a grave vulnerability.)
heartbeats 5y ago
How accurate a map did you end up with?
- leijurv 5y ago
  For tracking, the spatial resolution was 1 chunk, which is 16 by 16 blocks, and the temporal resolution was 1 second. So, we have a few billion rows of tracking data for every player, localizing them to roughly that distance, roughly once a second. For downloading bases, it is an essentially perfect recreation. Some things are missing, such as the color of banners, but for 99% of blocks, just setting the correct blockstates will recreate the build. Some disconnected sections of the build might not be found by the paintbucket floodfill algorithm though, so floating parts could be missing. We counteracted this by having a few random blocks in the chunk be checked on a schedule, so eventually it would get everything.

Really impressive project! How did you learn all the things that you needed to make this? Are there any particular books or resources you would recommend? Also, how much did this entire project cost? I'd imagine processing that much data and storing it wasn't cheap.

leijurv 5y ago
Mostly learned as I went! I had never before implemeneted Monte Carlo localization, or DBSCAN, or a disjoint union find; it was a lot of research and trial and error. I'd honestly recommend Wikipedia! I also read a lot of blog posts such as on data clustering in order to decide how to write an algorithm to tell what is and isn't a single contiguous base. The main expense was priority queue, which is a $20/account/month payment to 2b2t to be able to join the server faster. Probably nearing a thousand dollars on that. We lightened this by creating a proxy system where other builders could join through our headless, and it would add nocom exploit packets from their account to 2b2t, and peel off the responses on the way back. They got a more reliable experience with less disconnects, and we got some extra capacity for free (completely unbeknownst to them of course haha). By the end, all the overworld accounts were being paid for and used by other people, I only had to cover the 2 accounts in the nether. About $300 on a new 2TB nvme drive for postgres. An unknown amount on electricity. Other than that, all the big processing ran on an existing homelab server (thanks to fr1kin). About $150 on digitalocean for a droplet to run the headless minecraft with minimal ping to 2b2t ($10/month). Backups were just WAL rsyncs to postgres replicas ran (again on homelabs) by myself and a few others.

ucarion 5y ago

I'm curious what kind of tooling you developed as part of this process. It seems, from the linked YouTube video, like individuals were actively monitoring the data produced from the exfiltration attack. How did you go from the raw data to producing something that people could usefully monitor?

leijurv 5y ago
Sure! For monitoring on a meta-level, we used Prometheus and Grafana. Here's what that looked like near the very end: https://www.dropbox.com/s/ktlvvtmppq64u8o/trimmed%20grafana%202.mov?dl=0 https://www.dropbox.com/s/ktlvvtmppq64u8o/trimmed%20grafana%... Since the ratelimit was heavily in place by then, the numbers are pretty low. For example, the headliner "blocks per second" was well over three thousand a few weeks before then. One of the most common queries for actual usage though, was "what bases in the last 24 hours (or whatever) have the most chests currently". We'd then travel to these locations to borrow items. Here's that query https://pastebin.com/fjhHGSYz https://pastebin.com/fjhHGSYz Roughly, it grabs chunks with chests in that timeframe, then uses a recursive Postgres CTE to traverse the https://en.wikipedia.org/wiki/Disjoint-set_data_structure https://en.wikipedia.org/wiki/Disjoint-set_data_structure that we used to implement DBSCAN, from the leaf to the root. Then it groups by the root, which in practice means grouping chunks together by which ones semantically have been determined to be part of the same base, then makes a report by quantity per base. There was also a web UI that showed all active tracks. You could click any player and see their travel history, the route they had taken to get to that location. As well as clicking any cluster and seeing the usernames associated with it. We used this a lot to grab what members were present at a given item stash or base, to make sure we weren't stepping on anyone's toes. In terms of what it took to make this happen, the stars of the show were probably the main Postgres, which was hand-tuned on top of a hand-tuned ZFS, on top of a NVME drive that was IOMMU passthrough'd to this VM for maximum performance. As well as https://github.com/OpenHFT/Chronicle-Map https://github.com/OpenHFT/Chronicle-Map, which really helped out. We needed a mapping from block coordinate to a bit of data about that location, such as the last timestamp we checked it, what Minecraft block was there last, whether the block is interesting, uninteresting, different from seed, etc. High frequency trading software was perfect for this use case, because it greatly reduced GC pressure on the JVM, since the data was stored off-heap. It ended up at around (roughly) fifty million entries, served hundreds of thousands of requests for block data per second, and requested historical data (from the Postgres) at about thirty thousands SELECTs per second (from a table with over ten billion rows). The reason why this needed to be so fast, is that the use case is downloading bases. Sometimes someone would log in to 2b2t at their base, after having not played for weeks at a time (so the data is in Postgres, not the map in RAM), we wanted to be able to pick up right where we left off on downloading what they had built. This means it needs to get back up to speed with their entire base, as fast as possible, in the seconds before they move away to other chunks, or log off the server. We are downloading bases one click at a time, and bases are huge. The area that is loaded in by a player is 144 by 144 by 256. That’s 5 million blocks. And there are about 250 players online on 2b2t at any given moment in time. That’s well over a billion blocks that we could click on, and learn which Minecraft block they are. But, which locations should we use up our clicks per second budget on? It focused on areas that are being changed rapidly, and areas that are different from the base vanilla terrain. Those are certainly bases, large Minecraft builds. We do an expanding paintbucket pattern that expands in 3 dimensions to whatever has been built there. However, we have to do this quickly, and we have to pick up where we left off. Someone might be flying around their base with an elytra, at very fast speed. We would love to scoop up the terrain that they’re flying over. If the base is particularly interesting, someone flying over a certain area could have had about twenty Minecraft accounts spread all over 2b2t’s map suddenly refocus all of their left clicks on that one area, to grab all the interesting neighbors that it wasn’t able to last time. We peaked at about three thousand block clicks per second, averaged over weeks. So this was a massive data loading and management problem.

fffobar 5y ago

Tracking was completely broken by any teleporation, right? Such as enderpearl loading or a respawn at the bed. Also, how many login sessions would it take to associate a track with an account?

leijurv 5y ago
Correct, loading a pearl and respawning would throw us off the trail. This is why our aim was to follow the initial travel to the base location, rather than a future pearl TP. The easiest advice in order to throw off nocom is to travel in a straight line in the overworld for perhaps 10k blocks, set a bed quickly, then continue another 10k blocks, then /kill to the bed, then go perpendicularly for a render distance or two before the chunks unload at your death location. Even with some hypothetical tracking improvements (that never actually were written), such as walking back the last 10 minutes whenever a track goes cold, that would still throw it off. Generally 2 or 3. 1 point of association was given whenever a track went cold, split up among however many players logged off within -6 seconds to +1 seconds of that timestamp (because of chunks unloading on a 5 second schedule). 1 point could be a false positive, but 2 or 3 would be reliably trustworthy.

ThouYS 5y ago

what's your take on university as opposed to learning by doing? how do they complement each other?

leijurv 5y ago
I am not done with university yet, so I don't speak confidently. But I'm afraid I have no special perspective, I just echo what I hear a lot: Learning by doing is probably the most important, but structured courses provide a theoretical and practical foundation that fills in a lot of important gaps that I would have had if I were only self taught.
- kbenson 5y ago
  That echoes my own thoughts as someone that's been out of school for a long time and worked with people with and without degrees. There's some gaps without a degree, but not usually so much that they can't be overcome with a little time and effort, and I don't think it usually matters much. Connections from university though... I suspect that can be extremely useful, but didn't leverage that in my own time there, so can only guess as to how much (and also I didn't go to an ivy league school or anything where it could have made an even larger difference).

ajkjk 5y ago

Why did you do all this? I mean.. it's a ton of engineering, for what? For fun?

leijurv 5y ago
Yes absolutely, for fun! Of course, it's also fun to have a secret project that no one knows about, to get an edge. 2b2t is practically the only Minecraft server that allows any kind of hacking your client (without getting banned), while also having an active community of meaningful size, so that was the clear target.
- phgn 5y ago
  I'm curious (without having context on 2b2t): How did you all think about the morality of the exploit? Like it's fun to overcome technical challenges and interesting to see / follow projects other people build in game, but aren't you destroying their work? Or is that accepted practice on the server?
  - leijurv 5y ago
    Yes, this is a very fair question to ask. Of course, on the one hand, I could just say "this was fair game". The "point" of choosing to play on this server, is that it is THE no-rules server. The people who stick around more than a few months are the ones who accept that anything will eventually be griefed. Sometimes people call this the "sandcastle mentality", which means that a player accepts the destructive nature of 2b2t, and learns to live with it. Its name comes from building sandcastles on beaches: no one builds a sandcastle believing or expecting that it will last forever. The waves will always wash it away eventually, or someone else will break it. The only thing standing between your base and "the waves" is that no one knows where it is. One way to think about it, is that in "normal" Minecraft servers, it's you versus everyone, but in an anarchy server that allows hacks, it's you versus the server software. There have been many hacks to locate other players, like following the trails that they leave in the world, that you can only see if you inspect some specific bits in chunk packets from the server. Perhaps they posted a screenshot that shows bedrock, which you can brute force their location from in a few hours/days on a GPU with OpenCL that simulates Minecraft terrain generation. Nocom was similarly "fair game", in that it only used the Minecraft protocol, and stayed within the "bounds" of talking to 2b2t with Minecraft packets and extracting information from that. All that being said, it's still plainly the case that people put a ton of time into their bases, and no matter how much moral sleight of hand and cope you apply, it's still kicking over someone's sandcastle. When it comes to actually traveling to these locations and raiding them, our aim in that campaign was pretty much solely to amass a large stash of items/blocks for ourselves for the future. I would pass on bases by-default, only handing them along to be raided if there was bad faith construction (e.g. swastikas), or if most of the builders were known to be naughty. The main campaign was against standalone item stashes. These do take some time to create (by having exploiting past item duplication glitches), but they don't really have any creative expression in the same way that a base does. I have no logical argument for why I say that the same amount of hours invested in artistically making a build is more morally valuable than duplicating building blocks, it's just what feels about right. During the time when I was involved, we passed up hundreds of bases just because there was something actively being built there, and essentially only hit places that were just rows of chests placed on the ground in the world.

deleted 5y ago

[deleted]

barneygale 5y ago

Hello! I'm the original owner of the 'BibleBot' account/bot. Glad to see you put it to good use :-)

leijurv 5y ago
Lol! Sorry for getting it locked, it is reversible if you still have the email. Mojang, it appears, didn't like spam login from a different cloudflare warp IP every few minutes...
- barneygale 5y ago
  All is forgiven if you can share the trick to not getting AFK kicked? (first.last@gmail.com). I've wanted to bring the OG code back for years.
  - BbzzbB 5y ago
    Talking of OG, that Gmail account... looks rare.
    - Fnoord 5y ago
      Username is barneygale. You do the parsing.
      - BbzzbB 5y ago
        I'm not clear what you are trying to hint towards. With your comment I figured he would be a Google developer or a long-time dev (hence likely to be exposed to Gmail early on to snatch first.last), but from what I see he was a few years away from starting college or his Github when it launched. I'm afraid that I am missing your point.
        crazysim 5y ago
        There's was a method of email obfuscation here. It's not literally firstname.lastname@gmail.com . That's just a template.

phgn 5y ago

This whole project is incredibly impressive, good job! How much time would you say went into this both from yourself and the others? Probably quite a lot over the years? (because again, it's very impressive)

leijurv 5y ago
Hard to estimate, probably a few hundred hours from me, and a comparable amount from others. As a random data point, I sent 53000 messages in the group chat developing it. ¯\_(ツ)_/¯
- stickyricky 5y ago
  You're very lucky to have that. I've always wanted a circle of programmers to build stuff with. Its an incredible force multiplier and I'm sure you'll have fond memories of that camaraderie for life.
  - leijurv 5y ago
    :)

tommsy64 5y ago

I'm curious how you developed the "very cute headless" Minecraft client. Did you use the Forge build tools? How did you go about ripping out the rendering/keyboard stuff?

leijurv 5y ago
Yes, Forgegradle was used. Frankly, not much interesting here, just a whole lot of work going through all the Minecraft code, and using Java reflection where possible and JVM bytecode injection otherwise. There's a lot of access transforming, using theUnsafe to allocate classes without calling their constructors, etc. As a random example that I remember, one of the last things to patch was that we had it working, but when you got kicked from the server, it would crash. This is because it would try to create a currentScreen GUI for the "disconnected from server" message, which for some reason was long enough that it was considering whether to wrap the text to the next line, which needed something like a font size, or maybe the window width, which crashed since there is no window or font.

potatoman22 5y ago

Did you use some form of Bayesian statistics to choose which chunk to query?

leijurv 5y ago
Nothing so advanced, I'm afraid. The Monte Carlo system just creates a thousand points (2d position and velocity), and simulates them at 1 generation per second, culling points that are unlikely and reproducing points that are likely based on observations (clicking chunks and hearing back "loaded" or "unloaded").
- Vetch 5y ago
  That sounds a lot like sequential monte carlo or a particle filter, which is a type of bayesian filter for inference. Really impressive what you did.
  - whymauri 5y ago
    The write up actually does reference a paper on particle filters :o
    - leijurv 5y ago
      Yeah, https://users.isy.liu.se/rt/schon/Publications/HolSG2006.pdf https://users.isy.liu.se/rt/schon/Publications/HolSG2006.pdf was great to refer to, just the right level of complexity for me.

gwern 5y ago

Why did it take them so long to discover the issue when you and another group were practically DoSing them every second for years on end (and they kept lowering the packet limit)? Didn't they ever wonder what it was all for or what the outgoing packets contained?

leijurv 5y ago
This is one of life's great mysteries. Given the hosting provider for 2b2t, we know pretty much for a fact that the packet spam from 2018 and 2019 (before I knew about the project ;)) would have cost tens of thousands of dollars in bandwidth overages per month. Yet, he never added a packet limiter until an unrelated exploit could cause other clients to crash due to spamming an offhand swap sound effect. It took the exploit going fully public, plus another 36 hours or so, for it to actually get patched. ¯\_(ツ)_/¯
- goldenkey 5y ago
  How does monetization work? Was the server host making hundreds of thousands of dollars? That makes the extra cost negligible.
  - rebane2001 5y ago
    The server's main income is priority queue. By default, if the server is full (it almost always is) you are placed in the "normal queue" and have to wait for other people to leave. This normal queue is often hundreds of people long, leading to wait times several hours long. For $20 a month you can purchase the priority queue, which skips the wait. A lot of people pay for this as it is the only way to comfortably play on the server, so it is quite the income.
- gwern 5y ago
  I see. Maybe they were just lazy, and since they didn't see anything obviously going wrong other than wasted bandwidth, let it go. Sort of like the NASA management in the Challenger report; as Feynman pointed out, the fact that the O-rings were eroding when they were not supposed to erode at all indicated that the models were simply wrong, and thus there was no guarantee of safety at all, as the "safety margins" were only margins if the models were right; but since no shuttle had visibly blown up yet... Cryptographic and anonymity and information leaks are particularly dangerous that way, because when they fail, they fail silently: a bad crypto scheme still encrypts and decrypts your emails, it just lets the enemy decrypt them too. I'm reminded of my work on darknet markets: DNMs often leaked their IP or other information because there are so many ways for Apache or Linux to do so by default, and when they do so, there's no visible error; the DNM will continue working exactly as intended, right up until the police kick in your door. Aside from being proactive about investigating anomalies, information leaks are very hard to defend against in general. Hard to see how you could reasonably block this bug without some fancy capability security implementation which would let you do something like require the client to produce a proof that it is in sight of a block and permitted to query its current status. (Then requesting arbitrary blocks would fail by default, and a patch which mistakenly gave everyone a global capability to let requests succeed would be impossible to write without knowing one was introducing a grave vulnerability.)

heartbeats 5y ago

How accurate a map did you end up with?

leijurv 5y ago
For tracking, the spatial resolution was 1 chunk, which is 16 by 16 blocks, and the temporal resolution was 1 second. So, we have a few billion rows of tracking data for every player, localizing them to roughly that distance, roughly once a second. For downloading bases, it is an essentially perfect recreation. Some things are missing, such as the color of banners, but for 99% of blocks, just setting the correct blockstates will recreate the build. Some disconnected sections of the build might not be found by the paintbucket floodfill algorithm though, so floating parts could be missing. We counteracted this by having a few random blocks in the chunk be checked on a schedule, so eventually it would get everything.