5 ms·

Running a million-board chess MMO in a single process

186 points by isaiahwp 1y ago

dangoodmanUT 1y ago
We did a very similar thing with UltimateArcade (shut down long ago) where we optimized everything to be single threaded. We had a pretty intense updates/s rate from players, and that got pretty crazy pretty fast. Actor models with single threads made things easy because we could skip locks and play with contigious chunks of memory, which as the author found, is pretty fast.
jasonjmcghee 1y ago
Big thread from a few months ago with the author. One Million Chessboards 336 points | 76 comments https://news.ycombinator.com/item?id=43825336 https://news.ycombinator.com/item?id=43825336
mannyv 1y ago
Could you save even more bandwidth by updating static data and letting Cloudflare cache/serve it? ie: put the batches on disk then have the clients grab it? It would be the equivalent of frame differencing, with the total board state being saved occasionally as a keyframe equivalent. You're doing that dynamically anyway by sending batches and snapshots to the client. Using the above you're basically making your game board into an interactive movie that's replaying moves from disk most of the time.
- ethan_smith 1y ago
  WebSockets with binary frames would likely be more efficient than HTTP polling for this use case, giving you real-time updates with less overhead than repeatedly fetching from disk via CDN.
- eieio 1y ago
  I thought about not pushing snapshot/move data over websockets - one of the systems-y friends I ran my architecture by brought this up while I was speccing the site out. You can't really put move batches on disk and have clients grab them (afaik), since the set of moves you want to send to an individual client depends on their position (and you don't want to send every move to every client). But you could do this by not sending move batches at all, and instead having clients poll for the entire current state of the board. The thing is, for them to get realtime-ish move updates they'd have to poll constantly. Cloudflare also has a min TTL of 1 second so there'd be more latency, and also if I screwed something up or saw more cache misses than anticipated I could end up unintentionally hammering my server. Also if I'd had 100x more traffic (which would be crazy and well beyond what I prepared for!) I think I'd owe like $95 or so for bandwidth with my current setup. So the benefits to reducing bandwidth even more were a little marginal!
  - derivagral 1y ago
    Late thought, but you could set up "beacon" locations for each client to pull a local grid derived from coordinates? Wouldn't solve that 1s latency thing, but might be interesting in other applications.
topato 1y ago
I feel like I keep seeing one million checkboxes guy's experiments, and I wonder.... Where is the monetization? Is this just a FANGAM engineer with a lot of free time? Am I just beaten down by the SWE landscape of 2025?
- RobotCaleb 1y ago
  Have you never made anything without trying to make a dollar on it?
  - mock-possum 1y ago
    No, I think the operative question is - Have you never felt so exhausted from working your day job that you don’t feel like building anything during your free time?
    - 01HNNWZ0MV43FF 1y ago
      Yeah but once I've been on break for a couple weeks my marshmallow unsquishes and I start making free stuff again
    - maccard 1y ago
      I don’t think it is. I think it’s normal to feel that way sometimes in our line of work, but it’s not normal to always feel that way.
      - wiseowise 1y ago
        > it’s not normal to always feel that way. Welcome to the world of living for the paycheck.
      - hombre_fatal 1y ago
        I think it's normal to always feel that way if you want to build things on the side but also don't aspire to be on the computer all day.
    - 1y ago
heroku 1y ago
How about you can move to other boards, and you can only capture from the board you have moved into.
dvoros 1y ago
Really-really nice article! I'm currently working on a single-process, in-memory, multiplayer game written in Golang. (: Happy and relieved to hear that it worked out for you and I might even borrow some ideas (e.g. I'm less worried about bandwidth, but Protobuf might still make sense). Thank you very much for sharing!
butlike 1y ago
I don't understand how Rooklyn and Queens are indestructible structures? A rook can attack the structure and take pieces off the board if it's ok with sacking itself. Edit: nevermind, the secret is revealed at the beginning of the paragraph >I prevented pieces from capturing pieces on other boards. so normally it's not an indestructible structure, but with that rule it is. Got it.
sehansen 1y ago
I love the clearly Starcraft-inspired look of the statboxes for the pieces.
commandlinefan 1y ago
I love seeing efficient use of computing resources. As long as I've been coding, it's been sort of a mantra that "programmer time is more important than execution time" - leading to very inefficient implementations that waste not just processor time but user's time. I keep hoping that with the ongoing migration to "the cloud" that identifies the relationship between efficiency and real cost, there'll be a resurgence in proper optimization.
- freeone3000 1y ago
  We’ll shift it back with “distributed edge computing”. But even with AWS, computing resources are cheaper than developer resources. It is very hard to needlessly burn even $20/hr on AWS without it needing much more time to fix
  - pklausler 1y ago
    "computing resources are cheaper than developers up to some level of scaling" is more accurate.
- sph87 1y ago
  This is why I've started moving towards embedded. There is something really nice about being back on a chip with <1MB ram and a handful of Mhz. I know it's really just a leftward shift of a curve though. By the time I am getting ready to retire in 20 years these little 40mm devices will probably be running 500B parameter models.
CJefferson 1y ago
While you definately need rollback I wonder if it would have been easier, if a player got into a conflict position, to just dump their state and grab it a fresh copy from the server? As long as you'd sent all their moves off to the server, they should see all the moves they've done which were valid (hmm.. there could be a race condition there). I wonder how often it's worth doing the more complex unwinding described? I may well be missing some complex case (which I imagine is often the case with rollback!)
- eieio 1y ago
  Yeah the big race here is that you've made a move (which might not be valid) and you're waiting on a response for that move - and you can receive other moves while you're waiting. I don't think requesting a new snapshot really helps there. If you do that you're dramatically extending the amount of time that the user sees an invalid state, since you're adding a whole new server roundtrip to the reconciliation process.
omoikane 1y ago
Were there any bots that participated in One Million Chessboards? I remember people building some automation around One Million Checkboxes to do some crazy animation stuff, I wonder if that was attempted with One Million Chessboards. Also, if One Million Chessboards started with some bots making moves, that could solve the cold-start problem mentioned under "a lack of awe", although I can also see how existence of bots might be a deterrent for some people.
richwater 1y ago
I love this dude

186 points by isaiahwp 1y ago

We did a very similar thing with UltimateArcade (shut down long ago) where we optimized everything to be single threaded. We had a pretty intense updates/s rate from players, and that got pretty crazy pretty fast. Actor models with single threads made things easy because we could skip locks and play with contigious chunks of memory, which as the author found, is pretty fast.

jasonjmcghee 1y ago

Big thread from a few months ago with the author. One Million Chessboards 336 points | 76 comments https://news.ycombinator.com/item?id=43825336 https://news.ycombinator.com/item?id=43825336

mannyv 1y ago

Could you save even more bandwidth by updating static data and letting Cloudflare cache/serve it? ie: put the batches on disk then have the clients grab it? It would be the equivalent of frame differencing, with the total board state being saved occasionally as a keyframe equivalent. You're doing that dynamically anyway by sending batches and snapshots to the client. Using the above you're basically making your game board into an interactive movie that's replaying moves from disk most of the time.

ethan_smith 1y ago
WebSockets with binary frames would likely be more efficient than HTTP polling for this use case, giving you real-time updates with less overhead than repeatedly fetching from disk via CDN.
eieio 1y ago
I thought about not pushing snapshot/move data over websockets - one of the systems-y friends I ran my architecture by brought this up while I was speccing the site out. You can't really put move batches on disk and have clients grab them (afaik), since the set of moves you want to send to an individual client depends on their position (and you don't want to send every move to every client). But you could do this by not sending move batches at all, and instead having clients poll for the entire current state of the board. The thing is, for them to get realtime-ish move updates they'd have to poll constantly. Cloudflare also has a min TTL of 1 second so there'd be more latency, and also if I screwed something up or saw more cache misses than anticipated I could end up unintentionally hammering my server. Also if I'd had 100x more traffic (which would be crazy and well beyond what I prepared for!) I think I'd owe like $95 or so for bandwidth with my current setup. So the benefits to reducing bandwidth even more were a little marginal!
- derivagral 1y ago
  Late thought, but you could set up "beacon" locations for each client to pull a local grid derived from coordinates? Wouldn't solve that 1s latency thing, but might be interesting in other applications.

topato 1y ago

I feel like I keep seeing one million checkboxes guy's experiments, and I wonder.... Where is the monetization? Is this just a FANGAM engineer with a lot of free time? Am I just beaten down by the SWE landscape of 2025?

RobotCaleb 1y ago
Have you never made anything without trying to make a dollar on it?
- mock-possum 1y ago
  No, I think the operative question is - Have you never felt so exhausted from working your day job that you don’t feel like building anything during your free time?
  - 01HNNWZ0MV43FF 1y ago
    Yeah but once I've been on break for a couple weeks my marshmallow unsquishes and I start making free stuff again
  - maccard 1y ago
    I don’t think it is. I think it’s normal to feel that way sometimes in our line of work, but it’s not normal to always feel that way.
    - wiseowise 1y ago
      > it’s not normal to always feel that way. Welcome to the world of living for the paycheck.
    - hombre_fatal 1y ago
      I think it's normal to always feel that way if you want to build things on the side but also don't aspire to be on the computer all day.
  - 1y ago

heroku 1y ago

How about you can move to other boards, and you can only capture from the board you have moved into.

dvoros 1y ago

Really-really nice article! I'm currently working on a single-process, in-memory, multiplayer game written in Golang. (: Happy and relieved to hear that it worked out for you and I might even borrow some ideas (e.g. I'm less worried about bandwidth, but Protobuf might still make sense). Thank you very much for sharing!

butlike 1y ago

I don't understand how Rooklyn and Queens are indestructible structures? A rook can attack the structure and take pieces off the board if it's ok with sacking itself. Edit: nevermind, the secret is revealed at the beginning of the paragraph >I prevented pieces from capturing pieces on other boards. so normally it's not an indestructible structure, but with that rule it is. Got it.

sehansen 1y ago

I love the clearly Starcraft-inspired look of the statboxes for the pieces.

commandlinefan 1y ago

I love seeing efficient use of computing resources. As long as I've been coding, it's been sort of a mantra that "programmer time is more important than execution time" - leading to very inefficient implementations that waste not just processor time but user's time. I keep hoping that with the ongoing migration to "the cloud" that identifies the relationship between efficiency and real cost, there'll be a resurgence in proper optimization.

freeone3000 1y ago
We’ll shift it back with “distributed edge computing”. But even with AWS, computing resources are cheaper than developer resources. It is very hard to needlessly burn even $20/hr on AWS without it needing much more time to fix
- pklausler 1y ago
  "computing resources are cheaper than developers up to some level of scaling" is more accurate.
sph87 1y ago
This is why I've started moving towards embedded. There is something really nice about being back on a chip with <1MB ram and a handful of Mhz. I know it's really just a leftward shift of a curve though. By the time I am getting ready to retire in 20 years these little 40mm devices will probably be running 500B parameter models.

CJefferson 1y ago

While you definately need rollback I wonder if it would have been easier, if a player got into a conflict position, to just dump their state and grab it a fresh copy from the server? As long as you'd sent all their moves off to the server, they should see all the moves they've done which were valid (hmm.. there could be a race condition there). I wonder how often it's worth doing the more complex unwinding described? I may well be missing some complex case (which I imagine is often the case with rollback!)

eieio 1y ago
Yeah the big race here is that you've made a move (which might not be valid) and you're waiting on a response for that move - and you can receive other moves while you're waiting. I don't think requesting a new snapshot really helps there. If you do that you're dramatically extending the amount of time that the user sees an invalid state, since you're adding a whole new server roundtrip to the reconciliation process.

omoikane 1y ago

Were there any bots that participated in One Million Chessboards? I remember people building some automation around One Million Checkboxes to do some crazy animation stuff, I wonder if that was attempted with One Million Chessboards. Also, if One Million Chessboards started with some bots making moves, that could solve the cold-start problem mentioned under "a lack of awe", although I can also see how existence of bots might be a deterrent for some people.

richwater 1y ago

I love this dude