4 ms·

Author here! Sorry that some of the stuff went over your head! I wanted to include longer descriptions of the tech I was using but the post was already suuuper

by eieio 2y ago

Author here! Sorry that some of the stuff went over your head! I wanted to include longer descriptions of the tech I was using but the post was already suuuper long and I felt like I couldn't add more. Very happy to answer any questions that you've got here! I'm not sure how you'd simplify the architecture in a major way to be honest. There are certainly services you could use for stuff like this, but I think there you're offloading the complexity to someone else? Ultimately you need: * A database that tracks which boxes are checked (that's Redis) * A choice about how to put your data in your database (I chose to just store 1 million bits for simplicity) * A way to tell your clients what the current state is (I chose to send them all 1 million bits - it's nice that this is not that much data) * A way for clients to tell you when they check a box + update your state (that's Flask + the websocket) * A way to tell your clients when a box is checked/unchecked (that's also Flask + websockets. I chose to send both updates about individual boxes and also updates about all 1 million boxes) * A way to avoid rendering 1 million dom elements all the time (react-window) The other stuff (nginx for static content + a reverse proxy) is mostly just to make things easier to scale; you could implement this solution without those details and the site would work fine, it just wouldn't be able to handle the same load.

sa46 2y ago
Just spitballing: could you change the database to a bool array? Guard it with a RWMutex and persist on server shutdown. The bottleneck probably moves to pushing updates from a single server, but Go can probably handle a few tens of thousands of goroutines.
- summerlight 2y ago
  > persist on server shutdown Probably this is not the simplest thing to do if you want a certain degree of reliability. Should be definitely easier than writing the entire storage engine, but likely an overkill for this kind of overnight hobby projects.
- stefs 2y ago
  one RW mutex would mean you'd lock the whole array; that way data access becomes pretty much single-threaded. simplest solution that comes to mind: AtomicIntegerArray (or whatever it is in your language of choice). you could also implement a bitset over AtomicLongArray. more complicated: partition into x*x chunks and rw-locking those. this could be backed by an mmap'ed a million bytes for persistence, but no idea if that'd make the app disk io bound or something.

by eieio 2y ago

sa46 2y ago

Just spitballing: could you change the database to a bool array? Guard it with a RWMutex and persist on server shutdown. The bottleneck probably moves to pushing updates from a single server, but Go can probably handle a few tens of thousands of goroutines.

summerlight 2y ago
> persist on server shutdown Probably this is not the simplest thing to do if you want a certain degree of reliability. Should be definitely easier than writing the entire storage engine, but likely an overkill for this kind of overnight hobby projects.
stefs 2y ago
one RW mutex would mean you'd lock the whole array; that way data access becomes pretty much single-threaded. simplest solution that comes to mind: AtomicIntegerArray (or whatever it is in your language of choice). you could also implement a bitset over AtomicLongArray. more complicated: partition into x*x chunks and rw-locking those. this could be backed by an mmap'ed a million bytes for persistence, but no idea if that'd make the app disk io bound or something.