4 ms·

I can't get the first level to play, seems things are still overloaded. Really looking forward to this though, hope you will post or email again once things se

I can't get the first level to play, seems things are still overloaded. Really looking forward to this though, hope you will post or email again once things settle to remind us about it. edit: this is the message im getting trying to play first steps, if it helps, dashboard says "Trades.Exec() (Status: playable!)" (3:08 UTC) > We couldn't start the level: This level has been locked by a different server. (Sorry if this happened accidentally as a result of a server crash -- it will clear up in an hour.) Refresh page to check if server is down?

xfour 11y ago
Managed to get through level 2 eventually, lot of retries while boxes were being rebooted etc. The calls to /gm must allocate an instance or something of that sort as once you manage to get lucky on that step the game will work it seems.
patio11 11y ago
I'm going to try to build a self-help tool for unlocking level instances. Briefly: you get round-robined to a (healthy, importantly) GM machine the first time you try to start a level and that association needs to be fairly sticky. Once you've started the level, a piece of that box's memory is all your own until the level ends. To prevent a single player from getting memory on all the boxes we "lock" a level to a particular box. While the level is running, no other box will open the same level for the same player. The problem is when the GM with your box becomes unhealthy. Nginx will put you in touch with a different GM, which could possibly mean your level is still running on a box you can't contact. At that point, you have to wait for either that box to hurry up and finish dying, for me to do a redeploy, or for one of the other GMs to figure "OK, that level appears locked but it's a very old lock so it's unlikely it is legit. I'll unlock it for you and grab the lock." Levels generally die ~40 minutes after starting (but who knows with a failing GM). The lock ages open after ~1 hour.
- xfour 11y ago
  Perhaps it could be as simple as keeping a hash of levels tied to a box. When your monitoring alerts that your box is unhealthy, run a process to remove all locks associated with the dead box allowing the player to refresh and restart the session? Also, maybe a new error, is getting mixed in, response from /gm/levels {"ok":false,"error":"exit status 1"}
  - Rapzid 11y ago
    I was thinking you could treat is as a distributed hash ring with key shuffling, healing and the like ala Riak. Or a virtual actor cluster ala .NET's Orleans. You could use container migration to move people off dying nodes, or to redistribute the cluster. Ultimately though, it's probably a pretty lean startup and there is only so much experience and effort you can apply to each area.
- fizx 11y ago
  Distributed locking with liveness checks is exactly what zookeeper excels at. I wrote a one-click deploy (via cloudformation) self-healing zookeeper cluster on AWS (wrapping netflix exhibitor for the pretty GUI and config reloads) that has been running since the beginning of 2014 on ~$100/mo in production without issue. It's roughly 200-300 lines of ops config, because exhibitor makes everything easy. Consider biting the bullet! If you want to discuss further, my email is in my HN profile.

xfour 11y ago

Managed to get through level 2 eventually, lot of retries while boxes were being rebooted etc. The calls to /gm must allocate an instance or something of that sort as once you manage to get lucky on that step the game will work it seems.

patio11 11y ago

I'm going to try to build a self-help tool for unlocking level instances. Briefly: you get round-robined to a (healthy, importantly) GM machine the first time you try to start a level and that association needs to be fairly sticky. Once you've started the level, a piece of that box's memory is all your own until the level ends. To prevent a single player from getting memory on all the boxes we "lock" a level to a particular box. While the level is running, no other box will open the same level for the same player. The problem is when the GM with your box becomes unhealthy. Nginx will put you in touch with a different GM, which could possibly mean your level is still running on a box you can't contact. At that point, you have to wait for either that box to hurry up and finish dying, for me to do a redeploy, or for one of the other GMs to figure "OK, that level appears locked but it's a very old lock so it's unlikely it is legit. I'll unlock it for you and grab the lock." Levels generally die ~40 minutes after starting (but who knows with a failing GM). The lock ages open after ~1 hour.

xfour 11y ago
Perhaps it could be as simple as keeping a hash of levels tied to a box. When your monitoring alerts that your box is unhealthy, run a process to remove all locks associated with the dead box allowing the player to refresh and restart the session? Also, maybe a new error, is getting mixed in, response from /gm/levels {"ok":false,"error":"exit status 1"}
- Rapzid 11y ago
  I was thinking you could treat is as a distributed hash ring with key shuffling, healing and the like ala Riak. Or a virtual actor cluster ala .NET's Orleans. You could use container migration to move people off dying nodes, or to redistribute the cluster. Ultimately though, it's probably a pretty lean startup and there is only so much experience and effort you can apply to each area.
fizx 11y ago
Distributed locking with liveness checks is exactly what zookeeper excels at. I wrote a one-click deploy (via cloudformation) self-healing zookeeper cluster on AWS (wrapping netflix exhibitor for the pretty GUI and config reloads) that has been running since the beginning of 2014 on ~$100/mo in production without issue. It's roughly 200-300 lines of ops config, because exhibitor makes everything easy. Consider biting the bullet! If you want to discuss further, my email is in my HN profile.