4 ms·

Hey guys -- the most resource-intensive part of the infrastructure (the game master boxes) is currently overwhelmed by your enthusiasm. We're spinning up more

• archgoon 11y ago
  Congrats on having the problems you want to have. :)
• tptacek 11y ago
  I BLAME RAILS.
  • carbocation 11y ago
    I thought your stack was mostly go?
    • kasey_junk 11y ago
      I suspect that was irony...
      • mwcampbell 11y ago
        In a blog post some months ago [1], Patrick said that some components do use Rails. [1] http://www.kalzumeus.com/2015/08/20/designing-and-building-stockfighter-our-programming-game/ http://www.kalzumeus.com/2015/08/20/designing-and-building-s...
        • joshmn 11y ago
          I know using Rails isn't as popular here as it once was, but in my experience, if you cache correctly (like you should), it's trivial to scale.
          • patio11 11y ago
            Modestly less trivial if you happen to put your chat server on the box with Rails because Rails is certainly not going to fall over and your chat server attempts to open several thousand websockets on a box with a not-too-generous number of open file descriptors, it appears.
            • jerf 11y ago
              "Make sure you've turned up the file descriptors" is the single-process socket server equivalent of "Turn off HTTP KeepAlive" for Apache sites. :) (Yeah, I've had outage because of this too.)
              • ehsanu1 11y ago
                Surely you meant "Turn on HTTP pipelining"? Or is there something wrong with Apache's pipelining implementation?
                • MichaelGG 11y ago
                  As I understand, HTTP pipelining is basically a no-go in the real world. Broken implementations, plus performance implications since responses must go in order. Edit: Here's a FF discussion on it: https://bugzilla.mozilla.org/show_bug.cgi?id=264354 https://bugzilla.mozilla.org/show_bug.cgi?id=264354 -
                • jerf 11y ago
                  I misremembered a bit; it was Turn Off Apache Keepalive: https://hn.algolia.com/?query=patio11%20apache%20keepalive&sort=byPopularity&prefix&page=0&dateRange=all&type=comment https://hn.algolia.com/?query=patio11%20apache%20keepalive&s...
            • mbesto 11y ago
              I feel like a chat server is something best served by something not in your stack (especially Rails). I surprised you chose to do it in house. How about using https://pusher.com/ https://pusher.com/ or https://www.firebase.com/ https://www.firebase.com/ ?
      • lopatin 11y ago
        Looks like it's actually Go based on one of the JSON responses I found in the web page: { ok: false, error: "strconv.ParseInt: parsing "NaN": invalid syntax" }
    • tptacek 11y ago
      It's mostly Go except where it isn't.
    • gjm11 11y ago
      It currently appears to be mostly stop.
  • patio11 11y ago
    Thomas and I have a running gag about this. Rails is holding up admirably. You can still actually see the website. Our problem is presently that one player implies 1+ ruby (not Rails) processes running bots and doing orchestration for their levels, and that has pegged the CPU on all of our GM boxes and also, possibly, saturated their network links as they hit the venue with... I don't actually know how many orders have been placed already but I'm going to wager a guess that it's "a lot."
    • joeyspn 11y ago
      > Rails is holding up admirably. You can still actually see the website. Site is down now... (at least in Europe)
      • danielvf 11y ago
        Site has been mostly down for me in the US over the last ten minutes.
      • OJFord 11y ago
        Give it time, for me it thought about it a few minutes, and then rendered :)
    • deleted 11y ago
      [deleted]
    • viraptor 11y ago
      I so want to read a postmortem of the launch issues. It's one thing when a corporate project with crazy requirements goes down (almost expected these days :( ). Completely different when a service started/run by known tech people with their own schedule breaks on launch. I'd be really interested in what testing was done and why it didn't match reality. (not a jab/criticism, just something to actually learn from)
      • patio11 11y ago
        Meh? It's a play-money stock exchange which processed over 200k orders since a few seconds ago. We're in the process of finding the parts of the plumbing that weren't sized right for hundreds of simultaneous players and tens of thousands of simultaneous actors (each bot is approximately as taxing on the infra as a human -- more, at the moment, since many folks can't read the instructions to start hitting us with orders and the bots already know how to do that).
• zeeshanm 11y ago
  These are definitely good problems to have!
  • pg_is_a_butt 11y ago
    the kind of problems where i leave the site forever, and never trust the founders with anything ever again? YEAH!
• Uptrenda 11y ago
  I hope you get this fixed soon. It can be quite stressful having your exposure period potentially damaged because your software or infrastructure can't handle the sudden influx. Stick to deep breathes and think it through :)
  • patio11 11y ago
    I'm pretty calm at the moment, particularly now that the servers seem to be cooling down a bit. Startup culture says Launch Days Are Really Important. This is not our first rodeo. We know 0.01% of the people who are going to play this game are even aware it exists at the moment. We're pleased as punch that y'all are playing and some of you are progressing in it, but long-term, a few teething problems on launch day is a no-op.