4 ms·

> the cold startup time of the JVM itself (which is not relevant on a server in comparison to desktop Java apps). I disagree somewhat with this. We found that

> the cold startup time of the JVM itself (which is not relevant on a server in comparison to desktop Java apps). I disagree somewhat with this. We found that when we started writing microservices in languages that are not java, the short startup time changed how we did some error handling. For errors where we say lose connection to the database, or rabbitmq, we much rather have the nodejs-process die and restart, than try to construct reconnect logic. The problem with reconnect-logic is that it is code that (may) be tested very rarely. This in turn means it's easy to get strange long term problems there like very slow memory leak due to some listener being added to a connection object once the connection is initiated. We did a 180 on reconnect-logic in our nodejs-processes and let the exceptions just bubble unhandled and take the entire vm down. With automatic restart script, the process will be back in seconds anyway, and with docker having built in back-off timers for auto restart, we don't necessarily overload the shared resources.

mason55 10y ago
I guess I don't see the difference here if your VM startup time is 3 seconds or 15-30 seconds. If that's the difference between the site remaining stable and the whole thing collapsing then it seems like you're setting yourself up for a big outage one day when the nodejs process isn't able to come back in three seconds for whatever reason.
- algesten 10y ago
  I think it depends a bit on class of errors. Certainly not everything is suitable for this treatment. Lost connectivity to RabbitMQ or Elasticsearch would mean our site is dead anyhow (you can't do anything). So either of those errors should arguably result in some static 500 pardon-our-appearance page. But say someone messes up the network connection or we get a brief problem. Why wouldn't the nodejs process start quickly?
wtetzner 10y ago
> For errors where we say lose connection to the database, or rabbitmq, we much rather have the nodejs-process die and restart, than try to construct reconnect logic. This sounds like a very Erlang-ish way to handle the problem. Another advantage is that if the server/process is in some weird state that's causing problems, killing and restarting it lets you clear out the broken state, and get back into the state that it's most likely been tested under.
- algesten 10y ago
  Yes. We're almost now risking it going the other way, that some bad programming goes unchecked for a long time, because overall, the process sort of does what it should. Even if it restarts like 10 times a day.
  - wtetzner 10y ago
    Well, you do still want good reporting, so you know when failures are happening and can capture a stack trace.
mickronome 10y ago
The most effective way to handle these kind of errors in Java unfortunately requires understanding class loading, thread contexts, wrapping connection primitives in the right kind of references, and then making sure that all resource deallocation/closing always use the same codepath. Even though you really only need to implemnt it once, it is both somewhat tricky and technically challenging. It's a pity that so few Java projects have tried to use these mechanisms without building them as part of massive frameworks, sometimes apparently even without understanding what they have built.
- algesten 10y ago
  Yes, I did spend large parts of my java developer looking at class loaders and class loading delegations in servlet containers etc. I think it's a bit too hard to get it right. Like, suddenly some third party library starts pulling in log4j and your whole logging setup goes wrong in subtle yet very bad ways. Or you screwed up with that one reference to a ResultSet and even though it is closed, that reference keeps an entire class tree of Connection, PreparedStatement etc alive.
  - deleted 10y ago
    [deleted]
twic 10y ago
A JVM boots in about 100 milliseconds. A difference of 100 milliseconds made the difference in how you do error handling?
- brianwawok 10y ago
  On what box with what kind of disk for how big of app? You aren't getting 100ms starts on a 199 mb fatjar on Amazon EBS.
saryant 10y ago
Isn't this also solved by just load balancing so that the customer ends up reconnected to a healthy node while the downed node is replaced? We run our Scala apps on Aurora/Mesos behind a load balancer (hundreds of instances for just one app). If there's an issue that can't be handled within the app and error rates breach a given threshold, Aurora just kills the instance and creates a new one on another host.

mason55 10y ago

I guess I don't see the difference here if your VM startup time is 3 seconds or 15-30 seconds. If that's the difference between the site remaining stable and the whole thing collapsing then it seems like you're setting yourself up for a big outage one day when the nodejs process isn't able to come back in three seconds for whatever reason.

algesten 10y ago
I think it depends a bit on class of errors. Certainly not everything is suitable for this treatment. Lost connectivity to RabbitMQ or Elasticsearch would mean our site is dead anyhow (you can't do anything). So either of those errors should arguably result in some static 500 pardon-our-appearance page. But say someone messes up the network connection or we get a brief problem. Why wouldn't the nodejs process start quickly?

wtetzner 10y ago

> For errors where we say lose connection to the database, or rabbitmq, we much rather have the nodejs-process die and restart, than try to construct reconnect logic. This sounds like a very Erlang-ish way to handle the problem. Another advantage is that if the server/process is in some weird state that's causing problems, killing and restarting it lets you clear out the broken state, and get back into the state that it's most likely been tested under.

algesten 10y ago
Yes. We're almost now risking it going the other way, that some bad programming goes unchecked for a long time, because overall, the process sort of does what it should. Even if it restarts like 10 times a day.
- wtetzner 10y ago
  Well, you do still want good reporting, so you know when failures are happening and can capture a stack trace.

mickronome 10y ago

The most effective way to handle these kind of errors in Java unfortunately requires understanding class loading, thread contexts, wrapping connection primitives in the right kind of references, and then making sure that all resource deallocation/closing always use the same codepath. Even though you really only need to implemnt it once, it is both somewhat tricky and technically challenging. It's a pity that so few Java projects have tried to use these mechanisms without building them as part of massive frameworks, sometimes apparently even without understanding what they have built.

algesten 10y ago
Yes, I did spend large parts of my java developer looking at class loaders and class loading delegations in servlet containers etc. I think it's a bit too hard to get it right. Like, suddenly some third party library starts pulling in log4j and your whole logging setup goes wrong in subtle yet very bad ways. Or you screwed up with that one reference to a ResultSet and even though it is closed, that reference keeps an entire class tree of Connection, PreparedStatement etc alive.
- deleted 10y ago
  [deleted]

twic 10y ago

A JVM boots in about 100 milliseconds. A difference of 100 milliseconds made the difference in how you do error handling?

brianwawok 10y ago
On what box with what kind of disk for how big of app? You aren't getting 100ms starts on a 199 mb fatjar on Amazon EBS.

saryant 10y ago

Isn't this also solved by just load balancing so that the customer ends up reconnected to a healthy node while the downed node is replaced? We run our Scala apps on Aurora/Mesos behind a load balancer (hundreds of instances for just one app). If there's an issue that can't be handled within the app and error rates breach a given threshold, Aurora just kills the instance and creates a new one on another host.