5 ms·

One important learning from the postmortem: always set your servers up to be in UTC rather than any other time zone. Helps debugging and log correlation and eli

One important learning from the postmortem: always set your servers up to be in UTC rather than any other time zone. Helps debugging and log correlation and eliminates confusion during incidents.

sytse 12y ago
Very true, we'll update the graph server that was misconfigured.
- Alupis 12y ago
  This is a very open and candid write-up. Very much appreciated by the community. Too many companies try to hide/cover-up their outages, but Gitlab is letting it all hang out, their mess-ups as well as things outside their control. I think that speaks a lot to the character of the company, and shows they really care.
  - sytse 12y ago
    Thanks Alipus, we're trying to be a part of the GitLab community in everything we do. Even if our operations go south we want that to inform others.
wpietri 12y ago
Is that the lesson? Or just that everything should be in the same time zone? If the company's staff is all in one time zone, I'm inclined to use that for servers, as otherwise people have to mentally juggle two time zones: local and UTC.
- res0nat0r 12y ago
  This starts out to be the way most companies think, then before you know it you are big enough to start opening datacenters in other countries, then you have to deal with timestamp conversion.
  - maddev 12y ago
    Why would you ever need to convert a timestamp? You typically parse a timestamp like "1999-03-29 20:02:04 CEST" and return a number of seconds since a given point in time (e.g. UNIX epoch if it must be). The returned time is obviously +0000 == UTC. If it's not your language/os/software sucks and its no wonder your hair is on fire.
    - forgottenpass 12y ago
      You typically parse a timestamp like "1999-03-29 20:02:04 CEST" and return a number of seconds since a given point in time (e.g. UNIX epoch if it must be). There's your conversion. I'm not saying it's not easy, I'm saying you're just pulling you out of a hole that your tools were configured to dig.
      - supine 12y ago
        An example of some of the pitfalls involved: http://search.cpan.org/~drolsky/DateTime-1.10/lib/DateTime.pm#How_DateTime_Math_Works http://search.cpan.org/~drolsky/DateTime-1.10/lib/DateTime.p...
    - deleted 12y ago
      [deleted]
    - korzun 12y ago
      Because it's always easier said than done. Multiple applications in different languages that are running on different environments and managed by different teams posses a real challenge.
      - maddev 12y ago
        It's not like am speculating. I implemented these things. The hardest part is agreeing on TZ symbols (e.g. Indian Standard Time vs Ihoa Standard Time). E.g. its not very hard to get right.
- felixgallo 12y ago
  That's a good question. UTC is almost certainly still the best candidate even in that case because: * it's not affected by daylight savings time, which in many candidate timezones causes at least two complex and error fraught time discontinuities every year, and which requires manual intervention at the whims of the US Congress in the USA. * UTC is the lingua franca of machine time communication -- if you ever have to, e.g., send data about transaction times, server event logs, order histories, etc., to any third party or API, then because there are 23 other timezones besides yours, and data normalization is important, they will almost certainly expect ISO8601 UTC. Yes, ISO8601 has time offsets. No, not all libraries implement them properly on either side. * The first time you ever have logs from two different timezones -- e.g., a service provider log denominated in UTC and yours in PST -- you will hate computers so much that you will likely quit your job, throw your keys on the floor, and go run a hotdog stand down by the pier rather than deal with it. This is an honest response to time and localization issues but think of your children.
  - supine 12y ago
    ...and just when you thought it was safe to go back outside leap seconds rear their ugly head and ruin your day.
peterwwillis 12y ago
Of course all hardware clocks should be set UTC, but there's a lot of benefit from using the locale to understand what's going on in relation to the human viewing the log. If i'm a human waking up at 3AM and I look at the logs for this server, the first thing i'm going to think is "is the time on this server the same as where I am?" followed by "how long ago did these events occur in relation to me?" The easiest way for any human to do this is to compare timezones. If you condition yourself to know the UTC difference of every time zone, then this works automatically, but [i'd argue] most humans are better at estimating timezone difference from other timezones, such as how California and New York are 3 hours difference, and New York and London is 5 hours difference. In terms of correlating log events across the globe, you really, really want a log parser and correlating tool. They make your life easier and reduce time and complication during events. Splunk, Loggly, Graylog, Logstash, ELSA, etc. Don't look at your logs by hand. You'll be sitting there all day with 6 split windows looking at mysql, nginx, app logs, kernel logs, mail logs, security logs, blah blah blah, just on one server. When you horizontally scale your app, whether it's 2 or 2,000 servers, you need something to parse and correlate your logs so you can say "show me all logs from 11:00PM to 2:00AM from Web Cluster B", you get to save 10 minutes on your outage, and you don't miss anything.
- supine 12y ago
  I think you just made a better case for having everything in UTC. Instead of having to remember multiple offsets, one for each location, you just need to know your current offset to get a sense of relativity (i.e. "ok, that occurred an hour ago") and the correlation across different locations and sources takes care of itself. I'm not saying that a log aggregator is not needed, they are still an important of any system as your 3rd paragraph clearly explains, but your 2nd paragraph actually makes the case for keeping everything in UTC regardless.

One important learning from the postmortem: always set your servers up to be in UTC rather than any other time zone. Helps debugging and log correlation and eliminates confusion during incidents.

sytse 12y ago

Very true, we'll update the graph server that was misconfigured.

Alupis 12y ago
This is a very open and candid write-up. Very much appreciated by the community. Too many companies try to hide/cover-up their outages, but Gitlab is letting it all hang out, their mess-ups as well as things outside their control. I think that speaks a lot to the character of the company, and shows they really care.
- sytse 12y ago
  Thanks Alipus, we're trying to be a part of the GitLab community in everything we do. Even if our operations go south we want that to inform others.

wpietri 12y ago

Is that the lesson? Or just that everything should be in the same time zone? If the company's staff is all in one time zone, I'm inclined to use that for servers, as otherwise people have to mentally juggle two time zones: local and UTC.

res0nat0r 12y ago
This starts out to be the way most companies think, then before you know it you are big enough to start opening datacenters in other countries, then you have to deal with timestamp conversion.
- maddev 12y ago
  Why would you ever need to convert a timestamp? You typically parse a timestamp like "1999-03-29 20:02:04 CEST" and return a number of seconds since a given point in time (e.g. UNIX epoch if it must be). The returned time is obviously +0000 == UTC. If it's not your language/os/software sucks and its no wonder your hair is on fire.
  - forgottenpass 12y ago
    You typically parse a timestamp like "1999-03-29 20:02:04 CEST" and return a number of seconds since a given point in time (e.g. UNIX epoch if it must be). There's your conversion. I'm not saying it's not easy, I'm saying you're just pulling you out of a hole that your tools were configured to dig.
    - supine 12y ago
      An example of some of the pitfalls involved: http://search.cpan.org/~drolsky/DateTime-1.10/lib/DateTime.pm#How_DateTime_Math_Works http://search.cpan.org/~drolsky/DateTime-1.10/lib/DateTime.p...
  - deleted 12y ago
    [deleted]
  - korzun 12y ago
    Because it's always easier said than done. Multiple applications in different languages that are running on different environments and managed by different teams posses a real challenge.
    - maddev 12y ago
      It's not like am speculating. I implemented these things. The hardest part is agreeing on TZ symbols (e.g. Indian Standard Time vs Ihoa Standard Time). E.g. its not very hard to get right.
felixgallo 12y ago
That's a good question. UTC is almost certainly still the best candidate even in that case because: * it's not affected by daylight savings time, which in many candidate timezones causes at least two complex and error fraught time discontinuities every year, and which requires manual intervention at the whims of the US Congress in the USA. * UTC is the lingua franca of machine time communication -- if you ever have to, e.g., send data about transaction times, server event logs, order histories, etc., to any third party or API, then because there are 23 other timezones besides yours, and data normalization is important, they will almost certainly expect ISO8601 UTC. Yes, ISO8601 has time offsets. No, not all libraries implement them properly on either side. * The first time you ever have logs from two different timezones -- e.g., a service provider log denominated in UTC and yours in PST -- you will hate computers so much that you will likely quit your job, throw your keys on the floor, and go run a hotdog stand down by the pier rather than deal with it. This is an honest response to time and localization issues but think of your children.
- supine 12y ago
  ...and just when you thought it was safe to go back outside leap seconds rear their ugly head and ruin your day.

peterwwillis 12y ago

Of course all hardware clocks should be set UTC, but there's a lot of benefit from using the locale to understand what's going on in relation to the human viewing the log. If i'm a human waking up at 3AM and I look at the logs for this server, the first thing i'm going to think is "is the time on this server the same as where I am?" followed by "how long ago did these events occur in relation to me?" The easiest way for any human to do this is to compare timezones. If you condition yourself to know the UTC difference of every time zone, then this works automatically, but [i'd argue] most humans are better at estimating timezone difference from other timezones, such as how California and New York are 3 hours difference, and New York and London is 5 hours difference. In terms of correlating log events across the globe, you really, really want a log parser and correlating tool. They make your life easier and reduce time and complication during events. Splunk, Loggly, Graylog, Logstash, ELSA, etc. Don't look at your logs by hand. You'll be sitting there all day with 6 split windows looking at mysql, nginx, app logs, kernel logs, mail logs, security logs, blah blah blah, just on one server. When you horizontally scale your app, whether it's 2 or 2,000 servers, you need something to parse and correlate your logs so you can say "show me all logs from 11:00PM to 2:00AM from Web Cluster B", you get to save 10 minutes on your outage, and you don't miss anything.

supine 12y ago
I think you just made a better case for having everything in UTC. Instead of having to remember multiple offsets, one for each location, you just need to know your current offset to get a sense of relativity (i.e. "ok, that occurred an hour ago") and the correlation across different locations and sources takes care of itself. I'm not saying that a log aggregator is not needed, they are still an important of any system as your 3rd paragraph clearly explains, but your 2nd paragraph actually makes the case for keeping everything in UTC regardless.