6 ms·

Wind gusts were reaching 125 MPH in Boulder county, if anyone’s curious. A lot of power was shut off preemptively to prevent downed power lines from starting wi

by arn3n 10mo ago

Wind gusts were reaching 125 MPH in Boulder county, if anyone’s curious. A lot of power was shut off preemptively to prevent downed power lines from starting wildfires. Energy providers gave warning to locals in advance. Shame that NIST’s backup generator failed, though.

Maxion 10mo ago
Somewhat interesting that they themselves don't have access to the site. You'd think there would have been some disaster plans put in place?
- TylerE 10mo ago
  Step One of most disaster plans is not to create a second emergency.
  - amelius 10mo ago
    But can't NTP server downtime cause a disaster?
    - Vosporos 10mo ago
      One (amongst many) NTP server going down creates less issues than an NTP server spreading wrong time.
      - macintux 10mo ago
        General rule of thumb: a misbehaving/slow server in any well-architected distributed system is vastly worse than a dead server.
        xeonmc 10mo ago
        i.e. a gaslighting husband is vastly worse than a dead husband.
      - PunchyHamster 10mo ago
        technically if you have 3 or more sources that would be caught; NTP protocol was designed for that eventuality
        da_chicken 10mo ago
        Sure, but not needing a failure to cascade to yet another failsafe is still a good idea. After all, all software has bugs, and all networks have configuration errors.
        throw0101c 10mo ago
        > technically if you have 3 or more sources that would be caught; NTP protocol was designed for that eventuality Either go with one clock in your NTPd/Chrony configuration, or ≥4. Yes, if you have 3 they can triangulate, but if one goes offline now you have 2 with no tie-breaker. If you have (at least) 4 servers, then one can go away and triangulation / sanity-checking can still occur with the 3 remaining.
        ufocia 10mo ago
        Your probably meant trilaterate.
    - idiotsecant 10mo ago
      If your application is so critical that NTP timing loss causes disaster and your holdover fails in less than a day and you aren't generating your own via gps, you are incompetent, full stop
      - geerlingguy 10mo ago
        And if things are that critical, you might have other references besides just GPS...
        throw0101c 10mo ago
        E.g., * https://www.meinbergglobal.com/english/products/ https://www.meinbergglobal.com/english/products/
  - ronjakoi 10mo ago
    Or even just a microsecond emergency.
    - f5129cac 10mo ago
      Bravo
- ssl-3 10mo ago
  Maybe this is the disaster plan: There's not a smouldering hole where NIST's Boulder facility used to be, and it will be operational again soon enough. There's no present need for important hard-to-replace sciencey-dudes to go into the shop (which is probably both cold, and dark, and may have other problems that make it unsafe: it's deliberately closed) to futz around with the the time machines. We still have other NTP clocks. Spooky-accurate clocks that the public can get to, even, like just up the road at NIST in Fort Collins (where WWVB lives, and which is currently up), and in Maryland. This is just one set. And beyond that, we've also got clocks in GPS satellites orbiting, and a whole world of low-stratum NTP servers that distribute that time on the network. (I have one such GPS-backed NTP server on the shelf behind me; there's not much to it.) And the orbital GPS clocks are controlled by the US Navy, not NIST. So there's redundancy in distribution, and also control, and some of the clocks aren't even on the Earth. Some people may be bit by this if their systems rely on only one NTP server, or only on the subset of them that are down. And if we're following section 3.2 of RFC 8633 and using multiple diverse NTP sources for our important stuff, then this event (while certainly interesting!) is not presently an issue at all.
  - Balgair 10mo ago
    There are many backup clocks/clusters that NIST uses as redundancies all around Boulder too, no need to even go up to Fort Collins. As in, NIST has fiber to a few at CU and a few commercial companies, last I checked. They're used in cases just like this one. Fun facts about The clock: You can't put anything in the room or take anything out. That's how sensitive the clock is. The room is just filled with asbestos. The actual port for the actual clock, the little metal thingy that is going buzz, buzz, buzz with voltage every second on the dot? Yeah, that little port isn't actually hooked up to anything, as again, it's so sensitive (impedance matching). So they use the other ports on the card for actual data transfer to the rest of the world. They do the adjustments so it's all fine in the end. But you have to define something as the second, and that little unused port is it. You can take a few pictures in the cramped little room, but you can't linger, as again, just your extra mass and gravity affects things fairly quickly. If there are more questions about time and timekeeping in general, go ahead and ask, though I'll probably get back to them a bit later today.
- cornholio 10mo ago
  The disater plan is to have a few dozens stratum 1 servers spread around the world, each connected to a distinct primary atomic clock, so that a catastrophic disaster needs to take down the global internet itself for all servers to become unreachable. The failure of a single such server is far from a disaster.
  - sounds 10mo ago
    For those of us near Boulder, it's urgent. But the stratum 1 time servers can shrug and route around the damage.
  - Lammy 10mo ago
    And the disaster plan for the disaster plan is to realize that it isn't that important at the human-level to have a clock meticulously set to correspond to other meticulously-set clocks, and that every attempt to force rigid timekeeping on humans is to try to make humans work more like machines rather than to make machines work more like humans.
    - zzzeek 10mo ago
      oh....no, not really, no, the world needs GPS, so, yeah. this is not like scrooge mcduck telling you to be at work on time. scrooge still has a windup watch
    - basilgohar 10mo ago
      I really, really can't get behind this sentiment. Having a reliable, accurate time keeping mechanism doesn't seem like an outlandish issue to want to maintain. Timekeeping has been an important mechanism for humans for as long as recorded history. I don't understand the wisdom of shooting down establishing systems to make that better, even if the direct applicability to a single human's life is remote. We are all part of a huge, interconnected system whether we like it or not, and accurate, synchronized timekeeping across the world does not sound nefarious to me.
      - Lammy 10mo ago
- toast0 10mo ago
  If access to the site is unsafe and thus the site is closed; not having access seems reasonable. Time services are available from other locations. That's the disaster plan. I'm sure there will be some negative consequences from this downtime, especially if all the Boulder reference time sources lose power, but disaster plans mitigate negative consequences, they can't eliminate them. Utility power fails, automatic transfer switches fail, backup generators fail, building fires happen, etc. Sometimes the system has to be shut down.
  - quei7Ahm 10mo ago
    At this time the local power utility shows: Total of 140 outages affecting 24,981 customers. [1] Remember - the Marshall fire started on December 30, 2021 under similar high wind conditions. [2] 1,000 structures destroyed, two deaths. [1] https://co.my.xcelenergy.com/s/outage-safety/outage-map https://co.my.xcelenergy.com/s/outage-safety/outage-map [2] https://en.wikipedia.org/wiki/Marshall_Fire https://en.wikipedia.org/wiki/Marshall_Fire
IncreasePosts 10mo ago
Notably, we had the marshal fire here 4 years ago and recently Xcel settled for $680M for their role in the fire. So they're probably pretty keen not to be on the hook again
- tpoindex 10mo ago
  For more background on the Marshal Fire of Dec. 2021: https://en.wikipedia.org/wiki/Marshall_Fire https://en.wikipedia.org/wiki/Marshall_Fire tl;dr - the fire destroyed over 1,000 homes, two deaths. The local electrical utility, Xcel, was found as a contributing cause from sparking power lines during a strong wind storm. As a result, electrical utilities now cut power to affected areas during strong winds.
- simojo 10mo ago
  I guess that explains why they had no qualms shutting down half of Boulder's power with a vague time horizon. After losing everything in my fridge, though, they finally turned it back on today.
  - ssl-3 10mo ago
    Indeed. Losing the contents of (lots of) fridges is cheaper, as a whole, than incidentally burning the countryside. We all ultimately pay for the result no matter what, so that seems like a reasonably-sensible bet. On the fridge itself: You may find that the contents are insured against power outages. As an anecdote, my (completely not-special) homeowner's insurance didn't protest at all about writing a check for the contents of my fridge and freezer when I asked about that, after my house was without power for a couple of weeks following the 2008 derecho. This rather small claim didn't affect my rate in any way that I could perceive. And to digress a bit: I have a chest freezer. These days I fill up the extra space in the freezer with water -- with "single-use" plastic containers (water bottles, milk jugs) that would normally be landfilled or recycled. This does a couple of things: On normal days, it increases thermal mass of the freezer, and that improves the cycle times for the compressor in ways that tend to make it happier over time. In the abnormal event of a long power outage, it also provides a source of ice that is chilled to 0F/18C that I can relocate into the fridge (or into a cooler, perhaps for transport), to keep cold stuff cold. It's not a long-term solution, but it'll help ensure that I've got a fairly normal supply of fresh food to eat for a couple of days if the power dips. And it's pretty low-effort on my part. I've probably spent nearly as much effort writing about this system here just now as I have on implementing it.
HaZeust 10mo ago
Yup, here in Jefferson County - roughly 30 minutes south of Boulder County, we were getting wind gusts around 80mph.
goku12 10mo ago
> Wind gusts were reaching 125 MPH in Boulder county, if anyone’s curious. That's some strong winds! What's causing such strong sustained/gusty winds that long? I'm hearing about this weather phenomenon for the first time.

by arn3n 10mo ago

Maxion 10mo ago

Somewhat interesting that they themselves don't have access to the site. You'd think there would have been some disaster plans put in place?

TylerE 10mo ago
Step One of most disaster plans is not to create a second emergency.
- amelius 10mo ago
  But can't NTP server downtime cause a disaster?
  - Vosporos 10mo ago
    One (amongst many) NTP server going down creates less issues than an NTP server spreading wrong time.
    - macintux 10mo ago
      General rule of thumb: a misbehaving/slow server in any well-architected distributed system is vastly worse than a dead server.
      - xeonmc 10mo ago
        i.e. a gaslighting husband is vastly worse than a dead husband.
    - PunchyHamster 10mo ago
      technically if you have 3 or more sources that would be caught; NTP protocol was designed for that eventuality
      - da_chicken 10mo ago
        Sure, but not needing a failure to cascade to yet another failsafe is still a good idea. After all, all software has bugs, and all networks have configuration errors.
      - throw0101c 10mo ago
        > technically if you have 3 or more sources that would be caught; NTP protocol was designed for that eventuality Either go with one clock in your NTPd/Chrony configuration, or ≥4. Yes, if you have 3 they can triangulate, but if one goes offline now you have 2 with no tie-breaker. If you have (at least) 4 servers, then one can go away and triangulation / sanity-checking can still occur with the 3 remaining.
        ufocia 10mo ago
        Your probably meant trilaterate.
  - idiotsecant 10mo ago
    If your application is so critical that NTP timing loss causes disaster and your holdover fails in less than a day and you aren't generating your own via gps, you are incompetent, full stop
    - geerlingguy 10mo ago
      And if things are that critical, you might have other references besides just GPS...
      - throw0101c 10mo ago
        E.g., * https://www.meinbergglobal.com/english/products/ https://www.meinbergglobal.com/english/products/
- ronjakoi 10mo ago
  Or even just a microsecond emergency.
  - f5129cac 10mo ago
    Bravo
ssl-3 10mo ago
Maybe this is the disaster plan: There's not a smouldering hole where NIST's Boulder facility used to be, and it will be operational again soon enough. There's no present need for important hard-to-replace sciencey-dudes to go into the shop (which is probably both cold, and dark, and may have other problems that make it unsafe: it's deliberately closed) to futz around with the the time machines. We still have other NTP clocks. Spooky-accurate clocks that the public can get to, even, like just up the road at NIST in Fort Collins (where WWVB lives, and which is currently up), and in Maryland. This is just one set. And beyond that, we've also got clocks in GPS satellites orbiting, and a whole world of low-stratum NTP servers that distribute that time on the network. (I have one such GPS-backed NTP server on the shelf behind me; there's not much to it.) And the orbital GPS clocks are controlled by the US Navy, not NIST. So there's redundancy in distribution, and also control, and some of the clocks aren't even on the Earth. Some people may be bit by this if their systems rely on only one NTP server, or only on the subset of them that are down. And if we're following section 3.2 of RFC 8633 and using multiple diverse NTP sources for our important stuff, then this event (while certainly interesting!) is not presently an issue at all.
- Balgair 10mo ago
  There are many backup clocks/clusters that NIST uses as redundancies all around Boulder too, no need to even go up to Fort Collins. As in, NIST has fiber to a few at CU and a few commercial companies, last I checked. They're used in cases just like this one. Fun facts about The clock: You can't put anything in the room or take anything out. That's how sensitive the clock is. The room is just filled with asbestos. The actual port for the actual clock, the little metal thingy that is going buzz, buzz, buzz with voltage every second on the dot? Yeah, that little port isn't actually hooked up to anything, as again, it's so sensitive (impedance matching). So they use the other ports on the card for actual data transfer to the rest of the world. They do the adjustments so it's all fine in the end. But you have to define something as the second, and that little unused port is it. You can take a few pictures in the cramped little room, but you can't linger, as again, just your extra mass and gravity affects things fairly quickly. If there are more questions about time and timekeeping in general, go ahead and ask, though I'll probably get back to them a bit later today.
cornholio 10mo ago
The disater plan is to have a few dozens stratum 1 servers spread around the world, each connected to a distinct primary atomic clock, so that a catastrophic disaster needs to take down the global internet itself for all servers to become unreachable. The failure of a single such server is far from a disaster.
- sounds 10mo ago
  For those of us near Boulder, it's urgent. But the stratum 1 time servers can shrug and route around the damage.
- Lammy 10mo ago
  And the disaster plan for the disaster plan is to realize that it isn't that important at the human-level to have a clock meticulously set to correspond to other meticulously-set clocks, and that every attempt to force rigid timekeeping on humans is to try to make humans work more like machines rather than to make machines work more like humans.
  - zzzeek 10mo ago
    oh....no, not really, no, the world needs GPS, so, yeah. this is not like scrooge mcduck telling you to be at work on time. scrooge still has a windup watch
  - basilgohar 10mo ago
    I really, really can't get behind this sentiment. Having a reliable, accurate time keeping mechanism doesn't seem like an outlandish issue to want to maintain. Timekeeping has been an important mechanism for humans for as long as recorded history. I don't understand the wisdom of shooting down establishing systems to make that better, even if the direct applicability to a single human's life is remote. We are all part of a huge, interconnected system whether we like it or not, and accurate, synchronized timekeeping across the world does not sound nefarious to me.
    - Lammy 10mo ago
toast0 10mo ago
If access to the site is unsafe and thus the site is closed; not having access seems reasonable. Time services are available from other locations. That's the disaster plan. I'm sure there will be some negative consequences from this downtime, especially if all the Boulder reference time sources lose power, but disaster plans mitigate negative consequences, they can't eliminate them. Utility power fails, automatic transfer switches fail, backup generators fail, building fires happen, etc. Sometimes the system has to be shut down.
- quei7Ahm 10mo ago
  At this time the local power utility shows: Total of 140 outages affecting 24,981 customers. [1] Remember - the Marshall fire started on December 30, 2021 under similar high wind conditions. [2] 1,000 structures destroyed, two deaths. [1] https://co.my.xcelenergy.com/s/outage-safety/outage-map https://co.my.xcelenergy.com/s/outage-safety/outage-map [2] https://en.wikipedia.org/wiki/Marshall_Fire https://en.wikipedia.org/wiki/Marshall_Fire

IncreasePosts 10mo ago

Notably, we had the marshal fire here 4 years ago and recently Xcel settled for $680M for their role in the fire. So they're probably pretty keen not to be on the hook again

tpoindex 10mo ago
For more background on the Marshal Fire of Dec. 2021: https://en.wikipedia.org/wiki/Marshall_Fire https://en.wikipedia.org/wiki/Marshall_Fire tl;dr - the fire destroyed over 1,000 homes, two deaths. The local electrical utility, Xcel, was found as a contributing cause from sparking power lines during a strong wind storm. As a result, electrical utilities now cut power to affected areas during strong winds.
simojo 10mo ago
I guess that explains why they had no qualms shutting down half of Boulder's power with a vague time horizon. After losing everything in my fridge, though, they finally turned it back on today.
- ssl-3 10mo ago
  Indeed. Losing the contents of (lots of) fridges is cheaper, as a whole, than incidentally burning the countryside. We all ultimately pay for the result no matter what, so that seems like a reasonably-sensible bet. On the fridge itself: You may find that the contents are insured against power outages. As an anecdote, my (completely not-special) homeowner's insurance didn't protest at all about writing a check for the contents of my fridge and freezer when I asked about that, after my house was without power for a couple of weeks following the 2008 derecho. This rather small claim didn't affect my rate in any way that I could perceive. And to digress a bit: I have a chest freezer. These days I fill up the extra space in the freezer with water -- with "single-use" plastic containers (water bottles, milk jugs) that would normally be landfilled or recycled. This does a couple of things: On normal days, it increases thermal mass of the freezer, and that improves the cycle times for the compressor in ways that tend to make it happier over time. In the abnormal event of a long power outage, it also provides a source of ice that is chilled to 0F/18C that I can relocate into the fridge (or into a cooler, perhaps for transport), to keep cold stuff cold. It's not a long-term solution, but it'll help ensure that I've got a fairly normal supply of fresh food to eat for a couple of days if the power dips. And it's pretty low-effort on my part. I've probably spent nearly as much effort writing about this system here just now as I have on implementing it.

HaZeust 10mo ago

Yup, here in Jefferson County - roughly 30 minutes south of Boulder County, we were getting wind gusts around 80mph.

goku12 10mo ago

> Wind gusts were reaching 125 MPH in Boulder county, if anyone’s curious. That's some strong winds! What's causing such strong sustained/gusty winds that long? I'm hearing about this weather phenomenon for the first time.