5 ms·

> just to insert a CD and press reset That may have been all he did that time, but imagine if stuff had gone sideways, he'd have to troubleshoot it from Antarc

> just to insert a CD and press reset That may have been all he did that time, but imagine if stuff had gone sideways, he'd have to troubleshoot it from Antarctica! I thought working from home was bad...

cozzyd 8y ago
I work on an experiment at the south pole (not IceCube). sshing into fix stuff is painful, even when the "good" satellite is up. Protip: pressing up several times and then enter is very dangerous since the command that you will actually run may not have appeared on your screen yet.
- beefield 8y ago
  Jusr amoment ago learned about mosh.org that sounds like would be helpful for that kind of situations.
- thrownhwn 8y ago
  Similar experience to being a hacker-cracker 20 yrs ago, bouncing off servers around the world over a modem connection to telnet into some poor SunOS 4 boxes in South Korea or at GSFC (NASA). Sometimes you had to wait minutes before anything appeared on the screen while TCP was doing its thing. God forbid if you accidentally (yp)cat’d some large file..:)
  - deleted 8y ago
    [deleted]
- azernik 8y ago
  Since you're here! Is Iridium out of your price range, or does even that have latency/bandwidth issues for the kind of SSH work you're talking about?
  - cozzyd 8y ago
    iridium is available all the time but very slow and the scarce bandwidth is not prioritized for incidental ssh access. The problem is that there are a lot of people / experiments at Pole.
  - bacon_waffle 8y ago
    At least a few years ago, IceCube had it's own Iridium modems, in addition to a small pool of them for the station at large. Latency is very high, bandwidth is very low. It's much better when you can send an email over Iridium to ask for help from a human on the other side, than to SSH and do it yourself. I think the situation now is a lot better than it was, but a few years ago, most of the day there was no connectivity except Iridium. Most communications satellites are in orbits that don't take them near the poles, with Iridum being the obvious exception. Related: https://en.wikipedia.org/wiki/GOES_3 https://en.wikipedia.org/wiki/GOES_3 was an amazing piece of the South Pole infrastructure for several years. There was a period every year where the Earth would eclipse GOES-3 - the batteries onboard were long dead - so the satellite would power down and everyone just sorta crossed their fingers and hoped it came back online with the right orientation and with enough electronics going so that we could keep using it. Bandwidth wasn't great, but it was quite reliable especially considering that it was launched in 1978.
- foobarian 8y ago
  Even on a good console I put a # at the start of the line before editing or typing it in case I accidentally hit Enter too soon. And never ever paste commands without first trying it into a trash terminal or editor.
  - craftyguy 8y ago
    That's an interesting technique that I've never heard of. Do you have fast shortcuts/keybindings for removing the # when you are ready, or is it '(left arrow x N) + ^H' each time?
    - scarejunba 8y ago
      ^A ^D
      - foobarian 8y ago
        ^A ^D it is. But since you bring up moving the cursor, it also is very helpful to 'xset r rate 160 80' to speed it up substantially in Xwindows.
      - craftyguy 8y ago
        I assume (by trying on zsh) that this is a configuration change you made vs. some keybinding I've yet to discover?
        juki 8y ago
        Those are Emacs style keybindings, which are the default on most shells. You've probably switched to Vi style bindings in zsh.
        adrianN 8y ago
        https://dwmkerr.com/effective-shell-part-1-navigating-the-command-line/ https://dwmkerr.com/effective-shell-part-1-navigating-the-co...
        dredmorbius 8y ago
- tobias2014 8y ago
  Use mosh and you will have a much better experience. It's made for high latency connections.
  - craftyguy 8y ago
    Some mission critical production systems don't really want you installing <insert current trendy software here>. That's probably a highly unpopular view on HN, but having downtime on IceCube, for example, means they may miss an extremely rare event (neutrino hitting something and generating muons/photon shockwaves) that they could detect.
    - snovv_crash 8y ago
      Something released in 2012 isn't exactly what I'd call "trendy". Technically it really does have some improvements for poor latency and high-packetloss connections.
      - consp 8y ago
        I'm pretty sure they consider everything sub 10 years old "new and untested" in this field, on this server/array.
    - de_watcher 8y ago
      Mosh has been around for a while, it's not like "current trendy software". I thought people use it for bad connections (and envied them because of my stuff in the jungle that isn't performant enough for ssh).
    - eeZah7Ux 8y ago
      Indeed. Mosh is still new (less than 10 years old), albeit it could get an exception if it really helps avoiding errors and typos due to latency.
dekhn 8y ago
Yes, that's why he was sent. In case something went wrong, he was qualified to fix it in production. I've come to an appreciation for people who have jobs where they don't do much most of the time, but when the shit hits the fan, they do exactly what was required.
- munificent 8y ago
  You can think about it much like load managing servers: A job has a task load that varies over time. Those tasks also have some maximum latency before Bad Things happen. Thus, you need enough worker capacity to handle the peak load. You can allocate new workers on demand, but that also has a certain warm-up time which can be problematic if it's lower than the maximum latency. For most white-collar service-oriented jobs, the variance of that task load is quite smooth. If you're a software dev not working directly on a production service, there aren't that many urgent surprises. Also, the max latency is really high — many tasks can be put off till tomorrow, next sprint, etc. That makes it easy to allocate just enough workers to handle your average task load, keep them busy all day, and rarely need to spin up or spin down new ones. But some jobs just have really high variance or really low latency. No one knows when a patient is going to show up at a trauma hospital but when one does, you need the surgeon on site right now and not somewhere stuck in traffic. For that kind of work, the logical thing to do then is to have spare idle worker capacity. It seems inefficient, but it's cheaper than the cost of missing your latency targets when a spike happens. You can reduce the idle capacity needed by reducing variance. For example if you flew all your trauma patients to one hospital, that actually lets you allocate surgeons more efficiently. Because then at that point you have spikes frequently enough that the aggregate number of patients in a given day becomes more consistently predictable. Instead of one surgeon each sitting on their thumbs most of the day at ten city hospitals, you get three surgeons that reliably have a couple of patients a day at the regional hospital. (But of course there is the overhead of getting all the patients there.) Also, you can reduce idle capacity by lowering warm-up time. That's why doctors have pagers — it reduces the time between a patient coming in and the doctor being ready to help. Or you can increase the max latency, though this can be hard based on the nature of the work. For example, if you're better able to stabilize patients, you may be able to wait longer until the doctor is on-site. In this case, the "install Debian" job has really high variance. 99% of the time it's a click, 1% of the time, you've gotta know some Linux internals. The max latency is low because you don't want the IceCube array non-operational for long. And the warm-up time is really high — fly someone out to the South Pole. So the reasonable response is to throw money at idle capacity and send someone out pre-emptively.
robocat 8y ago
I met some scientists (at the pub) from NASA the other day - they had been on a plane going repeatably from sea level to stratosphere all the way from from Hawaii to Chirstchurch,NZ to sample the atmosphere. They had designed and built their own instruments, but they were on the plane so they could fix any problems, although the idea was to have no problems!

cozzyd 8y ago

I work on an experiment at the south pole (not IceCube). sshing into fix stuff is painful, even when the "good" satellite is up. Protip: pressing up several times and then enter is very dangerous since the command that you will actually run may not have appeared on your screen yet.

beefield 8y ago
Jusr amoment ago learned about mosh.org that sounds like would be helpful for that kind of situations.
thrownhwn 8y ago
Similar experience to being a hacker-cracker 20 yrs ago, bouncing off servers around the world over a modem connection to telnet into some poor SunOS 4 boxes in South Korea or at GSFC (NASA). Sometimes you had to wait minutes before anything appeared on the screen while TCP was doing its thing. God forbid if you accidentally (yp)cat’d some large file..:)
- deleted 8y ago
  [deleted]
azernik 8y ago
Since you're here! Is Iridium out of your price range, or does even that have latency/bandwidth issues for the kind of SSH work you're talking about?
- cozzyd 8y ago
  iridium is available all the time but very slow and the scarce bandwidth is not prioritized for incidental ssh access. The problem is that there are a lot of people / experiments at Pole.
- bacon_waffle 8y ago
  At least a few years ago, IceCube had it's own Iridium modems, in addition to a small pool of them for the station at large. Latency is very high, bandwidth is very low. It's much better when you can send an email over Iridium to ask for help from a human on the other side, than to SSH and do it yourself. I think the situation now is a lot better than it was, but a few years ago, most of the day there was no connectivity except Iridium. Most communications satellites are in orbits that don't take them near the poles, with Iridum being the obvious exception. Related: https://en.wikipedia.org/wiki/GOES_3 https://en.wikipedia.org/wiki/GOES_3 was an amazing piece of the South Pole infrastructure for several years. There was a period every year where the Earth would eclipse GOES-3 - the batteries onboard were long dead - so the satellite would power down and everyone just sorta crossed their fingers and hoped it came back online with the right orientation and with enough electronics going so that we could keep using it. Bandwidth wasn't great, but it was quite reliable especially considering that it was launched in 1978.
foobarian 8y ago
Even on a good console I put a # at the start of the line before editing or typing it in case I accidentally hit Enter too soon. And never ever paste commands without first trying it into a trash terminal or editor.
- craftyguy 8y ago
  That's an interesting technique that I've never heard of. Do you have fast shortcuts/keybindings for removing the # when you are ready, or is it '(left arrow x N) + ^H' each time?
  - scarejunba 8y ago
    ^A ^D
    - foobarian 8y ago
      ^A ^D it is. But since you bring up moving the cursor, it also is very helpful to 'xset r rate 160 80' to speed it up substantially in Xwindows.
    - craftyguy 8y ago
      I assume (by trying on zsh) that this is a configuration change you made vs. some keybinding I've yet to discover?
      - juki 8y ago
        Those are Emacs style keybindings, which are the default on most shells. You've probably switched to Vi style bindings in zsh.
      - adrianN 8y ago
        https://dwmkerr.com/effective-shell-part-1-navigating-the-command-line/ https://dwmkerr.com/effective-shell-part-1-navigating-the-co...
      - dredmorbius 8y ago
tobias2014 8y ago
Use mosh and you will have a much better experience. It's made for high latency connections.
- craftyguy 8y ago
  Some mission critical production systems don't really want you installing <insert current trendy software here>. That's probably a highly unpopular view on HN, but having downtime on IceCube, for example, means they may miss an extremely rare event (neutrino hitting something and generating muons/photon shockwaves) that they could detect.
  - snovv_crash 8y ago
    Something released in 2012 isn't exactly what I'd call "trendy". Technically it really does have some improvements for poor latency and high-packetloss connections.
    - consp 8y ago
      I'm pretty sure they consider everything sub 10 years old "new and untested" in this field, on this server/array.
  - de_watcher 8y ago
    Mosh has been around for a while, it's not like "current trendy software". I thought people use it for bad connections (and envied them because of my stuff in the jungle that isn't performant enough for ssh).
  - eeZah7Ux 8y ago
    Indeed. Mosh is still new (less than 10 years old), albeit it could get an exception if it really helps avoiding errors and typos due to latency.

dekhn 8y ago

Yes, that's why he was sent. In case something went wrong, he was qualified to fix it in production. I've come to an appreciation for people who have jobs where they don't do much most of the time, but when the shit hits the fan, they do exactly what was required.

munificent 8y ago
You can think about it much like load managing servers: A job has a task load that varies over time. Those tasks also have some maximum latency before Bad Things happen. Thus, you need enough worker capacity to handle the peak load. You can allocate new workers on demand, but that also has a certain warm-up time which can be problematic if it's lower than the maximum latency. For most white-collar service-oriented jobs, the variance of that task load is quite smooth. If you're a software dev not working directly on a production service, there aren't that many urgent surprises. Also, the max latency is really high — many tasks can be put off till tomorrow, next sprint, etc. That makes it easy to allocate just enough workers to handle your average task load, keep them busy all day, and rarely need to spin up or spin down new ones. But some jobs just have really high variance or really low latency. No one knows when a patient is going to show up at a trauma hospital but when one does, you need the surgeon on site right now and not somewhere stuck in traffic. For that kind of work, the logical thing to do then is to have spare idle worker capacity. It seems inefficient, but it's cheaper than the cost of missing your latency targets when a spike happens. You can reduce the idle capacity needed by reducing variance. For example if you flew all your trauma patients to one hospital, that actually lets you allocate surgeons more efficiently. Because then at that point you have spikes frequently enough that the aggregate number of patients in a given day becomes more consistently predictable. Instead of one surgeon each sitting on their thumbs most of the day at ten city hospitals, you get three surgeons that reliably have a couple of patients a day at the regional hospital. (But of course there is the overhead of getting all the patients there.) Also, you can reduce idle capacity by lowering warm-up time. That's why doctors have pagers — it reduces the time between a patient coming in and the doctor being ready to help. Or you can increase the max latency, though this can be hard based on the nature of the work. For example, if you're better able to stabilize patients, you may be able to wait longer until the doctor is on-site. In this case, the "install Debian" job has really high variance. 99% of the time it's a click, 1% of the time, you've gotta know some Linux internals. The max latency is low because you don't want the IceCube array non-operational for long. And the warm-up time is really high — fly someone out to the South Pole. So the reasonable response is to throw money at idle capacity and send someone out pre-emptively.

robocat 8y ago

I met some scientists (at the pub) from NASA the other day - they had been on a plane going repeatably from sea level to stratosphere all the way from from Hawaii to Chirstchurch,NZ to sample the atmosphere. They had designed and built their own instruments, but they were on the plane so they could fix any problems, although the idea was to have no problems!