9 ms·

I really appreciate the announcement from Hewlett Packard, which is very apologetic: https://www.iimc.kyoto-u.ac.jp/services/comp/pdf/file_loss_insident_2021122

by rvnx 5y ago

I really appreciate the announcement from Hewlett Packard, which is very apologetic: https://www.iimc.kyoto-u.ac.jp/services/comp/pdf/file_loss_insident_20211228.pdf https://www.iimc.kyoto-u.ac.jp/services/comp/pdf/file_loss_i... They do not try to blame it on complex systems or other factors. Users lost 1 day and 1/2 of recent work (which doesn't seem to be that bad). About file loss in Luster file system in your supercomputer system, we are 100% responsible. We deeply apologize for causing a great deal of inconvenience due to the serious failure of the file loss. We would like to report the background of the file disappearance, its root cause and future countermeasures as follows: We believe that this file loss is 100% our responsibility. We will offer compensation for users who have lost files. [...] Impact: -- Target file system: /LARGE0 Deleted files: December 14, 2021 17:32 to December 16, 2021 12:43 Files that were supposed to be deleted: Files that had not been updated since 17:32 on December 3, 2021 [...] Cause: -- The backup script uses the find command to delete log files that are older than 10 days. A variable name is passed to the delete process of the find command. A new improved version of the script was applied on the system. However, during deployment, there was a lack of consideration as the periodical script was not disabled. The modified shell script was reloaded from the middle. As a result, the find command containing undefined variables was executed and deleted the files. [...] Further measures: -- In the future, the programs to be applied to the system will be fully verified and applied. We will examine the extent of the impact and make improvements so that similar problems do not occur. In addition, we will re-educate the engineers in charge of human error and risk prediction / prevention to prevent recurrence. We will thoroughly implement the measures.

deleted 5y ago
[deleted]
nikanj 5y ago
Every shell script should start with set -e and set -u
- thadk 5y ago
  https://www.gnu.org/software/bash/manual/bash.html#The-Set-Builtin https://www.gnu.org/software/bash/manual/bash.html#The-Set-B...
- sillysaurusx 5y ago
  e doesn’t work for (subshell | commands), and u is inconvenient when appending to PATHs. Every tool has its place, and dogma is often unhelpful.
  - CyberShadow 5y ago
    > e doesn’t work for (subshell | commands) That's not an argument against enabling it. In bash, -o pipefail addresses this. > and u is inconvenient when appending to PATHs PATH should always be set. Try: env -i sh -c 'echo $PATH' If you're prioritizing convenience over correctness, prepare to face the consequences. > Every tool has its place, and dogma is often unhelpful. Visual Basic's "ON ERROR RESUME NEXT" perhaps also had its place. That doesn't mean that using it is good advice. If anything, I would consider the often cited wooledge etc. advice of not using -e/-u as dogma. Case in point: no one lost 77TB of data because they should not have used -e/-u.
    - sillysaurusx 5y ago
      I said PATHs, not PATH. There are at least four I use on a regular basis. Super not interested in a pedantic debate. It’s easy to armchair analyze. I found flaws in 55 codebases at Matasao, and yours is no exception. e makes it super annoying to pass a variable number of args to a script, since shift will fail and cause an exit. I do usually turn it on after, but you seem like the type to fail a code review if a script doesn’t start with it. I don’t think that’s a productive attitude.
      - CyberShadow 5y ago
        That's quite a number of bad-faith assumptions in your comment, which are also incidentally wrong. > e makes it super annoying I rest my case?
capableweb 5y ago
Interesting, seems the shell script was executed from the cron job just as it was being replaced on the server itself?
thaumasiotes 5y ago
> However, during deployment, there was a lack of consideration as the cronjob was not disabled. I'm intrigued to see that the report you link (which is in Japanese) mentions `find` and `bash` by those names, but doesn't contain the word `cron`. How does the report refer to the idea of a "cronjob"? Why is it different?
- rvnx 5y ago
  I took: "bash は、シェルスクリプトの実行中に適時シェ", which means it's either cronjob or sleep with a loop ( https://iww.hateblo.jp/entry/20211229/file_lost_insident https://iww.hateblo.jp/entry/20211229/file_lost_insident )
  - kragen 5y ago
    "シェルスクリプト" is "shell script", but my Japanese is too poor to understand 実行中 or 適時.
    - rvnx 5y ago
      I guess the most correct context is that the script was running "periodically". https://zenn.dev/mattn/articles/5af86b61004bdc https://zenn.dev/mattn/articles/5af86b61004bdc https://iww.hateblo.jp/entry/20211229/file_lost_insident https://iww.hateblo.jp/entry/20211229/file_lost_insident
    - numpad0 5y ago
      実行中: while executing 適時: appropriate times, or as needed
      - kragen 5y ago
        Thank you very much!
  - ramchip 5y ago
    I read this sentence as “Bash reads the shell script just-in-time while executing it”, with no context as to why it was running (cron, loop, by hand…)
doctor_eval 5y ago
So this is something I’ve never understood. If you modify a shell script while it’s running, the shell executes the modified file. This normally but not always causes the script to fail. Now I’ve known about this behaviour for a very long time and it always seemed very broken to me. It’s not how binaries work (at least not when I was doing that kind of thing). So I guess bash or whatever does an mmap of the script it’s running, which is presumably why modifications to the script are visible immediately. But if a new file was installed eg using cp/tar/unzip, I’m surprised that this didn’t just unlink the old script and create a new one - which would create a new inode and therefore make the operation atomic, right? And this (I assume) is why a recompiled binary doesn’t have the same problem (because the old binary is first unlinked). So, how could this (IMO) bad behaviour be fixed? Presumably mmap is used for efficiency, but isn’t it possible to mark a file as in use so it’s cant be modified? I’ve certainly seen on some old Unices that you can’t overwrite a running binary. Why can’t we do the same with shell scripts? Honestly, while it’s great that HP is accepting responsibility, and we know that this happens, the behaviour seems both arbitrary and unnecessary to me. Is it fixable?
- bluedino 5y ago
  It's nice to see the same mistakes that people have been making for as long as I've been alive, on small and large systems all over the world, still happen on projects with professional teams from HPE or IBM that cost hundreds of millions of dollars.
- helsinkiandrew 5y ago
  > If you modify a shell script while it’s running, the shell executes the modified file That is dependent on the OS. In this case wasn't the shell script just executed fresh from a cronjob? I remember on Digital Unix - on an Alpha so this was a few years ago - that you could change a c program (a loop that printed something then slept, for example), recompile and it would change the running binary.
  - doctor_eval 5y ago
    > wasn't the shell script just executed fresh from a cronjob? The description said that the script changed while it was running, so certain newly introduced environment variables didn’t have values and this triggered the issue. My reading was that this was just a terrible coincidence - the cron job must have started just before the upgrade. Regarding changing a C program, now you mention it I think that the behaviour you describe might also have happened on DG/UX, after an upgrade. IIRC it used to use ETXTBSY and after an upgrade it would just overwrite. Not really behaviour that you want (or expect) tho.
KaiserPro 5y ago
Ahhh the joy of lustre and the accidental cronjob. about 15 years ago I experienced the same thing. An updater script based on rsync was trying to keep one nfs machine image in sync with another. However for what ever reason, the script accidentally tries to sync the entire nfs root directory with its own, deleting everything show by show in reverse alphabetical order. At the time Lustre didn't really have any good monitoring tools for showing you who was doing what, so they had to wait till they hit a normal NFS server before they could figure out and stop what was deleting everything. Needless to say, a lot of the backups may have been failing.
- unixhero 5y ago
  For this reason I actually use simpler tools than rsync.
- throwaway75787 5y ago
  rsync has a number of safety and boundary options, not to mention --dry-run. Options, as in I also found out about them the hard way.
agumonkey 5y ago
What a strangely simple error
olliej 5y ago
1.5 days isn't too bad. If it were me my primary concern would be losing bash history :D
s5300 5y ago
Huh. I may be remembering incorrectly, but I recall having somebody somewhat entrenched in related business tell me that HP has been going downhill from an industry perspective roughly two years ago… Nice to see them completely own up to the mistake right away. I wonder who made the final call on doing so, companies admitting fault so transparently & immediately offering recourse seems pretty damn rare anymore. Without the intent of sounding xenophobic, I wonder if it’s because it’s HP Japan where reputation is much more culturally important. US MBA’s admitting fault… haha…
Closi 5y ago
Agreed - no corporate-speak, sounds like it was written by an actual human.
- Hamuko 5y ago
  It sounds a lot like Japanese corpo-speak.
  - Aeolun 5y ago
    Which is very formulaic, but also almost definitely written by a human :D
dustintrex 5y ago
Japanese companies structure apologies very differently from US ones, because the legal consequences are very different. In the US, an apology is considered an admission of responsibility and is often the starting point of legal action against the culprit, while in Japan, a sufficiently sincere* apology may well defuse the situation entirely. * 真 makoto, a word often glossed as "sincere" but not identical in meaning: it's about the amount of effort you're willing to take on, not how "honestly" you feel something Also, the culprit here is not HP proper but their consulting/SI wing HP Enterprise, which has a, uhh, less than stellar reputation for competence.
- deleted 5y ago
  [deleted]
- deleted 5y ago
  [deleted]
- cafard 5y ago
  I was deeply impressed some--30?--years ago when there was a minor scandal in sumo wrestling: some kid, who had been advanced too quickly did a few stupid things, as I recall the sort of thing you can find in American sports pages every week. The heads of the sumo wrestling association acknowledged that they had contributed to the situation, and docked their own pay. Do you think Roger Goodell is going to do that?
- p_l 5y ago
  HP and HPE are now two separate companies, split from post-Fiorina HP. HP does consumer grade stuff only, while HPE does the enterprise side (not just consulting, in fact non trivial portion of HPE consulting arm was spun off and merged into DXC)
  - exikyut 5y ago
    I think "sell the HP-35 for 3.14 × cost of materials" cool-HP became Agilent, right? If I wanted to follow the trail of awesomeness what forest should I be sticking my nose to the ground in? :)
codedokode 5y ago
> As a result, the find command containing undefined variables was executed And this is why shell should not execute commands with "undefined" variables and give an error instead.
taubek 5y ago
77TB in one and half day? Impressive. The style of apology is very nice. It is not extensive as some technical post mortem analysis that I've read, but all of the important things are here.
Solstinox 5y ago
I think I found the problem: “A new improved version of the script was applied on the system.”
jacquesm 5y ago
The sense of honor and responsibility shining through is refreshing.
AdamJacobMuller 5y ago
> The modified shell script was reloaded from the middle. This is an incredible edge case. I'm amazed they hit this issue and just as amazed that they correctly identified that issue and reported on it. This response is great, it's the exact opposite of the wishy-washy mealy-mouthed response to the lastpass security incident.
martin_vejmelka 5y ago
Just pointing out that those are most likely just the days the files were saved. There could still be some unlucky souls that ran computations for several days/weeks that happened to terminate on those days (and store the results). Those people could lose significantly more than a day and a half. On the flip side, HP jobs tend to be frequently checkpointed unless the storage cost is prohibitive for the type of job.
- freemint 5y ago
  Is checkpointing really that common anymore. When I talked to my local HPC admins they gave me the impression that nobody does it anymore

by rvnx 5y ago

deleted 5y ago

[deleted]

nikanj 5y ago

Every shell script should start with set -e and set -u

thadk 5y ago
https://www.gnu.org/software/bash/manual/bash.html#The-Set-Builtin https://www.gnu.org/software/bash/manual/bash.html#The-Set-B...
sillysaurusx 5y ago
e doesn’t work for (subshell | commands), and u is inconvenient when appending to PATHs. Every tool has its place, and dogma is often unhelpful.
- CyberShadow 5y ago
  > e doesn’t work for (subshell | commands) That's not an argument against enabling it. In bash, -o pipefail addresses this. > and u is inconvenient when appending to PATHs PATH should always be set. Try: env -i sh -c 'echo $PATH' If you're prioritizing convenience over correctness, prepare to face the consequences. > Every tool has its place, and dogma is often unhelpful. Visual Basic's "ON ERROR RESUME NEXT" perhaps also had its place. That doesn't mean that using it is good advice. If anything, I would consider the often cited wooledge etc. advice of not using -e/-u as dogma. Case in point: no one lost 77TB of data because they should not have used -e/-u.
  - sillysaurusx 5y ago
    I said PATHs, not PATH. There are at least four I use on a regular basis. Super not interested in a pedantic debate. It’s easy to armchair analyze. I found flaws in 55 codebases at Matasao, and yours is no exception. e makes it super annoying to pass a variable number of args to a script, since shift will fail and cause an exit. I do usually turn it on after, but you seem like the type to fail a code review if a script doesn’t start with it. I don’t think that’s a productive attitude.
    - CyberShadow 5y ago
      That's quite a number of bad-faith assumptions in your comment, which are also incidentally wrong. > e makes it super annoying I rest my case?

capableweb 5y ago

Interesting, seems the shell script was executed from the cron job just as it was being replaced on the server itself?

thaumasiotes 5y ago

> However, during deployment, there was a lack of consideration as the cronjob was not disabled. I'm intrigued to see that the report you link (which is in Japanese) mentions `find` and `bash` by those names, but doesn't contain the word `cron`. How does the report refer to the idea of a "cronjob"? Why is it different?

rvnx 5y ago
I took: "bash は、シェルスクリプトの実行中に適時シェ", which means it's either cronjob or sleep with a loop ( https://iww.hateblo.jp/entry/20211229/file_lost_insident https://iww.hateblo.jp/entry/20211229/file_lost_insident )
- kragen 5y ago
  "シェルスクリプト" is "shell script", but my Japanese is too poor to understand 実行中 or 適時.
  - rvnx 5y ago
    I guess the most correct context is that the script was running "periodically". https://zenn.dev/mattn/articles/5af86b61004bdc https://zenn.dev/mattn/articles/5af86b61004bdc https://iww.hateblo.jp/entry/20211229/file_lost_insident https://iww.hateblo.jp/entry/20211229/file_lost_insident
  - numpad0 5y ago
    実行中: while executing 適時: appropriate times, or as needed
    - kragen 5y ago
      Thank you very much!
- ramchip 5y ago
  I read this sentence as “Bash reads the shell script just-in-time while executing it”, with no context as to why it was running (cron, loop, by hand…)

doctor_eval 5y ago

So this is something I’ve never understood. If you modify a shell script while it’s running, the shell executes the modified file. This normally but not always causes the script to fail. Now I’ve known about this behaviour for a very long time and it always seemed very broken to me. It’s not how binaries work (at least not when I was doing that kind of thing). So I guess bash or whatever does an mmap of the script it’s running, which is presumably why modifications to the script are visible immediately. But if a new file was installed eg using cp/tar/unzip, I’m surprised that this didn’t just unlink the old script and create a new one - which would create a new inode and therefore make the operation atomic, right? And this (I assume) is why a recompiled binary doesn’t have the same problem (because the old binary is first unlinked). So, how could this (IMO) bad behaviour be fixed? Presumably mmap is used for efficiency, but isn’t it possible to mark a file as in use so it’s cant be modified? I’ve certainly seen on some old Unices that you can’t overwrite a running binary. Why can’t we do the same with shell scripts? Honestly, while it’s great that HP is accepting responsibility, and we know that this happens, the behaviour seems both arbitrary and unnecessary to me. Is it fixable?

bluedino 5y ago
It's nice to see the same mistakes that people have been making for as long as I've been alive, on small and large systems all over the world, still happen on projects with professional teams from HPE or IBM that cost hundreds of millions of dollars.
helsinkiandrew 5y ago
> If you modify a shell script while it’s running, the shell executes the modified file That is dependent on the OS. In this case wasn't the shell script just executed fresh from a cronjob? I remember on Digital Unix - on an Alpha so this was a few years ago - that you could change a c program (a loop that printed something then slept, for example), recompile and it would change the running binary.
- doctor_eval 5y ago
  > wasn't the shell script just executed fresh from a cronjob? The description said that the script changed while it was running, so certain newly introduced environment variables didn’t have values and this triggered the issue. My reading was that this was just a terrible coincidence - the cron job must have started just before the upgrade. Regarding changing a C program, now you mention it I think that the behaviour you describe might also have happened on DG/UX, after an upgrade. IIRC it used to use ETXTBSY and after an upgrade it would just overwrite. Not really behaviour that you want (or expect) tho.

KaiserPro 5y ago

Ahhh the joy of lustre and the accidental cronjob. about 15 years ago I experienced the same thing. An updater script based on rsync was trying to keep one nfs machine image in sync with another. However for what ever reason, the script accidentally tries to sync the entire nfs root directory with its own, deleting everything show by show in reverse alphabetical order. At the time Lustre didn't really have any good monitoring tools for showing you who was doing what, so they had to wait till they hit a normal NFS server before they could figure out and stop what was deleting everything. Needless to say, a lot of the backups may have been failing.

unixhero 5y ago
For this reason I actually use simpler tools than rsync.
throwaway75787 5y ago
rsync has a number of safety and boundary options, not to mention --dry-run. Options, as in I also found out about them the hard way.

agumonkey 5y ago

What a strangely simple error

olliej 5y ago

1.5 days isn't too bad. If it were me my primary concern would be losing bash history :D

s5300 5y ago

Huh. I may be remembering incorrectly, but I recall having somebody somewhat entrenched in related business tell me that HP has been going downhill from an industry perspective roughly two years ago… Nice to see them completely own up to the mistake right away. I wonder who made the final call on doing so, companies admitting fault so transparently & immediately offering recourse seems pretty damn rare anymore. Without the intent of sounding xenophobic, I wonder if it’s because it’s HP Japan where reputation is much more culturally important. US MBA’s admitting fault… haha…

Closi 5y ago

Agreed - no corporate-speak, sounds like it was written by an actual human.

Hamuko 5y ago
It sounds a lot like Japanese corpo-speak.
- Aeolun 5y ago
  Which is very formulaic, but also almost definitely written by a human :D

dustintrex 5y ago

Japanese companies structure apologies very differently from US ones, because the legal consequences are very different. In the US, an apology is considered an admission of responsibility and is often the starting point of legal action against the culprit, while in Japan, a sufficiently sincere* apology may well defuse the situation entirely. * 真 makoto, a word often glossed as "sincere" but not identical in meaning: it's about the amount of effort you're willing to take on, not how "honestly" you feel something Also, the culprit here is not HP proper but their consulting/SI wing HP Enterprise, which has a, uhh, less than stellar reputation for competence.

deleted 5y ago
[deleted]
deleted 5y ago
[deleted]
cafard 5y ago
I was deeply impressed some--30?--years ago when there was a minor scandal in sumo wrestling: some kid, who had been advanced too quickly did a few stupid things, as I recall the sort of thing you can find in American sports pages every week. The heads of the sumo wrestling association acknowledged that they had contributed to the situation, and docked their own pay. Do you think Roger Goodell is going to do that?
p_l 5y ago
HP and HPE are now two separate companies, split from post-Fiorina HP. HP does consumer grade stuff only, while HPE does the enterprise side (not just consulting, in fact non trivial portion of HPE consulting arm was spun off and merged into DXC)
- exikyut 5y ago
  I think "sell the HP-35 for 3.14 × cost of materials" cool-HP became Agilent, right? If I wanted to follow the trail of awesomeness what forest should I be sticking my nose to the ground in? :)

codedokode 5y ago

> As a result, the find command containing undefined variables was executed And this is why shell should not execute commands with "undefined" variables and give an error instead.

taubek 5y ago

77TB in one and half day? Impressive. The style of apology is very nice. It is not extensive as some technical post mortem analysis that I've read, but all of the important things are here.

Solstinox 5y ago

I think I found the problem: “A new improved version of the script was applied on the system.”

jacquesm 5y ago

The sense of honor and responsibility shining through is refreshing.

AdamJacobMuller 5y ago

> The modified shell script was reloaded from the middle. This is an incredible edge case. I'm amazed they hit this issue and just as amazed that they correctly identified that issue and reported on it. This response is great, it's the exact opposite of the wishy-washy mealy-mouthed response to the lastpass security incident.

martin_vejmelka 5y ago

Just pointing out that those are most likely just the days the files were saved. There could still be some unlucky souls that ran computations for several days/weeks that happened to terminate on those days (and store the results). Those people could lose significantly more than a day and a half. On the flip side, HP jobs tend to be frequently checkpointed unless the storage cost is prohibitive for the type of job.

freemint 5y ago
Is checkpointing really that common anymore. When I talked to my local HPC admins they gave me the impression that nobody does it anymore