16 ms·

Japan HP accidentally deleted 77TB data in Kyoto U. supercomputing system

260 points by rguiscard 5y ago

nh2 5y ago
However, during deployment, there was a lack of consideration as the periodical script was not disabled. The modified shell script was reloaded from the middle. In my opinion, this is the wrong takeaway, and an important lesson was not learned. It's not an operator "lack of consideration". The lesson should be "when dealing with important data, do not use outrageously bad programming languages that allow run-time code rewriting, and that continue to execute even in the presence of undefined variables". If you use shell scripting, this is bound to happen, and will happen again. "We'll use Python or anything else instead of shell" would fundamentally remove the possibility of this category of failure.
- 0xbadcafebee 5y ago
  The problem was not that they used shell scripts. The problem was that the people writing the shell scripts were just bad programmers. If you hire a bad programmer to write them in Python, they'll still have tons of bugs. The shell scripts I write have fewer bugs than the Python code I see other teams churn out. But that's because I know what I'm doing. Don't hire people who don't know what they're doing.
- toast0 5y ago
  > outrageously bad programming languages that allow run-time code rewriting Almost all languages allow run-time code rewriting. Some of them just make it easier than others, and some of them make it a very useful feature. If you're very careful, updating a bash script while you're running it can be useful, but most often it's a mistake; in Erlang, hot loading is usually intentional and often useful. Most other languages don't make it easy, so you'll probably only do it if it's useful.
vardump 5y ago
That's a lot of floppy disks!
- Too 5y ago
  Reference: https://news.ycombinator.com/item?id=29738298 https://news.ycombinator.com/item?id=29738298
0xbadcafebee 5y ago
Not really surprising. HPE has provided bottom-of-the-barrel support for decades.
marcan_42 5y ago
Everyone is mentioning error control for shell scripts or "don't use shell scripts", but neither of those are the solution to this problem. The solution to this problem is correctly implementing atomic deployment, which is important for any system using any programming language. What I like to do is have two directories I ping pong between when deploying, and a `cur` symlink that points to the current version. The symlink is atomically replaced (new symlink and rename it over) whenever the deploy process completes. Any software/scripts using that tree will be written to first chdir() in, which will resolve the symlink at that time, and thus won't be affected by the deploy (at least as long as you don't do it twice in a row; if that is a concern due to long running processes, you could use timestamped directories instead and a garbage collection process that cleans stuff up once it is certain there are no users left).
- db65edfc7996 5y ago
  The original blue-green deployment strategy. I have done a similar thing as well.
ohcomments 5y ago
Auch! That's a nice run for the backups eh?
chmaynard 5y ago
Clearly, the only honorable thing for the CEO of HPE to do is to ... er, blame Sunny Balwani!
rguiscard 5y ago
In the process of functional modification of the backup program by Hewlett-Packard Japan, the supplier of the supercomputer system, there was a problem in the unintentional modification of the program and its application procedure, which caused a malfunction in the process of deleting the files under the /LARGE0 directory instead of deleting the backup log files that are no longer needed. Translated with www.DeepL.com/Translator (free version)
deepsun 5y ago
Yet another bug due to using command-line interface (which is designed for humans not programs) by programs.
ketanip 5y ago
Did they had any other separated backups ?
rvnx 5y ago
I really appreciate the announcement from Hewlett Packard, which is very apologetic: https://www.iimc.kyoto-u.ac.jp/services/comp/pdf/file_loss_insident_20211228.pdf https://www.iimc.kyoto-u.ac.jp/services/comp/pdf/file_loss_i... They do not try to blame it on complex systems or other factors. Users lost 1 day and 1/2 of recent work (which doesn't seem to be that bad). About file loss in Luster file system in your supercomputer system, we are 100% responsible. We deeply apologize for causing a great deal of inconvenience due to the serious failure of the file loss. We would like to report the background of the file disappearance, its root cause and future countermeasures as follows: We believe that this file loss is 100% our responsibility. We will offer compensation for users who have lost files. [...] Impact: -- Target file system: /LARGE0 Deleted files: December 14, 2021 17:32 to December 16, 2021 12:43 Files that were supposed to be deleted: Files that had not been updated since 17:32 on December 3, 2021 [...] Cause: -- The backup script uses the find command to delete log files that are older than 10 days. A variable name is passed to the delete process of the find command. A new improved version of the script was applied on the system. However, during deployment, there was a lack of consideration as the periodical script was not disabled. The modified shell script was reloaded from the middle. As a result, the find command containing undefined variables was executed and deleted the files. [...] Further measures: -- In the future, the programs to be applied to the system will be fully verified and applied. We will examine the extent of the impact and make improvements so that similar problems do not occur. In addition, we will re-educate the engineers in charge of human error and risk prediction / prevention to prevent recurrence. We will thoroughly implement the measures.
- deleted 5y ago
  [deleted]
- nikanj 5y ago
  Every shell script should start with set -e and set -u
  - thadk 5y ago
    https://www.gnu.org/software/bash/manual/bash.html#The-Set-Builtin https://www.gnu.org/software/bash/manual/bash.html#The-Set-B...
Proven 5y ago
j1elo 5y ago
I guess this is as good of a time as any other to remind people to use the "unofficial" Bash strict mode: https://gist.github.com/robin-a-meade/58d60124b88b60816e8349d1e3938615 https://gist.github.com/robin-a-meade/58d60124b88b60816e8349... [^1] And always, always, use ShellCheck (https://www.shellcheck.net/ https://www.shellcheck.net/) to catch most pitfalls and common mistakes on this powerful but dangerous language that is shell scripting. [^1]: I think this gist is better than the original article in which it is based, because the article also suggested changing the IFS variable, which is not that good of an advice, so sadly the original text becomes a bad recommendation!
- thaumasiotes 5y ago
  > I guess this is as good of a time as any other to remind people to use the "unofficial" Bash strict mode Not really; the report doesn't mention any error in the script.
  - leoh 5y ago
    There is a reading which suggests that an environment variable being unset caused an overabundance of files being deleted. `set -u` causes the script to exit if any variables are unset.
- xvilka 5y ago
  And don't use shell for writing complex scripts, there are better automation tools and languages.
  - tryauuum 5y ago
    I'd say that as long as bash script fits on a single screen you won't get any benefit from switching to some other tool or language
  - rrll22 5y ago
    Such as?
mayurbirle 5y ago
nani!
pettycashstash2 5y ago
Looks like 10 of 14 groups were restored from backup.
quelsolaar 5y ago
Who brought tres commas?
- alekun 5y ago
  just in case someone didn’t see this masterpiece https://youtu.be/vvDK8tMyCic https://youtu.be/vvDK8tMyCic
booleandilemma 5y ago
It’s amazing how human errors scale with technology. Just imagine, one day we’ll be making mistakes at the Type III civilization level! :)
karlerss 5y ago
When communicating non-critical data-loss to teammates, I like to do it with this haiku: Three things are certain: Death, taxes, and lost data. Guess which has occurred. From https://www.gnu.org/fun/jokes/error-haiku.en.html https://www.gnu.org/fun/jokes/error-haiku.en.html
Jimmc414 5y ago
>the find command containing undefined variables was executed and deleted the files Just a note that "set -u" at the beginning of a bash script will cause it to throw an error for undefined variables. warning that of course this should be tested as it will also cause [[ $var ]] to fail. If that's the case [ -z "${VAR:-}" ] && echo "VAR is not set or is empty" || echo "VAR is set to $VAR" will help test that condition
pharmakom 5y ago
I have switched to F# for scripting tasks and have found F# scripts are (usually) either correct on the first try or fail at the type-checking stage. I would highly recommend it for anything near production.
gnufx 5y ago
Assuming this was a "scratch" HPC filesystem, as I'd guess, "scratch" is used advisedly -- users should be prepared to lose anything on it, not that it should happen with finger trouble. However, if I understand correctly from the comments, I'm surprised at the tools, and that the vendor was managing the filesystem. I'd expect to use https://github.com/cea-hpc/robinhood/wiki https://github.com/cea-hpc/robinhood/wiki with Lustre, though I thought I'd seen a Cray presentation about tools of their own.
fred_is_fred 5y ago
This is HPE - not HP. Servers, not Printers.
moonbug 5y ago
it's a lustre filesystem. the data would've been eaten eventually anyway.
- bayindirh 5y ago
  What would make you think of that?
  - moonbug 5y ago
    experience (bitter).
l33tman 5y ago
I've been a Linux coder and user forever, and I didn't know that bash "reloads" a script while running if the file is modified. Good to learn before I also delete a whole filesystem due to this! :)
- deleted 5y ago
  [deleted]
- tyingq 5y ago
  Is that what happened? I can't reproduce that by changing a bash script that's running a while [ 1 ] loop. Is it maybe that they were editing or copying the file and a cron job kicked off?
  - marcan_42 5y ago
    That's because it's a loop, so it's already read. Try appending a line to a running script instead.
    - tyingq 5y ago
      Ah, yep. This does work, and prints out both "one" and "two": printf "echo one\nsleep 3\n" > s1;(bash s1 &);sleep 1 && printf "echo two\n" >> s1 That's interesting. And changing the "sleep 1" to "sleep 4" make it only output "one".
- deleted 5y ago
  [deleted]
ananonymoususer 5y ago
The cause of this is a known behavior of Unix/Linux scripts, but unfortunately not everyone knows this. If you change a script while it is running, the shell that runs it will read (what it thinks is) the next line from the old script, but it will be reading at the expected position in the old script file, but from the new script file. So what it reads and executes will probably not be what you wanted.

260 points by rguiscard 5y ago

nh2 5y ago

However, during deployment, there was a lack of consideration as the periodical script was not disabled. The modified shell script was reloaded from the middle. In my opinion, this is the wrong takeaway, and an important lesson was not learned. It's not an operator "lack of consideration". The lesson should be "when dealing with important data, do not use outrageously bad programming languages that allow run-time code rewriting, and that continue to execute even in the presence of undefined variables". If you use shell scripting, this is bound to happen, and will happen again. "We'll use Python or anything else instead of shell" would fundamentally remove the possibility of this category of failure.

0xbadcafebee 5y ago
The problem was not that they used shell scripts. The problem was that the people writing the shell scripts were just bad programmers. If you hire a bad programmer to write them in Python, they'll still have tons of bugs. The shell scripts I write have fewer bugs than the Python code I see other teams churn out. But that's because I know what I'm doing. Don't hire people who don't know what they're doing.
toast0 5y ago
> outrageously bad programming languages that allow run-time code rewriting Almost all languages allow run-time code rewriting. Some of them just make it easier than others, and some of them make it a very useful feature. If you're very careful, updating a bash script while you're running it can be useful, but most often it's a mistake; in Erlang, hot loading is usually intentional and often useful. Most other languages don't make it easy, so you'll probably only do it if it's useful.

vardump 5y ago

That's a lot of floppy disks!

Too 5y ago
Reference: https://news.ycombinator.com/item?id=29738298 https://news.ycombinator.com/item?id=29738298

0xbadcafebee 5y ago

Not really surprising. HPE has provided bottom-of-the-barrel support for decades.

marcan_42 5y ago

Everyone is mentioning error control for shell scripts or "don't use shell scripts", but neither of those are the solution to this problem. The solution to this problem is correctly implementing atomic deployment, which is important for any system using any programming language. What I like to do is have two directories I ping pong between when deploying, and a `cur` symlink that points to the current version. The symlink is atomically replaced (new symlink and rename it over) whenever the deploy process completes. Any software/scripts using that tree will be written to first chdir() in, which will resolve the symlink at that time, and thus won't be affected by the deploy (at least as long as you don't do it twice in a row; if that is a concern due to long running processes, you could use timestamped directories instead and a garbage collection process that cleans stuff up once it is certain there are no users left).

db65edfc7996 5y ago
The original blue-green deployment strategy. I have done a similar thing as well.

ohcomments 5y ago

Auch! That's a nice run for the backups eh?

chmaynard 5y ago

Clearly, the only honorable thing for the CEO of HPE to do is to ... er, blame Sunny Balwani!

rguiscard 5y ago

In the process of functional modification of the backup program by Hewlett-Packard Japan, the supplier of the supercomputer system, there was a problem in the unintentional modification of the program and its application procedure, which caused a malfunction in the process of deleting the files under the /LARGE0 directory instead of deleting the backup log files that are no longer needed. Translated with www.DeepL.com/Translator (free version)

deepsun 5y ago

Yet another bug due to using command-line interface (which is designed for humans not programs) by programs.

ketanip 5y ago

Did they had any other separated backups ?

rvnx 5y ago

I really appreciate the announcement from Hewlett Packard, which is very apologetic: https://www.iimc.kyoto-u.ac.jp/services/comp/pdf/file_loss_insident_20211228.pdf https://www.iimc.kyoto-u.ac.jp/services/comp/pdf/file_loss_i... They do not try to blame it on complex systems or other factors. Users lost 1 day and 1/2 of recent work (which doesn't seem to be that bad). About file loss in Luster file system in your supercomputer system, we are 100% responsible. We deeply apologize for causing a great deal of inconvenience due to the serious failure of the file loss. We would like to report the background of the file disappearance, its root cause and future countermeasures as follows: We believe that this file loss is 100% our responsibility. We will offer compensation for users who have lost files. [...] Impact: -- Target file system: /LARGE0 Deleted files: December 14, 2021 17:32 to December 16, 2021 12:43 Files that were supposed to be deleted: Files that had not been updated since 17:32 on December 3, 2021 [...] Cause: -- The backup script uses the find command to delete log files that are older than 10 days. A variable name is passed to the delete process of the find command. A new improved version of the script was applied on the system. However, during deployment, there was a lack of consideration as the periodical script was not disabled. The modified shell script was reloaded from the middle. As a result, the find command containing undefined variables was executed and deleted the files. [...] Further measures: -- In the future, the programs to be applied to the system will be fully verified and applied. We will examine the extent of the impact and make improvements so that similar problems do not occur. In addition, we will re-educate the engineers in charge of human error and risk prediction / prevention to prevent recurrence. We will thoroughly implement the measures.

deleted 5y ago
[deleted]
nikanj 5y ago
Every shell script should start with set -e and set -u
- thadk 5y ago
  https://www.gnu.org/software/bash/manual/bash.html#The-Set-Builtin https://www.gnu.org/software/bash/manual/bash.html#The-Set-B...

Proven 5y ago

j1elo 5y ago

I guess this is as good of a time as any other to remind people to use the "unofficial" Bash strict mode: https://gist.github.com/robin-a-meade/58d60124b88b60816e8349d1e3938615 https://gist.github.com/robin-a-meade/58d60124b88b60816e8349... [^1] And always, always, use ShellCheck (https://www.shellcheck.net/ https://www.shellcheck.net/) to catch most pitfalls and common mistakes on this powerful but dangerous language that is shell scripting. [^1]: I think this gist is better than the original article in which it is based, because the article also suggested changing the IFS variable, which is not that good of an advice, so sadly the original text becomes a bad recommendation!

thaumasiotes 5y ago
> I guess this is as good of a time as any other to remind people to use the "unofficial" Bash strict mode Not really; the report doesn't mention any error in the script.
- leoh 5y ago
  There is a reading which suggests that an environment variable being unset caused an overabundance of files being deleted. `set -u` causes the script to exit if any variables are unset.
xvilka 5y ago
And don't use shell for writing complex scripts, there are better automation tools and languages.
- tryauuum 5y ago
  I'd say that as long as bash script fits on a single screen you won't get any benefit from switching to some other tool or language
- rrll22 5y ago
  Such as?

mayurbirle 5y ago

nani!

pettycashstash2 5y ago

Looks like 10 of 14 groups were restored from backup.

quelsolaar 5y ago

Who brought tres commas?

alekun 5y ago
just in case someone didn’t see this masterpiece https://youtu.be/vvDK8tMyCic https://youtu.be/vvDK8tMyCic

booleandilemma 5y ago

It’s amazing how human errors scale with technology. Just imagine, one day we’ll be making mistakes at the Type III civilization level! :)

karlerss 5y ago

When communicating non-critical data-loss to teammates, I like to do it with this haiku: Three things are certain: Death, taxes, and lost data. Guess which has occurred. From https://www.gnu.org/fun/jokes/error-haiku.en.html https://www.gnu.org/fun/jokes/error-haiku.en.html

Jimmc414 5y ago

>the find command containing undefined variables was executed and deleted the files Just a note that "set -u" at the beginning of a bash script will cause it to throw an error for undefined variables. warning that of course this should be tested as it will also cause [[ $var ]] to fail. If that's the case [ -z "${VAR:-}" ] && echo "VAR is not set or is empty" || echo "VAR is set to $VAR" will help test that condition

pharmakom 5y ago

I have switched to F# for scripting tasks and have found F# scripts are (usually) either correct on the first try or fail at the type-checking stage. I would highly recommend it for anything near production.

gnufx 5y ago

Assuming this was a "scratch" HPC filesystem, as I'd guess, "scratch" is used advisedly -- users should be prepared to lose anything on it, not that it should happen with finger trouble. However, if I understand correctly from the comments, I'm surprised at the tools, and that the vendor was managing the filesystem. I'd expect to use https://github.com/cea-hpc/robinhood/wiki https://github.com/cea-hpc/robinhood/wiki with Lustre, though I thought I'd seen a Cray presentation about tools of their own.

fred_is_fred 5y ago

This is HPE - not HP. Servers, not Printers.

moonbug 5y ago

it's a lustre filesystem. the data would've been eaten eventually anyway.

bayindirh 5y ago
What would make you think of that?
- moonbug 5y ago
  experience (bitter).

l33tman 5y ago

I've been a Linux coder and user forever, and I didn't know that bash "reloads" a script while running if the file is modified. Good to learn before I also delete a whole filesystem due to this! :)

deleted 5y ago
[deleted]
tyingq 5y ago
Is that what happened? I can't reproduce that by changing a bash script that's running a while [ 1 ] loop. Is it maybe that they were editing or copying the file and a cron job kicked off?
- marcan_42 5y ago
  That's because it's a loop, so it's already read. Try appending a line to a running script instead.
  - tyingq 5y ago
    Ah, yep. This does work, and prints out both "one" and "two": printf "echo one\nsleep 3\n" > s1;(bash s1 &);sleep 1 && printf "echo two\n" >> s1 That's interesting. And changing the "sleep 1" to "sleep 4" make it only output "one".
deleted 5y ago
[deleted]

ananonymoususer 5y ago

The cause of this is a known behavior of Unix/Linux scripts, but unfortunately not everyone knows this. If you change a script while it is running, the shell that runs it will read (what it thinks is) the next line from the old script, but it will be reading at the expected position in the old script file, but from the new script file. So what it reads and executes will probably not be what you wanted.