6 ms·

mkdir /var/lib/postgresql/data/old mv -v /var/lib/postgresql/data/* /var/lib/postgresql/data/old/ mkdir /var/lib/postgresql/data/new You should create the

mkdir /var/lib/postgresql/data/old mv -v /var/lib/postgresql/data/* /var/lib/postgresql/data/old/ mkdir /var/lib/postgresql/data/new You should create the both dirs first, then check if they do really exist and only then move the files. Shit happens and it's better be safe than sorry. Also I would replace ../old and ../new with $OLD and $NEW to have a bit less clutter in the next block with the explicit upgrade calls, ie: /usr/local/bin/pg_upgrade --link -d $OLD -D $NEW -b /usr/local-pg9.5/bin -B /usr/local/bin Overall it's nice idea, just need some more safety checking before starting the process.

justinclift 3y ago
Cool, good thinking, thanks heaps. I'll implement that in the morning. :)
- javajosh 3y ago
  This looks like this will use twice the disk.
  - justinclift 3y ago
    Nah. The concept of verifying that the mkdir's worked is sound and also easy to do. They just need doing individually in the script, as doing both in the same spot would muck up the wildcard a bit later on. Using variable names better ($OLD, $NEW) is a good idea too. Should cut down any potential typo risk as well. :)
    - justsomehnguy 3y ago
      If you prefer to work with the dirs under pgdata (well it makes sense...) you can just make the list of files (maybe even write it to the file, maybe even write a batch file which would move them) and use it for moving the data from pgdata to old. This saves an unnecessary move. Add: are sure about "${NEW}"/* in this? 444 mv -v "${NEW}"/* "${PGDATA}"
      - justinclift 3y ago
        Yeah, this syntax looks a bit unwieldy: "${NEW}"/* But it's specifically to do wildcard expansion of the quoted string, and the shell interpreter is happy with it. I'm open to suggestions for improvements though. :) --- This is confusing to me: ... you can just make the list of files (maybe even write it to the file, maybe even write a batch file which would move them) and use it for moving the data from pgdata to old. This saves an unnecessary move. I'm not understanding what you're meaning here. I understand the "make a list of files" bit, but I'm not grokking why doing that is an improvement, and I'm not seeing where there's an unnecessary move that could be eliminated? The pg_upgrade process is pretty much: 1. Initialise a fresh data directory using the new PostgreSQL version 2. Run pg_upgrade, pointing at both the old and new data directories 3. Start the database using the new data directory For the "automatic upgrade container" purposes, we need to do everything under the single mount point so the "--link" option to pg_upgrade is effective and uses hard links. Thus the "move things into an 'old' directory" first, then the "move the converted 'new' data into place" bit afterwards.
        justsomehnguy 3y ago
        > I'm not understanding what you're meaning here. If this was PowerShell then I would just get the list of files in $PGDATA, create folders and then move the files in the list, ie $files = gci $PGDATA try { New-item "$PGDATA/old" -erroraction stop New-item "$PGDATA/new" -erroraction stop } catch { throw "Failed to create the necessary dirs" } try { $files | Move-Item -Destination "$PGDATA/old" -erroraction stop } catch { # throw "Failed to move pg_data, your databases are now borked, good luck" gci "$PGDATA/new" | move-item $PGDATA } It's way more streamlined and if creating the dirs would fail (especially `new`) then it would fail before moving the data. And you don't need to `set +e` in this part. I tried to replicate this in Linux and... it's a mess. `find` includes the directory in the list, `ls -1` does the thing but bash stores it's output as a single string, redirecting it to the file get this file included in the list... I even tried xargs, but quickly abandoned the idea. Though if you can create the redirected output file in some other place than $PGDATA (`/tmp` perharps?) then `ls -1` trick would work.
- justinclift 3y ago
  Implemented in https://github.com/justinclift/docker-pgautoupgrade/commit/098caee5717044d91306fcff8282bf5629a0fdbc https://github.com/justinclift/docker-pgautoupgrade/commit/0.... I think I got everything right, and it passes the test harness, but please give it a look over if you have a few minutes. :)

justinclift 3y ago

Cool, good thinking, thanks heaps. I'll implement that in the morning. :)

javajosh 3y ago
This looks like this will use twice the disk.
- justinclift 3y ago
  Nah. The concept of verifying that the mkdir's worked is sound and also easy to do. They just need doing individually in the script, as doing both in the same spot would muck up the wildcard a bit later on. Using variable names better ($OLD, $NEW) is a good idea too. Should cut down any potential typo risk as well. :)
  - justsomehnguy 3y ago
    If you prefer to work with the dirs under pgdata (well it makes sense...) you can just make the list of files (maybe even write it to the file, maybe even write a batch file which would move them) and use it for moving the data from pgdata to old. This saves an unnecessary move. Add: are sure about "${NEW}"/* in this? 444 mv -v "${NEW}"/* "${PGDATA}"
    - justinclift 3y ago
      Yeah, this syntax looks a bit unwieldy: "${NEW}"/* But it's specifically to do wildcard expansion of the quoted string, and the shell interpreter is happy with it. I'm open to suggestions for improvements though. :) --- This is confusing to me: ... you can just make the list of files (maybe even write it to the file, maybe even write a batch file which would move them) and use it for moving the data from pgdata to old. This saves an unnecessary move. I'm not understanding what you're meaning here. I understand the "make a list of files" bit, but I'm not grokking why doing that is an improvement, and I'm not seeing where there's an unnecessary move that could be eliminated? The pg_upgrade process is pretty much: 1. Initialise a fresh data directory using the new PostgreSQL version 2. Run pg_upgrade, pointing at both the old and new data directories 3. Start the database using the new data directory For the "automatic upgrade container" purposes, we need to do everything under the single mount point so the "--link" option to pg_upgrade is effective and uses hard links. Thus the "move things into an 'old' directory" first, then the "move the converted 'new' data into place" bit afterwards.
      - justsomehnguy 3y ago
        > I'm not understanding what you're meaning here. If this was PowerShell then I would just get the list of files in $PGDATA, create folders and then move the files in the list, ie $files = gci $PGDATA try { New-item "$PGDATA/old" -erroraction stop New-item "$PGDATA/new" -erroraction stop } catch { throw "Failed to create the necessary dirs" } try { $files | Move-Item -Destination "$PGDATA/old" -erroraction stop } catch { # throw "Failed to move pg_data, your databases are now borked, good luck" gci "$PGDATA/new" | move-item $PGDATA } It's way more streamlined and if creating the dirs would fail (especially `new`) then it would fail before moving the data. And you don't need to `set +e` in this part. I tried to replicate this in Linux and... it's a mess. `find` includes the directory in the list, `ls -1` does the thing but bash stores it's output as a single string, redirecting it to the file get this file included in the list... I even tried xargs, but quickly abandoned the idea. Though if you can create the redirected output file in some other place than $PGDATA (`/tmp` perharps?) then `ls -1` trick would work.
justinclift 3y ago
Implemented in https://github.com/justinclift/docker-pgautoupgrade/commit/098caee5717044d91306fcff8282bf5629a0fdbc https://github.com/justinclift/docker-pgautoupgrade/commit/0.... I think I got everything right, and it passes the test harness, but please give it a look over if you have a few minutes. :)