3 ms·

I made this, and have been using it in production and for local development for the past year or two. It's been immensely helpful to quickly save or easily rest

by akie 7y ago

I made this, and have been using it in production and for local development for the past year or two. It's been immensely helpful to quickly save or easily restore the database to a previously know 'good point'. I have also been using it to quickly load staging databases into development and vice versa. So far it's MySQL only, but the connection points for other databases such as PostgreSQL are in place. It's main goal is to simplify development - it's not intended as a replacement for writing database migrations or to work with production databases. I would be grateful for any feedback, preferably in the form of pull requests ;-)

westoque 7y ago
I like this pattern of being able to restore snapshots in your development environment. For the uninitiated, docker actually has all these functions. For example, if you create a container for postgres and mount a volume, you can create a snapshot of the volume for distribution for your team so they can mount/unmount anytime.
- oauea 7y ago
  > you can create a snapshot of the volume how? note that docker commit/export does not include volumes
  - remmargorp64 7y ago
    This functionality for Docker was proposed by NetApp (https://netapp.io/2017/06/22/snapshots-clones-docker-volume-paradigm/ https://netapp.io/2017/06/22/snapshots-clones-docker-volume-...) There was also a github issue with numerous people voting for it (https://github.com/moby/moby/issues/33782 https://github.com/moby/moby/issues/33782) But for whatever reason, the docker team apparently decided not to build it. We still don't have a properly easy way to snapshot or share docker volumes. So I came up with my own method. Basically, you have to mount the volume and then tar the parts of it that you want snapshotted (typically your data directories). You can do this with any linux machine image. Here's an example (where "foo" is the docker container you want to snapshot): mkdir -p docker-volume-snapshots && docker run --rm --volumes-from foo -v $(shell pwd)/docker-volume-snapshots:/docker-volume-snapshots debian:stretch-slim tar -czvf /docker-volume-snapshots/snapshot.tar.gz /var/lib/my_data_folder Then, to restore the snapshot, here's what you do: docker run --rm --volumes-from mysql8.0 -v $(shell pwd)/docker-volume-snapshots:/docker-volume-snapshots:delegated debian:stretch-slim /bin/sh -c "cd / && tar -xvf /docker-volume-snapshots/snapshot.tar.gz" (Edit: Don't forget to start up/initialize and then pause/stop the running "foo" docker container before you run these commands to snapshot the volume)
    - oauea 7y ago
      Looks interesting but I doubt that'll be a consistent snapshit, right? I guess you could combine it with `docker pause`
      - remmargorp64 7y ago
        oh, yeah, I forgot to mention that I stop the running docker container before I do this
    - ericschrock 7y ago
      FYI, in the Titan project (https://titan-data.io https://titan-data.io), we solve this problem by running a privileges storage container (with ZFS on Linux) that then is able to export volumes into other containers, giving us ZFS snapshots and clones under the hood. This was inspired by the approach taken by dotmesh (https://dotmesh.com/ https://dotmesh.com/) but is definitely tailor-made to our use case. If someone was sufficiently motivated, I'm sure you could build a different CLI on top of our server and use it simply to manage docker volumes without the Titan trimmings (e.g. push and pull).
- akhilcacharya 7y ago
  I was under the impression container file systems make running a DB in a container a bad idea? Or is it just in performance critical situations?
  - tehbeard 7y ago
    Doing so into the main container filesystem (which is usually an overlay/union stack) is probably bad for performance. But a bind mount (which is usually the recommended way to do it so that data survives the container lifespan) will avoid that and just leave you with whatever "problems" the host filesystem has re: database files.
silvester23 7y ago
Very interesting, my go-to workflow so far has been creating backup copies in phpMyAdmin but this seems much more convenient and reliable. I'll give it a try soon!
anonymousjunior 7y ago
Interesting utility. I can see a few really exciting use cases for this. If I had one wish it'd be native support for syncing generated snapshots to S3 or Google Storage for easy transfers between authenticated systems and users. Maybe I'll take a look at building out a PR for that over the weekend if that's okay
- akie 7y ago
  Yes, by all means please do. Simplicity of use is very important to me, so make sure it's generally applicable and easily understandable, but other than that - go for it! Awesome.

by akie 7y ago

westoque 7y ago

I like this pattern of being able to restore snapshots in your development environment. For the uninitiated, docker actually has all these functions. For example, if you create a container for postgres and mount a volume, you can create a snapshot of the volume for distribution for your team so they can mount/unmount anytime.

oauea 7y ago
> you can create a snapshot of the volume how? note that docker commit/export does not include volumes
- remmargorp64 7y ago
  This functionality for Docker was proposed by NetApp (https://netapp.io/2017/06/22/snapshots-clones-docker-volume-paradigm/ https://netapp.io/2017/06/22/snapshots-clones-docker-volume-...) There was also a github issue with numerous people voting for it (https://github.com/moby/moby/issues/33782 https://github.com/moby/moby/issues/33782) But for whatever reason, the docker team apparently decided not to build it. We still don't have a properly easy way to snapshot or share docker volumes. So I came up with my own method. Basically, you have to mount the volume and then tar the parts of it that you want snapshotted (typically your data directories). You can do this with any linux machine image. Here's an example (where "foo" is the docker container you want to snapshot): mkdir -p docker-volume-snapshots && docker run --rm --volumes-from foo -v $(shell pwd)/docker-volume-snapshots:/docker-volume-snapshots debian:stretch-slim tar -czvf /docker-volume-snapshots/snapshot.tar.gz /var/lib/my_data_folder Then, to restore the snapshot, here's what you do: docker run --rm --volumes-from mysql8.0 -v $(shell pwd)/docker-volume-snapshots:/docker-volume-snapshots:delegated debian:stretch-slim /bin/sh -c "cd / && tar -xvf /docker-volume-snapshots/snapshot.tar.gz" (Edit: Don't forget to start up/initialize and then pause/stop the running "foo" docker container before you run these commands to snapshot the volume)
  - oauea 7y ago
    Looks interesting but I doubt that'll be a consistent snapshit, right? I guess you could combine it with `docker pause`
    - remmargorp64 7y ago
      oh, yeah, I forgot to mention that I stop the running docker container before I do this
  - ericschrock 7y ago
    FYI, in the Titan project (https://titan-data.io https://titan-data.io), we solve this problem by running a privileges storage container (with ZFS on Linux) that then is able to export volumes into other containers, giving us ZFS snapshots and clones under the hood. This was inspired by the approach taken by dotmesh (https://dotmesh.com/ https://dotmesh.com/) but is definitely tailor-made to our use case. If someone was sufficiently motivated, I'm sure you could build a different CLI on top of our server and use it simply to manage docker volumes without the Titan trimmings (e.g. push and pull).
akhilcacharya 7y ago
I was under the impression container file systems make running a DB in a container a bad idea? Or is it just in performance critical situations?
- tehbeard 7y ago
  Doing so into the main container filesystem (which is usually an overlay/union stack) is probably bad for performance. But a bind mount (which is usually the recommended way to do it so that data survives the container lifespan) will avoid that and just leave you with whatever "problems" the host filesystem has re: database files.

silvester23 7y ago

Very interesting, my go-to workflow so far has been creating backup copies in phpMyAdmin but this seems much more convenient and reliable. I'll give it a try soon!

anonymousjunior 7y ago

Interesting utility. I can see a few really exciting use cases for this. If I had one wish it'd be native support for syncing generated snapshots to S3 or Google Storage for easy transfers between authenticated systems and users. Maybe I'll take a look at building out a PR for that over the weekend if that's okay

akie 7y ago
Yes, by all means please do. Simplicity of use is very important to me, so make sure it's generally applicable and easily understandable, but other than that - go for it! Awesome.