4 ms·

The beauty of containers is how ephemeral they are — need more? Here’s more. Need less? Bye. Oh, this one died? Kill it. Terrible fit for databases, inherently

The beauty of containers is how ephemeral they are — need more? Here’s more. Need less? Bye. Oh, this one died? Kill it. Terrible fit for databases, inherently.

ghshephard 6y ago
But, don't you want to code your application stack such that your database being killed is a no-op? Shouldn't the state mostly reside in your persistent store (which decidedly needs to be robust)
- kjeetgill 6y ago
  Maybe I'm not newfangled enough but isn't the database your persistent store? If not, maybe this is where the db in vs out of container folks are taking past each other.
  - ghshephard 6y ago
    I'm calling out the difference between the storage mechanism (SSDs, Block Storage, etc...) - where transactions and journal-logs are dumped, so as to allow recovery if the database server goes down. Once you design your database server/application/transactions such they don't care whatsoever if the database server goes offline - you not only get the ability to run fine in containers, you also make your application stack just significantly more robust for all sorts of reasons. I have colleagues who bemoan a major application server going down - which is a no-op in my world where the ephemeral nodes rarely last more than a day or two in GKE - so we're completely used to them going down. Building transactions, idempotency, tasks in rabbitmq/celery rather than memory - not only do they make deploying in containers straightforward, they also give an overall robustness win for all applications. I mean, physical servers go down as well.
    - kjeetgill 6y ago
      Sorry, I'm not quite piecing together what scheme you're proposing. Like, where are you storing things like your user/password tables that aren't persistent databases? I'm familiar with the topics around HA and how you could ignore individual hosts going down, but the whole database?
      - singingboyo 6y ago
        Not who you replied to, but: you're probably thinking Frontend->Backend->DB or similar, right? Frontend runs in browser and is either served by its own container or the backend, backend is in a containerized server, DB in something more persistent. Except, the reality is it's Frontend->Backend->DB->Storage. To some extent, the DB is just a standardized (and very quick) caching/sorting mechanism in front of an SSD, HDD, network drive, whatever. Storage can be separate and persistent, even if the DB is in a container and gets killed/dies. If that's the case, then even if the DB is in an ephemeral container the backing storage should always survive. With it, you can bring up a new DB container with all the same data being available. So yes, your passwords etc are in the "DB", but really they're in storage on disk. Yeah, your app won't work well if the DB disappears, but that's a given no matter whether your DB is in a container or somewhere else. Once it's back up, everything should work fine.
      - ghshephard 6y ago
        When the database is down, no transactions occur. But as long as the storage medium (SSD, Spinning disks, PVC, whatever) - doesn't go down - then when the database comes back up - anything that needs to be rolled back is rolled back, and things just keep running along without any problem. So - the user will see a "Pause" when using the application if the database is down - but that's it. The great thing about architecting your application stack in such a way that every one of the servers/applications/databases can disappear in a moments notice, is that your application becomes super robust to hardware failure - because having a sever go "down" is Business As Usual in Kubernetes.
- slaymaker1907 6y ago
  While this is somewhat true, most databases are not designed for _fast_ recovery.
boardwaalk 6y ago
You’re making a feature into a disadvantage when you could just not use the feature.
slaymaker1907 6y ago
That's only one way of looking at containers. Another way is to see them as a modularization tool since they abstract most of the machine away while being much lighter weight than a VM. I've recently been looking into containerizing my personal server. I use the cheapest VM on AWS (I work on Azure now, but vendor lock in is real). Adding another VM would double my costs, but with containers I would more isolation and can upgrade sites I run on this server independently. I usually use sqlite, but I can see why containers would be nice for similar reasons. Even if you aren't resource constrained with VMs, running a DB via containers might be nice to ensure a consistent dev setup even if you don't use containers in prod.
- jasonpeacock 6y ago
  I do this using Lightsail to run my personal blogs, each blog is running in its own container w/sqlite mounted from a local for persistence (too lazy to do actual Docker volumes). Ditto for Nginx, it's also containerized. It's a great setup b/c I can also deploy the containers locally (and SCP the sqlite DB locally) to run the blogs on my laptop if I need to.

The beauty of containers is how ephemeral they are — need more? Here’s more. Need less? Bye. Oh, this one died? Kill it. Terrible fit for databases, inherently.

ghshephard 6y ago

But, don't you want to code your application stack such that your database being killed is a no-op? Shouldn't the state mostly reside in your persistent store (which decidedly needs to be robust)

kjeetgill 6y ago
Maybe I'm not newfangled enough but isn't the database your persistent store? If not, maybe this is where the db in vs out of container folks are taking past each other.
- ghshephard 6y ago
  I'm calling out the difference between the storage mechanism (SSDs, Block Storage, etc...) - where transactions and journal-logs are dumped, so as to allow recovery if the database server goes down. Once you design your database server/application/transactions such they don't care whatsoever if the database server goes offline - you not only get the ability to run fine in containers, you also make your application stack just significantly more robust for all sorts of reasons. I have colleagues who bemoan a major application server going down - which is a no-op in my world where the ephemeral nodes rarely last more than a day or two in GKE - so we're completely used to them going down. Building transactions, idempotency, tasks in rabbitmq/celery rather than memory - not only do they make deploying in containers straightforward, they also give an overall robustness win for all applications. I mean, physical servers go down as well.
  - kjeetgill 6y ago
    Sorry, I'm not quite piecing together what scheme you're proposing. Like, where are you storing things like your user/password tables that aren't persistent databases? I'm familiar with the topics around HA and how you could ignore individual hosts going down, but the whole database?
    - singingboyo 6y ago
      Not who you replied to, but: you're probably thinking Frontend->Backend->DB or similar, right? Frontend runs in browser and is either served by its own container or the backend, backend is in a containerized server, DB in something more persistent. Except, the reality is it's Frontend->Backend->DB->Storage. To some extent, the DB is just a standardized (and very quick) caching/sorting mechanism in front of an SSD, HDD, network drive, whatever. Storage can be separate and persistent, even if the DB is in a container and gets killed/dies. If that's the case, then even if the DB is in an ephemeral container the backing storage should always survive. With it, you can bring up a new DB container with all the same data being available. So yes, your passwords etc are in the "DB", but really they're in storage on disk. Yeah, your app won't work well if the DB disappears, but that's a given no matter whether your DB is in a container or somewhere else. Once it's back up, everything should work fine.
    - ghshephard 6y ago
      When the database is down, no transactions occur. But as long as the storage medium (SSD, Spinning disks, PVC, whatever) - doesn't go down - then when the database comes back up - anything that needs to be rolled back is rolled back, and things just keep running along without any problem. So - the user will see a "Pause" when using the application if the database is down - but that's it. The great thing about architecting your application stack in such a way that every one of the servers/applications/databases can disappear in a moments notice, is that your application becomes super robust to hardware failure - because having a sever go "down" is Business As Usual in Kubernetes.
slaymaker1907 6y ago
While this is somewhat true, most databases are not designed for _fast_ recovery.

boardwaalk 6y ago

You’re making a feature into a disadvantage when you could just not use the feature.

slaymaker1907 6y ago

That's only one way of looking at containers. Another way is to see them as a modularization tool since they abstract most of the machine away while being much lighter weight than a VM. I've recently been looking into containerizing my personal server. I use the cheapest VM on AWS (I work on Azure now, but vendor lock in is real). Adding another VM would double my costs, but with containers I would more isolation and can upgrade sites I run on this server independently. I usually use sqlite, but I can see why containers would be nice for similar reasons. Even if you aren't resource constrained with VMs, running a DB via containers might be nice to ensure a consistent dev setup even if you don't use containers in prod.

jasonpeacock 6y ago
I do this using Lightsail to run my personal blogs, each blog is running in its own container w/sqlite mounted from a local for persistence (too lazy to do actual Docker volumes). Ditto for Nginx, it's also containerized. It's a great setup b/c I can also deploy the containers locally (and SCP the sqlite DB locally) to run the blogs on my laptop if I need to.