4 ms·

There are still a whole lots of pieces of the puzzle missing. I think it's all about creating ready to use containers which are running on different hosts whic

by Lennie 11y ago

There are still a whole lots of pieces of the puzzle missing. I think it's all about creating ready to use containers which are running on different hosts which can replicate their data between themselves and handle failovers (they need to connect to etcd or zookeeper or consul for that). For PostgreSQL it seem to have already solved (?) 2 times by different people: https://github.com/zalando/patroni https://github.com/zalando/patroni https://flynn.io/ https://flynn.io/ But we don't only need it for PostgreSQL. You need ready to use data-replicating Docker images for storing/indexing log files. And for statistics. Probably ready to use containers for replicated/clustered Redis. We need something for a replicated S3-like service for storing static files. Which can be used for different things like: - let's say you are running a website with Wordpress it can be configured to put it's files in a 'CDN'. So now whenever you you deploy Wordpress you point reverse proxy with caching (like Varnish) at your S3-like service. A S3-like service could also be used to hold your own Docker images. You could put a Docker registry in front of it to push and pull to. We need something, maybe also S3-type service, for storing the files of your local git repositories (which holds the sources of your Docker-images). As far as I know in the ecosystem there is still a scheduler missing which can deploy these containers on the right hosts. And there is also no standard API for starting new containers or whole machines when the other containers (or some monitoring tool) in a cluster noticed one is missing. These things take time. Lots of time. :-(

Titanous 11y ago
Flynn co-founder here. We plan to expand to support many more popular open source databases next year (and expose a framework for others to do the same). The goal is that a supported database should "just work" on the platform and be sanely configured out of the box with high availability, automatic failover/recovery, encrypted streaming backups, app credential provisioning, etc. Our Postgres appliance is unique in that it is explicitly designed to not lose data in the face of failure. Simply wrapping up a database in a container with leader election and replication is not enough, as there are many pitfalls that can cause data to be lost during failures. Of course we are limited by the guarantees the database can provide, so some datastore appliances we build may have clearly defined caveats. Storing binary blobs is another tricky matter, and we're exploring what we can do with little to no configuration. We currently store app images and git repos in Postgres, which works but will only scale so far. We are also improving our scheduler so that it is capable of providing the constraints necessary to place stateful services properly.
- Lennie 11y ago
  I think your ideas are sound, but last time I tried the manual install I couldn't get it running. So that was disappointing. That was many months ago, a lot might be changed now. I'll need to find time to have an other look. Anyway keep up the good work.
lobster_johnson 11y ago
Can anyone explain why Docker image handling is so terrible? For one, building an image creates lots of unnecessary overlays (resulting in the absurd need for "garbage collection"), rather than just compacting everything into one file? But also the whole registry thing. Why can't I just point Docker at an S3 bucket? Sure, you can run a private registry, but that involves (last I checked) running a daemon, Redis, and using "docker login" to get credentials set up... instead of, you know, a file path (like an NFS volume), an HTTP URL, or an S3 bucket. The registry seems like a nice thing for publishing public images, but it has nothing to do with private images.
- sfilipov 11y ago
  1. The idea behind the overlays is to "cache" and reuse as much as you can between images. If you have a long list of packages you install before you copy your app source to the container, you don't have to install the packages again and again every time you do a change to the source and rebuild. 2. You can point Docker to a file - use 'docker save' and 'docker load'.

by Lennie 11y ago

Titanous 11y ago

Flynn co-founder here. We plan to expand to support many more popular open source databases next year (and expose a framework for others to do the same). The goal is that a supported database should "just work" on the platform and be sanely configured out of the box with high availability, automatic failover/recovery, encrypted streaming backups, app credential provisioning, etc. Our Postgres appliance is unique in that it is explicitly designed to not lose data in the face of failure. Simply wrapping up a database in a container with leader election and replication is not enough, as there are many pitfalls that can cause data to be lost during failures. Of course we are limited by the guarantees the database can provide, so some datastore appliances we build may have clearly defined caveats. Storing binary blobs is another tricky matter, and we're exploring what we can do with little to no configuration. We currently store app images and git repos in Postgres, which works but will only scale so far. We are also improving our scheduler so that it is capable of providing the constraints necessary to place stateful services properly.

Lennie 11y ago
I think your ideas are sound, but last time I tried the manual install I couldn't get it running. So that was disappointing. That was many months ago, a lot might be changed now. I'll need to find time to have an other look. Anyway keep up the good work.

lobster_johnson 11y ago

Can anyone explain why Docker image handling is so terrible? For one, building an image creates lots of unnecessary overlays (resulting in the absurd need for "garbage collection"), rather than just compacting everything into one file? But also the whole registry thing. Why can't I just point Docker at an S3 bucket? Sure, you can run a private registry, but that involves (last I checked) running a daemon, Redis, and using "docker login" to get credentials set up... instead of, you know, a file path (like an NFS volume), an HTTP URL, or an S3 bucket. The registry seems like a nice thing for publishing public images, but it has nothing to do with private images.

sfilipov 11y ago
1. The idea behind the overlays is to "cache" and reuse as much as you can between images. If you have a long list of packages you install before you copy your app source to the container, you don't have to install the packages again and again every time you do a change to the source and rebuild. 2. You can point Docker to a file - use 'docker save' and 'docker load'.