3 ms·

Well, we have exactly 1 client (our 1 control server process). So architecturally it's: 3 or 5 etcd (forget what we last deployed) <--> 1 control process <-->

Well, we have exactly 1 client (our 1 control server process). So architecturally it's: 3 or 5 etcd (forget what we last deployed) <--> 1 control process <--> every Tailscale client in the world The "Future" section is about bumping "1 control process" to "N control processes" where N will be like 2 or max 5 perhaps. The memory overhead isn't bad, as the "database" isn't big. Modern computers have tons of RAM.

judofyr 6y ago
You're able to serve all your clients from a single control process? And this would probably work for quite a while? Then I struggle to see why you couldn't just use SQLite. On startup read the full database into memory. Serve reads straight from memory. Writes go to SQLite first and if it succeeds then you update the data in memory. What am I missing here?
- crawshaw 6y ago
  We could use SQLite. (I love SQLite and have written about it before!) The goal is N control processes not for scale, but for more flexibility with deployment, canarying, etc.
  - judofyr 6y ago
    That makes sense. Thanks for answering all of my critical questions. Looks like a very nice piece of technology you’re building!
robmccoll 6y ago
I'm curious what drove the decision to move to an external store (and multinode HA config at that) now compared to using a local Go KV store like Badger or Pebble? Given that the goals seem to be improving performance over serializing a set of maps to disk as JSON on every change and keeping complexity down for fast and simple testing, a KV library would seem to accomplish both with less effort, without introducing dependence on an external service, and would enable the DB to grow out of memory if needed. Do you envision going to 2+ control processes that soon? Any consideration given to running the KV store inside the control processes themselves (either by embedding something like an etcd or by integrating a raft library and a KV store to reinvent that wheel) since you are replicating the entire DB into the client anyway? Meanwhile I'm working with application-sharded PG clusters with in-client caches with coherence maintained through Redis pubsub, so who am I to question the complexity of this setup haha.
- bradfitz 6y ago
  Yes, we're going to be moving to 2+ control servers for HA + blue/green reasons pretty soon here.

judofyr 6y ago

You're able to serve all your clients from a single control process? And this would probably work for quite a while? Then I struggle to see why you couldn't just use SQLite. On startup read the full database into memory. Serve reads straight from memory. Writes go to SQLite first and if it succeeds then you update the data in memory. What am I missing here?

crawshaw 6y ago
We could use SQLite. (I love SQLite and have written about it before!) The goal is N control processes not for scale, but for more flexibility with deployment, canarying, etc.
- judofyr 6y ago
  That makes sense. Thanks for answering all of my critical questions. Looks like a very nice piece of technology you’re building!

robmccoll 6y ago

I'm curious what drove the decision to move to an external store (and multinode HA config at that) now compared to using a local Go KV store like Badger or Pebble? Given that the goals seem to be improving performance over serializing a set of maps to disk as JSON on every change and keeping complexity down for fast and simple testing, a KV library would seem to accomplish both with less effort, without introducing dependence on an external service, and would enable the DB to grow out of memory if needed. Do you envision going to 2+ control processes that soon? Any consideration given to running the KV store inside the control processes themselves (either by embedding something like an etcd or by integrating a raft library and a KV store to reinvent that wheel) since you are replicating the entire DB into the client anyway? Meanwhile I'm working with application-sharded PG clusters with in-client caches with coherence maintained through Redis pubsub, so who am I to question the complexity of this setup haha.

bradfitz 6y ago
Yes, we're going to be moving to 2+ control servers for HA + blue/green reasons pretty soon here.