5 ms·

> and a tailetc cluster What do you mean by this part? tailetc is a library used by the client of etcd. Running an etcd cluster is much easier than running an

> and a tailetc cluster What do you mean by this part? tailetc is a library used by the client of etcd. Running an etcd cluster is much easier than running an HA PostgreSQL or MySQL config. (I previously made LiveJournal and ran its massively sharded HA MySQL setup)

judofyr 6y ago
> What do you mean by this part? tailetc is a library used by the client of etcd. Oh. Since they have a full cache of the database I thought it was intended to be used as a separate set of servers layered in front of etcd to lessen the read load. But you're actually using it directly? Interesting. What's the impact on memory usage and scalability? Are you not worried that this will not scale over time since all clients need to have all the data?
- bradfitz 6y ago
  Well, we have exactly 1 client (our 1 control server process). So architecturally it's: 3 or 5 etcd (forget what we last deployed) <--> 1 control process <--> every Tailscale client in the world The "Future" section is about bumping "1 control process" to "N control processes" where N will be like 2 or max 5 perhaps. The memory overhead isn't bad, as the "database" isn't big. Modern computers have tons of RAM.
  - judofyr 6y ago
    You're able to serve all your clients from a single control process? And this would probably work for quite a while? Then I struggle to see why you couldn't just use SQLite. On startup read the full database into memory. Serve reads straight from memory. Writes go to SQLite first and if it succeeds then you update the data in memory. What am I missing here?
    - crawshaw 6y ago
      We could use SQLite. (I love SQLite and have written about it before!) The goal is N control processes not for scale, but for more flexibility with deployment, canarying, etc.
      - judofyr 6y ago
        That makes sense. Thanks for answering all of my critical questions. Looks like a very nice piece of technology you’re building!
  - robmccoll 6y ago
    I'm curious what drove the decision to move to an external store (and multinode HA config at that) now compared to using a local Go KV store like Badger or Pebble? Given that the goals seem to be improving performance over serializing a set of maps to disk as JSON on every change and keeping complexity down for fast and simple testing, a KV library would seem to accomplish both with less effort, without introducing dependence on an external service, and would enable the DB to grow out of memory if needed. Do you envision going to 2+ control processes that soon? Any consideration given to running the KV store inside the control processes themselves (either by embedding something like an etcd or by integrating a raft library and a KV store to reinvent that wheel) since you are replicating the entire DB into the client anyway? Meanwhile I'm working with application-sharded PG clusters with in-client caches with coherence maintained through Redis pubsub, so who am I to question the complexity of this setup haha.
    - bradfitz 6y ago
      Yes, we're going to be moving to 2+ control servers for HA + blue/green reasons pretty soon here.
mwcampbell 6y ago
> Running an etcd cluster is much easier than running an HA PostgreSQL or MySQL config. What if you used one of the managed RDBMS services offered by the big cloud providers? BTW, if you don't mind sharing, where are you hosting the control plane?
- bradfitz 6y ago
  > What if you used one of the managed RDBMS services offered by the big cloud providers? We could (and likely would, despite the costs) but that doesn't address our testing requirements. The control plane is on AWS. We use 4 or 5 different cloud providers (Tailscale makes that much easier) but the most important bit is on AWS.
  - nicoburns 6y ago
    Why is testing Postgres/MySQL difficult? You can easily run a server locally (or on CI) and create new databases for test runs, etc.
    - bradfitz 6y ago
      It's not difficult. We've done it before and have code for it. See the article.
  - ignoramous 6y ago
    > ...but the most important bit is on AWS. Curious: Was running DynamoDB with DAX (DynamoDB Accelerator) in front ever in contention? If not, is it due to vendor lock-in (for example, not being able to migrate out) or because tailscale doesn't feel the need to use managed offerings especially for core infrastructure?
- cakoose 6y ago
  > > Running an etcd cluster is much easier than running an HA PostgreSQL or MySQL config. > What if you used one of the managed RDBMS services offered by the big cloud providers? Yeah, AWS RDS "multi-AZ" does a good job of taking care of HA for you. (Google Cloud SQL's HA setup is extremely similar.) But you still get 1-2 minutes of full unavailability when hardware fails. I haven't operated etcd in production myself, but I assume it does better because it's designed specifically for HA. You can't even run less than three nodes. (The etcd docs talk about election timeouts on the order of 1s, which is encouraging.) For many use cases, 1-2 minutes of downtime is tolerable. But I can imagine situations where availability is paramount and you're willing to give up scale/performance/features to gain another 9.
jgraettinger1 6y ago
Neat. This is very similar to [0], which is _not_ a cache but rather a complete mirror of an Etcd keyspace. It does Key/Value decoding up front, into a user-defined & validated runtime type, and promises to never mutate an existing instance (instead decoding into a new instance upon revision change). The typical workflow is do do all of your "reads" out of the keyspace, attempt to apply Etcd transactions, and (if needed) block until your keyspace has caught up such that you read your write -- or someone else's conflicting write. [0] https://pkg.go.dev/go.gazette.dev/core/keyspace https://pkg.go.dev/go.gazette.dev/core/keyspace
- bradfitz 6y ago
  Whoa, we hadn't seen that! At first glance it indeed appears to be perhaps exactly identical to what we did.
  - jgraettinger1 6y ago
    Slightly different trade-offs. This package is emphatically just "for" Etcd, choosing to directly expose MVCC types & concepts from the client. It also doesn't wrap transactions -- you use the etcd client directly for that. The Nagel delay it implements helps quite a bit with scaling, though, while keeping the benefits of a tightly packed sorted keyspace. And you can directly access / walk decoded state without copies.
- crawshaw 6y ago
  Drat! I went looking for people doing something similar when I sat down to design our client, but did not find your package. That's a real pity, I would love to have collaborated on this. I guess Go package discovery remains an unsolved problem.
  - staunch 6y ago
    > I guess Go package discovery remains an unsolved problem. Or did you just not really search, like most of us excited to DIY? :-D Godoc is pretty good, the package shows up for the searches I'd probably do in a similar situation. https://godoc.org/?q=etcd https://godoc.org/?q=etcd https://godoc.org/?q=etcd+watch https://godoc.org/?q=etcd+watch

judofyr 6y ago

> What do you mean by this part? tailetc is a library used by the client of etcd. Oh. Since they have a full cache of the database I thought it was intended to be used as a separate set of servers layered in front of etcd to lessen the read load. But you're actually using it directly? Interesting. What's the impact on memory usage and scalability? Are you not worried that this will not scale over time since all clients need to have all the data?

bradfitz 6y ago
Well, we have exactly 1 client (our 1 control server process). So architecturally it's: 3 or 5 etcd (forget what we last deployed) <--> 1 control process <--> every Tailscale client in the world The "Future" section is about bumping "1 control process" to "N control processes" where N will be like 2 or max 5 perhaps. The memory overhead isn't bad, as the "database" isn't big. Modern computers have tons of RAM.
- judofyr 6y ago
  You're able to serve all your clients from a single control process? And this would probably work for quite a while? Then I struggle to see why you couldn't just use SQLite. On startup read the full database into memory. Serve reads straight from memory. Writes go to SQLite first and if it succeeds then you update the data in memory. What am I missing here?
  - crawshaw 6y ago
    We could use SQLite. (I love SQLite and have written about it before!) The goal is N control processes not for scale, but for more flexibility with deployment, canarying, etc.
    - judofyr 6y ago
      That makes sense. Thanks for answering all of my critical questions. Looks like a very nice piece of technology you’re building!
- robmccoll 6y ago
  I'm curious what drove the decision to move to an external store (and multinode HA config at that) now compared to using a local Go KV store like Badger or Pebble? Given that the goals seem to be improving performance over serializing a set of maps to disk as JSON on every change and keeping complexity down for fast and simple testing, a KV library would seem to accomplish both with less effort, without introducing dependence on an external service, and would enable the DB to grow out of memory if needed. Do you envision going to 2+ control processes that soon? Any consideration given to running the KV store inside the control processes themselves (either by embedding something like an etcd or by integrating a raft library and a KV store to reinvent that wheel) since you are replicating the entire DB into the client anyway? Meanwhile I'm working with application-sharded PG clusters with in-client caches with coherence maintained through Redis pubsub, so who am I to question the complexity of this setup haha.
  - bradfitz 6y ago
    Yes, we're going to be moving to 2+ control servers for HA + blue/green reasons pretty soon here.

mwcampbell 6y ago

> Running an etcd cluster is much easier than running an HA PostgreSQL or MySQL config. What if you used one of the managed RDBMS services offered by the big cloud providers? BTW, if you don't mind sharing, where are you hosting the control plane?

bradfitz 6y ago
> What if you used one of the managed RDBMS services offered by the big cloud providers? We could (and likely would, despite the costs) but that doesn't address our testing requirements. The control plane is on AWS. We use 4 or 5 different cloud providers (Tailscale makes that much easier) but the most important bit is on AWS.
- nicoburns 6y ago
  Why is testing Postgres/MySQL difficult? You can easily run a server locally (or on CI) and create new databases for test runs, etc.
  - bradfitz 6y ago
    It's not difficult. We've done it before and have code for it. See the article.
- ignoramous 6y ago
  > ...but the most important bit is on AWS. Curious: Was running DynamoDB with DAX (DynamoDB Accelerator) in front ever in contention? If not, is it due to vendor lock-in (for example, not being able to migrate out) or because tailscale doesn't feel the need to use managed offerings especially for core infrastructure?
cakoose 6y ago
> > Running an etcd cluster is much easier than running an HA PostgreSQL or MySQL config. > What if you used one of the managed RDBMS services offered by the big cloud providers? Yeah, AWS RDS "multi-AZ" does a good job of taking care of HA for you. (Google Cloud SQL's HA setup is extremely similar.) But you still get 1-2 minutes of full unavailability when hardware fails. I haven't operated etcd in production myself, but I assume it does better because it's designed specifically for HA. You can't even run less than three nodes. (The etcd docs talk about election timeouts on the order of 1s, which is encouraging.) For many use cases, 1-2 minutes of downtime is tolerable. But I can imagine situations where availability is paramount and you're willing to give up scale/performance/features to gain another 9.

jgraettinger1 6y ago

Neat. This is very similar to [0], which is _not_ a cache but rather a complete mirror of an Etcd keyspace. It does Key/Value decoding up front, into a user-defined & validated runtime type, and promises to never mutate an existing instance (instead decoding into a new instance upon revision change). The typical workflow is do do all of your "reads" out of the keyspace, attempt to apply Etcd transactions, and (if needed) block until your keyspace has caught up such that you read your write -- or someone else's conflicting write. [0] https://pkg.go.dev/go.gazette.dev/core/keyspace https://pkg.go.dev/go.gazette.dev/core/keyspace

bradfitz 6y ago
Whoa, we hadn't seen that! At first glance it indeed appears to be perhaps exactly identical to what we did.
- jgraettinger1 6y ago
  Slightly different trade-offs. This package is emphatically just "for" Etcd, choosing to directly expose MVCC types & concepts from the client. It also doesn't wrap transactions -- you use the etcd client directly for that. The Nagel delay it implements helps quite a bit with scaling, though, while keeping the benefits of a tightly packed sorted keyspace. And you can directly access / walk decoded state without copies.
crawshaw 6y ago
Drat! I went looking for people doing something similar when I sat down to design our client, but did not find your package. That's a real pity, I would love to have collaborated on this. I guess Go package discovery remains an unsolved problem.
- staunch 6y ago
  > I guess Go package discovery remains an unsolved problem. Or did you just not really search, like most of us excited to DIY? :-D Godoc is pretty good, the package shows up for the searches I'd probably do in a similar situation. https://godoc.org/?q=etcd https://godoc.org/?q=etcd https://godoc.org/?q=etcd+watch https://godoc.org/?q=etcd+watch