6 ms·

So to recap their timeline: * Instead of an actual database use a JSON file. * Write a blog post about how that didn't scale. * Instead of an actual data

So to recap their timeline: * Instead of an actual database use a JSON file. * Write a blog post about how that didn't scale. * Instead of an actual database hand-roll something else. * Write a blog post about how that didn't scale. * Instead of a database with built-in replication which is tailor built for key-value storage use SQLite with a single table containing key-value pairs and some fresh glue to make it replicate. I'm sorry, but what the fuck? Just writing the blog posts alone might have consumed more work-hours than what it would have taken to set a database up in the first place. Are we supposed to expect another blog post somewhere around a year down the road titled "SQLite didn't scale" or "The new library for replication that was released a few months ago contained a bug that took our prod down"?

srameshc 5y ago
Why not ? You can be a spectator to it watch them succeed or fail.
Dylan16807 5y ago
Okay, but this is a database now. Is there a reason the replication needs to be built-in?
- artificialLimbs 5y ago
  I thought replication was useful for backups or testing. Maybe other things. edit: I know practically nothing about db.
- deleted 5y ago
  [deleted]
- musjleman 5y ago
  Sorry, I probably didn't phrase it very clearly. I don't think there is anything wrong with needing some external help for that, but the combination of their choices is IMO a bit bizarre: * SQLite which I mostly see as an embeddable database for applications, used on a server. * Single table with key-value pairs. * Having the tool for replication be a quite young project that's in active development.
  - tptacek 5y ago
    Plenty of people use sqlite on servers. They have a single table with KV pairs right now as an artifact of having used the "database with built-in replication which is tailor built for key-value storage" you complain that they're moving away from. The point of the blog post is that they're doing that in part to move away from the single table of KV pairs. They're not "sharding".
  - ripley12 5y ago
    > SQLite which I mostly see as an embeddable database for applications This is common but incorrect view of SQLite. It's able to handle many server-side workloads, as long as you're aware of its limitations (and the Tailscale team certainly are). https://www.sqlite.org/whentouse.html https://www.sqlite.org/whentouse.html
    - tptacek 5y ago
      Those limitations are also the limitations of the backing stores of many distributed state solutions; you just don't see them because they're hidden behind complicated APIs.
deleted 5y ago
[deleted]
bradfitz 5y ago
Tailscale employee here. Our database needs are tiny, as explained in the earlier post. So we optimize for things like: "can we run all our tests quickly and easily in many environments without containers and VMs?" All three of our storage schemes have had that property. We have MySQL and PostgreSQL veterans on the team. We know those options well.
- cogman10 5y ago
  > without containers Why? The official docker postgres package weighs in at 100MB. Assuming you standardized on that, then every environment would end up having a fresh postgres image just waiting to be startup. Meaning, the actual cost is the memory for the server and startup time, not the image itself. Mount the data on tmpfs and you can startup and setup a db in very little time (I know, because that's what we do). You might be able to eek out a bit more performance by hand tuning test files... but.. to what end?
  - tptacek 5y ago
    I can't speak for Tailscale, but our whole raison d'etre is running containers for people, and almost none of our development environments are containerized either. Containers (and remote dev environments) create friction. They're useful tools when that friction is unavoidable anyways, but when it isn't --- and you can design systems with that being a goal --- it makes sense to take full advantage of that.
  - dgb23 5y ago
    Have a look at this headline: > Zero config VPN. Installs on any device in minutes, manages firewall rules for you, and works from anywhere. I don't know how their product works, but my intuition is that your suggestions aren't feasible.
    - cogman10 5y ago
      I suggest reading the article. The reason they haven't chosen "boring" tech has nothing to do with the clients running their software and everything to do with their internal development experiences. > MySQL (or PostgreSQL) would come next. I’m not particularly familiar with anything MySQL post 1998, but I’m sure it would work. The HA story for open source databases is somewhat surprising, though: you can either have traditional lagging replicas, or commit to no-primary-replica clusters that have very surprising transaction semantics. I wasn’t excited about trying to design a stable API or good network graph calculations on top of those semantics. CockroachDB looked very promising, and indeed still does! But it’s relatively new for a database and I was a little concerned about getting attached to features in a fresh DBMS that would be hard to migrate away from if we needed to. The reason they are ruling out products isn't "this won't work where it runs" its "I don't like this". They talk about worrying about network graphs and proceed to use SQLite and copy garbage around.
VWWHFSfQ 5y ago
It seems like they're _really_ struggling with is. It's kind of entertaining to watch though edit: also don't forget these blog posts are content marketing for their product. that's really the point anyway
- yawaramin 5y ago
  If from your perspective, a super-successful VPN product is 'really struggling', you must have incredibly high standards! We all hope to hear more about your successes, I'm sure.
  - VWWHFSfQ 5y ago
    do you want to hear about my successes? they're not going to be about shoe-horning a JSON file or a SQLite database file into my enterprise product.
    - samhw 5y ago
      Sure, so what are they about?
    - yawaramin 5y ago
      Just waiting with bated breath to find out!
Aeolun 5y ago
> In the first place Yes, but ongoing maintenance isn’t free. If anything, this whole DB funkiness indicates to me just how nice a product you can build despite it.
tptacek 5y ago
Do you want to know how popular systems are actually built, from the inside, or do you want carefully groomed triumphal announcements of new, perfectly-formed features? I struggle with this a lot in my writing too, but I've drawn a conclusion I'm sticking with: it's better to relay what the team is actually doing, "warts" and all. We ranked on HN a few weeks ago with a post about user-mode WireGuard that was basically a litany of engineering mistakes, and it's one of the things I've most enjoyed writing. Every serious product gets built the way Tailscale is describing. You start some components with the simplest thing that can reasonably work, and see how far you can take them. The JSON file Tailscale used to use was one of the most interesting and valuable things they wrote about. If you want content about how to built best-practices-compliant hyperscaler-grade systems, there are places to go to get that kind of content. But you've never had a problem with Tailscale that owed to their JSON file backend not scaling --- how I know that is, you've likely never experienced a stability problem with Tailscale at all, because it just doesn't break. That they got it to work, for a long time, with a JSON file backend should make you think harder about how you build things.
- rob_c 5y ago
  Yes but userland wireguard is unique and useful. This is ignoring textbooks, competitors who write about their exploits warts and all and screams of get a more experienced db team to work with you... shoving it into a json screams of throw it at a disk and let the hardware deal with it
  - lostcolony 5y ago
    >> shoving it into a json screams of throw it at a disk and let the hardware deal with it Which...worked. For quite a long time. And required no dev effort. Freeing devs up to focus on other things. They did it as a POC, and it worked, better even than they had anticipated; why over-engineer it up front? Even when it turns out they are gonna need it (i.e., YAGNI no longer applies), they have been able to push off their "solve problems customers don't care about" to instead favor "problems customers do care about". That...seems like sound engineering to me?
dan-robertson 5y ago
Doesn’t the blog post explain why they chose this step rather than doing the thing you suggest? In particular I don’t see how this comment really addresses any of their arguments. And if they got on ok with the json file, maybe you should update towards the features of a complicated RDBMS not being worth the cost?
- cogman10 5y ago
  No, it actually doesn't. The blog post explains "Hey, we've chosen a bunch of crazy ways to store data on the backend. We looked at non-crazy ways, but decided instead it'd be more fun to rsync SQLite data around because SQLite is my favorite DB!" > MySQL (or PostgreSQL) would come next. I’m not particularly familiar with anything MySQL post 1998, but I’m sure it would work. The HA story for open source databases is somewhat surprising, though: you can either have traditional lagging replicas, or commit to no-primary-replica clusters that have very surprising transaction semantics. I wasn’t excited about trying to design a stable API or good network graph calculations on top of those semantics. CockroachDB looked very promising, and indeed still does! But it’s relatively new for a database and I was a little concerned about getting attached to features in a fresh DBMS that would be hard to migrate away from if we needed to. Seriously.. consider this.. they are worried about the transaction semantics of using db replicas and instead CHOSE SQL LITE!
  - tptacek 5y ago
    And? (It's sqlite, or SQLite if you're being pedantic; it's not "SQL LITE").
  - bccdee 5y ago
    Sqlite replication via litestream has bascially no overhead. It's a great choice if your database only needs to service one client. It sounds like you're essentially arguing that people should always use the most robust data store available regardless of cost or requirements. Their needs were served just fine by a json file for quite some time — if that's all you need, then maintaining a full Posgres cluster is the "crazy" solution actually.
deleted 5y ago
[deleted]
- deleted 5y ago
  [deleted]

srameshc 5y ago

Why not ? You can be a spectator to it watch them succeed or fail.

Dylan16807 5y ago

Okay, but this is a database now. Is there a reason the replication needs to be built-in?

artificialLimbs 5y ago
I thought replication was useful for backups or testing. Maybe other things. edit: I know practically nothing about db.
deleted 5y ago
[deleted]
musjleman 5y ago
Sorry, I probably didn't phrase it very clearly. I don't think there is anything wrong with needing some external help for that, but the combination of their choices is IMO a bit bizarre: * SQLite which I mostly see as an embeddable database for applications, used on a server. * Single table with key-value pairs. * Having the tool for replication be a quite young project that's in active development.
- tptacek 5y ago
  Plenty of people use sqlite on servers. They have a single table with KV pairs right now as an artifact of having used the "database with built-in replication which is tailor built for key-value storage" you complain that they're moving away from. The point of the blog post is that they're doing that in part to move away from the single table of KV pairs. They're not "sharding".
- ripley12 5y ago
  > SQLite which I mostly see as an embeddable database for applications This is common but incorrect view of SQLite. It's able to handle many server-side workloads, as long as you're aware of its limitations (and the Tailscale team certainly are). https://www.sqlite.org/whentouse.html https://www.sqlite.org/whentouse.html
  - tptacek 5y ago
    Those limitations are also the limitations of the backing stores of many distributed state solutions; you just don't see them because they're hidden behind complicated APIs.

deleted 5y ago

[deleted]

bradfitz 5y ago

Tailscale employee here. Our database needs are tiny, as explained in the earlier post. So we optimize for things like: "can we run all our tests quickly and easily in many environments without containers and VMs?" All three of our storage schemes have had that property. We have MySQL and PostgreSQL veterans on the team. We know those options well.

cogman10 5y ago
> without containers Why? The official docker postgres package weighs in at 100MB. Assuming you standardized on that, then every environment would end up having a fresh postgres image just waiting to be startup. Meaning, the actual cost is the memory for the server and startup time, not the image itself. Mount the data on tmpfs and you can startup and setup a db in very little time (I know, because that's what we do). You might be able to eek out a bit more performance by hand tuning test files... but.. to what end?
- tptacek 5y ago
  I can't speak for Tailscale, but our whole raison d'etre is running containers for people, and almost none of our development environments are containerized either. Containers (and remote dev environments) create friction. They're useful tools when that friction is unavoidable anyways, but when it isn't --- and you can design systems with that being a goal --- it makes sense to take full advantage of that.
- dgb23 5y ago
  Have a look at this headline: > Zero config VPN. Installs on any device in minutes, manages firewall rules for you, and works from anywhere. I don't know how their product works, but my intuition is that your suggestions aren't feasible.
  - cogman10 5y ago
    I suggest reading the article. The reason they haven't chosen "boring" tech has nothing to do with the clients running their software and everything to do with their internal development experiences. > MySQL (or PostgreSQL) would come next. I’m not particularly familiar with anything MySQL post 1998, but I’m sure it would work. The HA story for open source databases is somewhat surprising, though: you can either have traditional lagging replicas, or commit to no-primary-replica clusters that have very surprising transaction semantics. I wasn’t excited about trying to design a stable API or good network graph calculations on top of those semantics. CockroachDB looked very promising, and indeed still does! But it’s relatively new for a database and I was a little concerned about getting attached to features in a fresh DBMS that would be hard to migrate away from if we needed to. The reason they are ruling out products isn't "this won't work where it runs" its "I don't like this". They talk about worrying about network graphs and proceed to use SQLite and copy garbage around.

VWWHFSfQ 5y ago

It seems like they're _really_ struggling with is. It's kind of entertaining to watch though edit: also don't forget these blog posts are content marketing for their product. that's really the point anyway

yawaramin 5y ago
If from your perspective, a super-successful VPN product is 'really struggling', you must have incredibly high standards! We all hope to hear more about your successes, I'm sure.
- VWWHFSfQ 5y ago
  do you want to hear about my successes? they're not going to be about shoe-horning a JSON file or a SQLite database file into my enterprise product.
  - samhw 5y ago
    Sure, so what are they about?
  - yawaramin 5y ago
    Just waiting with bated breath to find out!

Aeolun 5y ago

> In the first place Yes, but ongoing maintenance isn’t free. If anything, this whole DB funkiness indicates to me just how nice a product you can build despite it.

tptacek 5y ago

Do you want to know how popular systems are actually built, from the inside, or do you want carefully groomed triumphal announcements of new, perfectly-formed features? I struggle with this a lot in my writing too, but I've drawn a conclusion I'm sticking with: it's better to relay what the team is actually doing, "warts" and all. We ranked on HN a few weeks ago with a post about user-mode WireGuard that was basically a litany of engineering mistakes, and it's one of the things I've most enjoyed writing. Every serious product gets built the way Tailscale is describing. You start some components with the simplest thing that can reasonably work, and see how far you can take them. The JSON file Tailscale used to use was one of the most interesting and valuable things they wrote about. If you want content about how to built best-practices-compliant hyperscaler-grade systems, there are places to go to get that kind of content. But you've never had a problem with Tailscale that owed to their JSON file backend not scaling --- how I know that is, you've likely never experienced a stability problem with Tailscale at all, because it just doesn't break. That they got it to work, for a long time, with a JSON file backend should make you think harder about how you build things.

rob_c 5y ago
Yes but userland wireguard is unique and useful. This is ignoring textbooks, competitors who write about their exploits warts and all and screams of get a more experienced db team to work with you... shoving it into a json screams of throw it at a disk and let the hardware deal with it
- lostcolony 5y ago
  >> shoving it into a json screams of throw it at a disk and let the hardware deal with it Which...worked. For quite a long time. And required no dev effort. Freeing devs up to focus on other things. They did it as a POC, and it worked, better even than they had anticipated; why over-engineer it up front? Even when it turns out they are gonna need it (i.e., YAGNI no longer applies), they have been able to push off their "solve problems customers don't care about" to instead favor "problems customers do care about". That...seems like sound engineering to me?

dan-robertson 5y ago

Doesn’t the blog post explain why they chose this step rather than doing the thing you suggest? In particular I don’t see how this comment really addresses any of their arguments. And if they got on ok with the json file, maybe you should update towards the features of a complicated RDBMS not being worth the cost?

cogman10 5y ago
No, it actually doesn't. The blog post explains "Hey, we've chosen a bunch of crazy ways to store data on the backend. We looked at non-crazy ways, but decided instead it'd be more fun to rsync SQLite data around because SQLite is my favorite DB!" > MySQL (or PostgreSQL) would come next. I’m not particularly familiar with anything MySQL post 1998, but I’m sure it would work. The HA story for open source databases is somewhat surprising, though: you can either have traditional lagging replicas, or commit to no-primary-replica clusters that have very surprising transaction semantics. I wasn’t excited about trying to design a stable API or good network graph calculations on top of those semantics. CockroachDB looked very promising, and indeed still does! But it’s relatively new for a database and I was a little concerned about getting attached to features in a fresh DBMS that would be hard to migrate away from if we needed to. Seriously.. consider this.. they are worried about the transaction semantics of using db replicas and instead CHOSE SQL LITE!
- tptacek 5y ago
  And? (It's sqlite, or SQLite if you're being pedantic; it's not "SQL LITE").
- bccdee 5y ago
  Sqlite replication via litestream has bascially no overhead. It's a great choice if your database only needs to service one client. It sounds like you're essentially arguing that people should always use the most robust data store available regardless of cost or requirements. Their needs were served just fine by a json file for quite some time — if that's all you need, then maintaining a full Posgres cluster is the "crazy" solution actually.

deleted 5y ago

[deleted]

deleted 5y ago
[deleted]