4 ms·

I think I'm missing some context. Using a text file and using etcd as a DB for a production system seems like a terrible engineering decision. It seems like som

I think I'm missing some context. Using a text file and using etcd as a DB for a production system seems like a terrible engineering decision. It seems like something you'd do as a proof of concept or side project. It's interesting that they're blogging about this, as if they're proud of it. I guess I'm just missing the point. This is their 3rd DB Migration, something that I prefer to avoid at all costs. I guess they have different priorities and values. But I'm just confused, why would an engineer want to join a company that is making these decisions? Why would the company want their users to know about these decisions? They started using etcd knowing that he would become a bottle neck, that should have been a non-starter, right?

bradfitz 5y ago
Tailscale engineer here. > Using a text file and ... > It seems like something you'd do as a proof of concept I mean, using a text file for the proof of concept is exactly what happened. And then it grew too quickly and had to get off of it eventually, but we always knew that. We were just amazed how long that worked. It survived much longer than we'd thought.
- mardifoufs 5y ago
Aeolun 5y ago
> why would an engineer want to join a company that is making these decisions? Hmm, I can’t quite articulate why, but I would. Something about having the courage to be different.
- fiddlerwoaroof 5y ago
  Yeah, it’s perfectly reasonable to use a text file instead of a database until you find evidence that you need a “real” database.
- dave78 5y ago
  I would actually love to work at a company like that. I've been a professional developer for over 20 years now and at least in my experience, too much unneeded complexity has been the root cause of so many defects and product failures. I'm constantly fighting battles to keep things simple. All the young devs on my team want to use every fancy new technology that passes by, but I just want things to work for my customers with the smallest amount of downtime and late night pages. If a single JSON file worked at the time, then that's clearly all they needed. It certainly hasn't seemed to limit their success.
  - billywhizz 5y ago
    > All the young devs on my team want to use every fancy new technology that passes by, but I just want things to work for my customers with the smallest amount of downtime and late night pages. i feel your pain dave. resume driven development and FAANG envy seem to be the dominant characteristics of far too many folks in the industry these days.
  - CRConrad 5y ago
    > I'm constantly fighting battles to keep things simple. All the young devs on my team want to use every fancy new technology that passes by... For me, it's not so much my fellow developers -- half my age, but quite bright enough to realise the benefits of simplicity -- but that the vendors of the tools our corporation has decided to use have in turn decided to implement them not just as a spaghetti-reinforced mudball, but as a spaghetti-reinforced mudball of spaghetti-reinforced mudballs. (That these SRMB^2s are then just part of a larger spaghetti-reinforced mudball of SRMB^2s, i.e a spaghetti-reinforced mudball cubed, is of course not the vendors' fault but corporate. But then, aren't corporate systems always SRMBs? So if at least the components were simple, the end result would be only an SRM to the power of one, not three... Sigh.)
matthewmacleod 5y ago
Using a text file and using etcd as a DB for a production system seems like a terrible engineering decision I'd say this could easily be a great engineering decision if it meets the needs of an organisation in the moment. The thing with any engineering decision is that it involves a bunch of trade-offs – doing something that you know will hit a scaling wall can be worth it, if it can get you moving faster. There will never be a perfect solution, and in these decisions I find that simplicity is often a massively under-rated benefit. I'd even venture to say that I've seen more organisational failures as a result of optimistic over-engineering than I have optimistic under-engineering. That includes many of my own projects – ones where I've gone "oh I guess I'll do it properly this time" and then found myself six months down the line throwing out a ton of code, schema, or infrastructure I spent ages on because some requirements changed or some assumptions didn't pan out. It's hard to remember that lesson sometimes and I've been doing this for 15 years. It's like a company I used to work for where the whole product was driven by a massive semi-structured graph database – but instead of being implemented like that it was a big ball of MongoDB JSON documents glued together with PHP by someone who was more of a domain expert than an engineer. Often pretty sketchy to work on, but an engineering decision that was a success – in the sense that it allowed that product to exist at all where it otherwise might not have. why would an engineer want to join a company that is making these decisions? Why would the company want their users to know about these decisions? Sign me up as both an engineer and a user. It is extraordinarily refreshing to hear "we tried a bunch of different stuff, some of it was dumb, and this is where we're at now."
kevin_nisbet 5y ago
> But I'm just confused, why would an engineer want to join a company that is making these decisions? Why would the company want their users to know about these decisions? This might actually be a really good filter for which types of engineers are good fit for their company. Because my read was very different than yours, as I looked at the decision lineage and thought to myself I could see myself making every one of those decisions. I of course can't say I would've with the same problems and constraints they had, just that I know I've considered similar solutions. The big difference might be, on the third iteration I tend to land on embedding raft directly into the backend, because lots of the problems I look at tend to benefit from a in-order log that is fully replicated to a quorum of servers. But based on the post and efficiency they're going for backing up the WAL on sqlite may make alot of sense. Actually litestream looks like it might have some fairly similar properties, so might be something else I'd like to add to my toolbox. Although I wonder what sort of replication lag there might be to replicas when using s3 as the stable store.
- lmeyerov 5y ago
  There's a world of a difference of someone doing that for a Google scale thing with big teams supporting it, or say a core part of making the best wireguard product that only they can do, vs a system that could have been postgres and now when the authors leave for their own sqlite-by-mail startup or are partying at SREConf for the 'our n+1th attempt to not do postgres' and an outage causes an unnecessary and ungoogleable shitshow for their colleagues. A good engineer can build all sorts of surprising things. A senior one knows why they should reserve that for only the special bits. I like reading the first two types of articles -- we frequently give tech talks ourselves -- but for making boring systems weird, I get flashbacks of consultantware/resumeware/NIH that took ages to weed out. I try to give talks on say GPUs for weirdly shaped problems, not on why we celebrated when we finally had time to rip out mongo from where it didn't belong :)
- billywhizz 5y ago
  the live replication feature litestream is implementing won't be using s3 - it will be direct node to node replication. hasn't been released yet and currently s3 is used for a cold backup that can be quickly restored to a fresh node as far as i know. https://github.com/benbjohnson/litestream/issues/8 https://github.com/benbjohnson/litestream/issues/8
hokus_p2 5y ago

bradfitz 5y ago

Tailscale engineer here. > Using a text file and ... > It seems like something you'd do as a proof of concept I mean, using a text file for the proof of concept is exactly what happened. And then it grew too quickly and had to get off of it eventually, but we always knew that. We were just amazed how long that worked. It survived much longer than we'd thought.

mardifoufs 5y ago

Aeolun 5y ago

> why would an engineer want to join a company that is making these decisions? Hmm, I can’t quite articulate why, but I would. Something about having the courage to be different.

fiddlerwoaroof 5y ago
Yeah, it’s perfectly reasonable to use a text file instead of a database until you find evidence that you need a “real” database.
dave78 5y ago
I would actually love to work at a company like that. I've been a professional developer for over 20 years now and at least in my experience, too much unneeded complexity has been the root cause of so many defects and product failures. I'm constantly fighting battles to keep things simple. All the young devs on my team want to use every fancy new technology that passes by, but I just want things to work for my customers with the smallest amount of downtime and late night pages. If a single JSON file worked at the time, then that's clearly all they needed. It certainly hasn't seemed to limit their success.
- billywhizz 5y ago
  > All the young devs on my team want to use every fancy new technology that passes by, but I just want things to work for my customers with the smallest amount of downtime and late night pages. i feel your pain dave. resume driven development and FAANG envy seem to be the dominant characteristics of far too many folks in the industry these days.
- CRConrad 5y ago
  > I'm constantly fighting battles to keep things simple. All the young devs on my team want to use every fancy new technology that passes by... For me, it's not so much my fellow developers -- half my age, but quite bright enough to realise the benefits of simplicity -- but that the vendors of the tools our corporation has decided to use have in turn decided to implement them not just as a spaghetti-reinforced mudball, but as a spaghetti-reinforced mudball of spaghetti-reinforced mudballs. (That these SRMB^2s are then just part of a larger spaghetti-reinforced mudball of SRMB^2s, i.e a spaghetti-reinforced mudball cubed, is of course not the vendors' fault but corporate. But then, aren't corporate systems always SRMBs? So if at least the components were simple, the end result would be only an SRM to the power of one, not three... Sigh.)

matthewmacleod 5y ago

Using a text file and using etcd as a DB for a production system seems like a terrible engineering decision I'd say this could easily be a great engineering decision if it meets the needs of an organisation in the moment. The thing with any engineering decision is that it involves a bunch of trade-offs – doing something that you know will hit a scaling wall can be worth it, if it can get you moving faster. There will never be a perfect solution, and in these decisions I find that simplicity is often a massively under-rated benefit. I'd even venture to say that I've seen more organisational failures as a result of optimistic over-engineering than I have optimistic under-engineering. That includes many of my own projects – ones where I've gone "oh I guess I'll do it properly this time" and then found myself six months down the line throwing out a ton of code, schema, or infrastructure I spent ages on because some requirements changed or some assumptions didn't pan out. It's hard to remember that lesson sometimes and I've been doing this for 15 years. It's like a company I used to work for where the whole product was driven by a massive semi-structured graph database – but instead of being implemented like that it was a big ball of MongoDB JSON documents glued together with PHP by someone who was more of a domain expert than an engineer. Often pretty sketchy to work on, but an engineering decision that was a success – in the sense that it allowed that product to exist at all where it otherwise might not have. why would an engineer want to join a company that is making these decisions? Why would the company want their users to know about these decisions? Sign me up as both an engineer and a user. It is extraordinarily refreshing to hear "we tried a bunch of different stuff, some of it was dumb, and this is where we're at now."

kevin_nisbet 5y ago

> But I'm just confused, why would an engineer want to join a company that is making these decisions? Why would the company want their users to know about these decisions? This might actually be a really good filter for which types of engineers are good fit for their company. Because my read was very different than yours, as I looked at the decision lineage and thought to myself I could see myself making every one of those decisions. I of course can't say I would've with the same problems and constraints they had, just that I know I've considered similar solutions. The big difference might be, on the third iteration I tend to land on embedding raft directly into the backend, because lots of the problems I look at tend to benefit from a in-order log that is fully replicated to a quorum of servers. But based on the post and efficiency they're going for backing up the WAL on sqlite may make alot of sense. Actually litestream looks like it might have some fairly similar properties, so might be something else I'd like to add to my toolbox. Although I wonder what sort of replication lag there might be to replicas when using s3 as the stable store.

lmeyerov 5y ago
There's a world of a difference of someone doing that for a Google scale thing with big teams supporting it, or say a core part of making the best wireguard product that only they can do, vs a system that could have been postgres and now when the authors leave for their own sqlite-by-mail startup or are partying at SREConf for the 'our n+1th attempt to not do postgres' and an outage causes an unnecessary and ungoogleable shitshow for their colleagues. A good engineer can build all sorts of surprising things. A senior one knows why they should reserve that for only the special bits. I like reading the first two types of articles -- we frequently give tech talks ourselves -- but for making boring systems weird, I get flashbacks of consultantware/resumeware/NIH that took ages to weed out. I try to give talks on say GPUs for weirdly shaped problems, not on why we celebrated when we finally had time to rip out mongo from where it didn't belong :)
billywhizz 5y ago
the live replication feature litestream is implementing won't be using s3 - it will be direct node to node replication. hasn't been released yet and currently s3 is used for a cold backup that can be quickly restored to a fresh node as far as i know. https://github.com/benbjohnson/litestream/issues/8 https://github.com/benbjohnson/litestream/issues/8

hokus_p2 5y ago