15 ms·
A database for 2022
- infogulch 5y agoThey're using litestream [1] to replicate a SQLite database, which streams additions to the SQLite WAL file to object storage and can replay it back. This is fairly hands-off from SQLite's perspective. There's also the SQLite Session Extension [2] that is a built-in way to support generating and applying "patches". I'm curious how these tools will mature, it seems like a good match for microservices. [1]: https://github.com/benbjohnson/litestream https://github.com/benbjohnson/litestream [2]: https://www.sqlite.org/sessionintro.html https://www.sqlite.org/sessionintro.html
- Xeoncross 5y agoThe litestream project was created by https://github.com/benbjohnson https://github.com/benbjohnson who wrote https://github.com/boltdb/bolt https://github.com/boltdb/bolt (a key value store) which has been instrumental (from my point of view) in the Go community as one of the original choices for an embedded database as it was not only fast, but had transactions with stable snapshots. It was used by https://github.com/blevesearch/bleve https://github.com/blevesearch/bleve, https://github.com/etcd-io/etcd https://github.com/etcd-io/etcd, and number of other projects. These days, https://github.com/dgraph-io/badger https://github.com/dgraph-io/badger is often favored because of it's improved write throughput.
- otterley 5y agoIt should be noted that BoltDB has been retired as it suffered from a number of congenital defects. It was implicated in the major Roblox outage that happened a couple months ago, and Consul doesn't use it anymore. https://news.ycombinator.com/item?id=30015913 https://news.ycombinator.com/item?id=30015913 (This comment is not to suggest that the Litestream project is of poor quality or anything like that.)
- benbjohnson 5y agoBoltDB author here. Just to clarify, the project was retired because of maintenance burden. CoreOS wanted to make some changes but I didn't have the bandwidth to test and maintain it all. Since this was before Go had good version management, we decided that they could fork the project as "bbolt" and users could move over as needed. They did a good job maintaining it so I eventually archived the original project.
- infogulch 5y agoAnother one of Ben's projects is on the front page right now: Postgres wire compatible SQLite proxy - https://news.ycombinator.com/item?id=30875837 https://news.ycombinator.com/item?id=30875837
- benbjohnson 5y agoBolt author here. Badger is a good database but it's mostly a tradeoff of write versus read throughput, specifically, range queries. Badger is an LSM so it doesn't perform as well (last time I checked) with iterating over ordered key/value pairs. LSMs have bloom filters to speed up point queries so those aren't as much of an issue.
- tapirl 5y agoIt is strange that, although https://github.com/syndtr/goleveldb https://github.com/syndtr/goleveldb is used in the go-ethereum project for years, it is mentioned rarely (comparing to other go dbs).
- Xeoncross 5y agoYes, it has it's own trade-offs, but is certainly a top-performer for certain workloads: https://github.com/smallnest/kvbench https://github.com/smallnest/kvbench
- silisili 5y agoI watched an interview of Mr. Hipp, creator of SQLite, that I can't find now but was pretty interesting. Aside from being way different than I expected for some reason, in a good way - very down to earth and friendly, he was asked specifically about that, and more or less answered that his job was to write a solid DB, and replication can be done elsewhere. That's a pretty honest answer, and looks like someone took up the challenge.
- qbasic_forever 5y agoHe was on an episode of the Changelog podcast in the fall and went into similar topics: https://changelog.com/news/RRAw/visit https://changelog.com/news/RRAw/visit It's a good listen.
- billywhizz 5y agoprobably this one? https://corecursive.com/066-sqlite-with-richard-hipp/ https://corecursive.com/066-sqlite-with-richard-hipp/
- ithkuil 5y ago> This gives you near real time backups (*or with a couple deft modifications, lets your app block at critical sections until the backup is done*) anyone has an example of how to do this?
- endisneigh 5y agoI'm so desperate for a SQL database where I can just put it on a bunch of commodity hardware via Docker, connect them up and never worry about this again. Ideally it'd monitor my queries and create indexes for me. pleeeeeeaaaaaaaaaaaassssseeeeeeeeee /dream FoundationDB is similar but you have to do so much yourself. It's more or less what I'm describing for a key value store though. Not sure why it's not more popular.
- database_guy 5y agoI lamented this exact situation in a post I made last week. I'm with you on this completely.
- Nican 5y agoI am having quite a good time with CockroachDB. It has been mostly set-and-forget, and it auto-balances, and does rolling upgrades pretty smoothly.
- btgeekboy 5y agoFoundationDB is an Apple thing, isn’t it? Might explain why. It feels kinda like they’ve put it out there and it just hasn’t had anyone with a marketing budget to push it along.
- database_guy 5y agoIt started as a proprietary database. Apple acquired it and then open sourced it eventually.
- AtlasBarfed 5y ago... this is just replication of a database? As in the full database per node/replica? Hasn't AWS's SQL db-as-a-services had this for years now?
- butlerm 5y agoPretty sure you could track down instances of this technology going back decades now. Nothing is new under the sun. Doesn't mean you can use it.
- detaro 5y agoI don't think they are claiming to have invented database replication, so what?
- ignoramous 5y agoTailscale mention they want to run tests locally, not deal with MySQL/Postgres ops overhead themselves, and also avoid vendor lock-in (rules out RDS / Aurora, PlanetScale, CockroachDB, YugaByte, Spanner et al). May be Oracle remains an option. ;)
- jlokier 5y agoI don't see how CockroachDB ends up in your list of DBs that lead to vendor lock-in or being unable to run a cluster locally, because it's open source and you can run your own cluster. Is there something about it I haven't spotted that puts it in that list?
- ignoramous 5y agoOps overhead would be my guess. Take a look at https://litestream.io/guides/systemd/ https://litestream.io/guides/systemd/ and compare that with CockroachDB's. In all fairness, litestream is no CockroachDB/PlanetScale/YugaByte; but why pay the ops overhead (and your lawyers to make sense of their "OSS" licenses) when litestream would do just as nicely. TFA goes (emphasis mine): Litestream..grabs a lock so that no other process can checkpoint. It then watches the WAL file and streams the appended blocks up to S3, periodically checkpointing the database for you when it has the necessary segments uploaded. ___This gives you near real time backups and lets you replay your database from S3 trivially..It's a great hack, and a small enough program that I could read my way through it entirely before committing to it.___
- timcavel 5y ago
- KwisaksHaderach 5y agoThe obvious candidates were MySQL (or one of its renamed variants given who bought it) or PostgreSQL, but several of us on the team have operational experience running these databases and didn’t enjoy the prospect of wrestling with the ops overhead of making live replication work and behave well. Other databases like CockroachDB looked very tempting, but we had zero experience with it. And we didn’t want to lock ourselves into a cloud provider with a managed product like Spanner. What about managed mysql/posgresql? no lock-in and installing them locally is trivial.
- butlerm 5y agoIf you install it locally, that is pretty much the definition of an unmanaged database. Other than by the user of course.
- alexjurkiewicz 5y agoYou can use a managed version of the database in production, and an "unmanaged" local Docker image during development.
- mrkurt 5y agoManaged DBs have tremendous lock in. Just try migrating off RDS with zero downtime. You can't, because they've "managed" your ability to configure external replicas. Then built a whole brittle data migration service that probably won't work for your DB.
- jeffffff 5y agothey've already had to migrate twice due to questionable technology decisions. having to migrate again due to needing to change cloud providers seems a lot less likely than having to migrate again because their outside the box technology choice didn't pan out.
- tptacek 5y ago"Questionable technology decisions". You're trying to dunk, but I don't think you understand where the hoop is. Their technology decisions have panned out Tailscale well. We should all be so fortunate. This isn't Twitter with the "fail whale"; the only reason you know about any of this stuff is because they wrote about it. They ran their entire service with a JSON file backend for 18 months, and switched from it to etcd without you even knowing about it.
- musjleman 5y agoSo to recap their timeline: * Instead of an actual database use a JSON file. * Write a blog post about how that didn't scale. * Instead of an actual database hand-roll something else. * Write a blog post about how that didn't scale. * Instead of a database with built-in replication which is tailor built for key-value storage use SQLite with a single table containing key-value pairs and some fresh glue to make it replicate. I'm sorry, but what the fuck? Just writing the blog posts alone might have consumed more work-hours than what it would have taken to set a database up in the first place. Are we supposed to expect another blog post somewhere around a year down the road titled "SQLite didn't scale" or "The new library for replication that was released a few months ago contained a bug that took our prod down"?
- srameshc 5y agoWhy not ? You can be a spectator to it watch them succeed or fail.
- Dylan16807 5y agoOkay, but this is a database now. Is there a reason the replication needs to be built-in?
- artificialLimbs 5y agoI thought replication was useful for backups or testing. Maybe other things. edit: I know practically nothing about db.
- deleted 5y ago[deleted]
- musjleman 5y agoSorry, I probably didn't phrase it very clearly. I don't think there is anything wrong with needing some external help for that, but the combination of their choices is IMO a bit bizarre: * SQLite which I mostly see as an embeddable database for applications, used on a server. * Single table with key-value pairs. * Having the tool for replication be a quite young project that's in active development.
- ramenmeal 5y agoI think I'm missing some context. Using a text file and using etcd as a DB for a production system seems like a terrible engineering decision. It seems like something you'd do as a proof of concept or side project. It's interesting that they're blogging about this, as if they're proud of it. I guess I'm just missing the point. This is their 3rd DB Migration, something that I prefer to avoid at all costs. I guess they have different priorities and values. But I'm just confused, why would an engineer want to join a company that is making these decisions? Why would the company want their users to know about these decisions? They started using etcd knowing that he would become a bottle neck, that should have been a non-starter, right?
- bradfitz 5y agoTailscale engineer here. > Using a text file and ... > It seems like something you'd do as a proof of concept I mean, using a text file for the proof of concept is exactly what happened. And then it grew too quickly and had to get off of it eventually, but we always knew that. We were just amazed how long that worked. It survived much longer than we'd thought.
- mardifoufs 5y ago
- Aeolun 5y ago> why would an engineer want to join a company that is making these decisions? Hmm, I can’t quite articulate why, but I would. Something about having the courage to be different.
- fiddlerwoaroof 5y agoYeah, it’s perfectly reasonable to use a text file instead of a database until you find evidence that you need a “real” database.
- dave78 5y agoI would actually love to work at a company like that. I've been a professional developer for over 20 years now and at least in my experience, too much unneeded complexity has been the root cause of so many defects and product failures. I'm constantly fighting battles to keep things simple. All the young devs on my team want to use every fancy new technology that passes by, but I just want things to work for my customers with the smallest amount of downtime and late night pages. If a single JSON file worked at the time, then that's clearly all they needed. It certainly hasn't seemed to limit their success.
- rudasn 5y agoThis is an April fool's joke, right? Edit: no it's not.. > Footnote: coworkers point out it’s April Fool’s today and request that I clarify this isn’t a joke. Joke’s on them: every day’s April Fool’s in the Tailscale Database Engineering department.
- deleted 5y ago[deleted]
- bradfitz 5y agoI suggested that footnote text just for these comments. :)
- deleted 5y ago[deleted]
- maximilianroos 5y agoI see a dozen comments here, all critiquing their proposal. Surely this is April Fools? > I'm sorry, but what the fuck? Just writing the blog posts alone might have consumed more work-hours than what it would have taken to set a database up in the first place. > Using a text file and using etcd as a DB for a production system seems like a terrible engineering decision. It seems like something you'd do as a proof of concept or side project. It's interesting that they're blogging about this, as if they're proud of it. > This is their 3rd DB Migration, something that I prefer to avoid at all costs. I guess they have different priorities and values. But I'm just confused, why would an engineer want to join a company that is making these decisions?
- jjeaff 5y agoThey put an update at the bottom of the article clarifying that it is not, in fact, a joke.
- norom 5y agoI for one am enjoying this. Sure they could probably just use a regular ol' database, but where is the excitement in that!
- jjeaff 5y agoI really hope they aren't taking the same tack when dealing with the encryption portion of their service.
- Aeolun 5y agoWhat would be the analogue? Using openssl with a wrapper service? Doesn’t seem particularly dubious to me.
- ignoramous 5y ago> Using openssl with a wrapper service? Coincidentally, one of tailscale's founders already built this: https://github.com/sshuttle/sshuttle https://github.com/sshuttle/sshuttle
- jjeaff 5y agoThey have been trying to create or reinvent the database rather than using tried and true solutions. The analogue would be rolling your own cryptography.
- acrispino 5y agoThey're using WireGuard for the tunneling. I don't think a lot of people will object
- mrkurt 5y agosqlite is, like, the regular-est of the databases.
- isbvhodnvemrwvn 5y agoNot quite, I got bitten by weak type system more than once.
- chubot 5y agoHonest question: what do the sqlite authors think of litestream? I seem to recall Richard Hipp on the "changelog" podcast mentioning it, but I don't remember what he said. 2 episodes here: https://changelog.com/person/drh/podcasts#feed https://changelog.com/person/drh/podcasts#feed I think it was very neutral, something to the effect of "there are multiple solutions". But I know essentially nothing about sqlite internals so I can't judge, or maybe that's why I didn't understand what he said. I guess I'm wondering if it's a recommended/supported mode of operation in sqlite. What are the failure cases? How much data can you lose?
- benbjohnson 5y ago> Honest question: what do the sqlite authors think of litestream? Litestream author here. Dr Hipp and his team reached out when I first released Litestream and we had a video call together. They were fantastic. Really friendly and down to earth. I explained how I put together Litestream and we went back and forth on different approaches. They were really helpful with understanding some of the shared memory stuff that's what makes the new read replication work. They ended the call by asking me if there's anything they could help with or anything I needed. > I guess I'm wondering if it's a recommended/supported mode of operation in sqlite. I don't think the SQLite team endorses any tooling outside of what they build AFAIK. So in that sense, no, it's not officially supported. However, the API for maintaining checkpointing is publicly available and there are docs for controlling it from outside processes (which is what Litestream does). > What are the failure cases? How much data can you lose? Litestream is designed so that it keeps retrying in the event that you can't connect to a replication destination. S3 is pretty reliable so it may be a network outage that could cut you off. You can also enable Prometheus metrics to be reported out of Litestream if you want to add monitoring and alerting. As far as data loss, by default it's setup to bundle database changes together every 1 second, compress them, and upload them to somewhere like S3. So your window for data loss is 1 second unless S3 goes down. You can also run regular backups with the SQLite CLI and upload those as a fallback. That's what I typically do since it's really cheap and easy to setup an hourly cronjob and I'm overly paranoid. :)
- infogulch 5y ago
- flatiron 5y agoHow did they think going from a single json to either MySQL or Postgres? I know they say this isn’t April fools but I’m not sure I would put it past the engineering team. This can’t be real. “We looked at real solutions like cockroach but nobody could be bothered to rtfm”
- eminence32 5y agoI'm sympathetic to the idea of avoiding a database because no one on your team has experience with it. On my old team, our product supported Oracle DB (along with some other databases), but on one on the team really was an oracle expert, and we would frequently run into questions that we didn't really know how to answer. We would have dropped it if we could. It's easy to say "rtfm" but doing this doesn't make you an expert in the system. And as I bet many NH readers know, there's a big difference between "I read the quickstart guide and got a DB working in an afternoon" and "We've hit an error an error that the manual says nothing about, but our DB expert saw this once, 7 years ago, and remembers how to recover"
- samhw 5y ago> "We've hit an error an error that the manual says nothing about, but our DB expert saw this once, 7 years ago, and remembers how to recover" Yup, this rings true. Once worked on a large Cassandra deployment – holding people's bank accounts, so, ya know, somewhat critical – and we eventually had to get one of the authors of Cassandra onto our on-call team. Even he was stumped by some of the bugs and failure modes. (And, incidentally, the word 'anticompaction' is still enough to trigger my PTSD.) The notion that the documentation is an accurate and exhaustive explanation of every single thing about a product is a junior engineer mistake that no one with rough ops experience would say. Nor, for that matter, anyone who's written a database that's in use (as I unfortunately have), or really any other piece of software. And the same goes for the notion that "everyone should go with MySQL or Postgres for everything because they are Proper Databases that are scalable and won't go down". If I were being charitable, I'd say it's a 'no one got fired for buying IBM' argument; if I were being honest, I'd say it puts me in mind of 'MongoDB is webscale' (https://www.youtube.com/watch?v=b2F-DItXtZs https://www.youtube.com/watch?v=b2F-DItXtZs). I don't know why bad engineers are so terrified of thinking for themselves specifically – well, generally, but especially – when it comes to databases. They aren't magical contraptions. They are just portable filesystem wrappers.
- brycelarkin 5y agoIt looks like you guys use AWS. Why not just use RDS if you don’t want to deal with database management?
- epolanski 5y agoSome engineering teams seem to take overcomplication always one step too far. It's very hard to estimate future work and overconfident engineers consistently downplay the costs.
- ignoramous 5y agoI'm not sure why you're being downvoted. You're right: running databases isn't for every team! Reach out for a managed database if you can. But tailscale isn't some random group of engs. They've probably got the chops to pull off literally anything they want to. I mean TFA casually mentions online cross-database transfers, multiple zero-downtime schema migrations, inspecting litestream's replication code for feasibility, deftly modifying sqlite WAL checkpoints... all in one breath.
- jeffffff 5y agoif they're this talented, is their time really best spent on dba work rather than improving the product?
- ignoramous 5y ago> is their time really best spent on dba work... It seems to me that tailscale engs want to avoid DBA work but also not use managed offerings, and so, they're comfortable paying the costs they have to (such as multiple migrations). > ...rather than improving the product? Well, you'd guess they want to be able to continually improve their already credible product too. When TFA points out that zero vendor lock-in and hassle-free, local end-to-end tests are non-negotiable, I think it is for this reason. ---- > if they're this talented, is their time really best spent on... From: https://tailscale.com/blog/go-linker/ https://tailscale.com/blog/go-linker/ "People are often surprised and sometimes horrified when they learn that Tailscale maintains its own fork of the Go toolchain. Tailscale is a small startup. Isn't that a horrible distraction, a flagrant burning of innovation tokens?" "Maybe. But the thing is, you write code with the engineers you have." "We had a problem: We kept crashing on iOS, and in addition to being awful, it was preventing us from adding features." "Another team might have decided to cut even more features on iOS to try to achieve stability, or limited in some way the size of the tailnet that iOS could interact with." "Another team might have radically redesigned the data structures to squeeze every last drop out of them." "Another team might have rewritten the entire thing in Rust or C." "Another team might have decided to accept the crashes and attempted to mitigate the pain by making re-establishment of connections faster." "Another team might have decided to just live with it and put their focus elsewhere." "The Tailscale team has Go expertise, spanning the standard library to the toolchain to the runtime to the ecosystem. It’s an asset, and it would be foolish not to use it when the occasion arises. And the fun thing about working on low level, performance-sensitive code is that that occasion arises with surprising frequency." "Blog posts about how people solve their problems are fun and interesting, but they must always be taken with a healthy dose of context. There may be no other startups in existence for which working on the Go linker would be a sensible choice, but it was for us."
- nosefrog 5y agoThere's a lot of negativity here, so I just wanted to comment that I think the approach Tailscale is taking with sqlite is pretty neat. MySQL and Postgres are both pretty complicated, so gluing together two simpler systems, sqlite and litestream, sounds appealing, and I'm interested to hear how it turns out in the long run.
- tptacek 5y agoThreads on HN are very sensitive to initial conditions. A very negative (and I think pretty facile) comment took an early lead here, most likely because of its snark/dunk value. Give the thread time, it'll even out.
- hokus_p2 5y ago
- deleted 5y ago[deleted]
- deleted 5y ago[deleted]
- bestinterest 5y agoAgreed, it's a bit of a shame for HN imo. I see a team who is being brilliant and understanding their domain down to the core with as minimal resources as they can get away with. There is incredible value in incrementally adding complexity only when needed.
- billywhizz 5y agotailscale = YAGNI to the MAX! i applaud their willingness to think outside the box 99% of devs seem to be happy to squeeze themselves into just because they read about it in a book by some self-appointed tech god.
- acrispino 5y ago
- deleted 5y ago[deleted]
- tptacek 5y agoThis thread has (at least at the moment) serious Bob Martin Sudoku Solver energy to it. Tailscale has solved an infamously complicated problem using, for the most part, simple tools. They're not just successful; they're remarkably successful, spookily successful, upsettingly successful. Consider whether the secret sauce here might not be au courant database choices, but rather something much harder for random teams to pick up and run with: just straight up good design and programming. This is a thing that happens in other fields, too; some of the hardest dishes to cook seem extremely simple and have few ingredients. They're hard because there's no place to hide: your technique has to be flawless or they don't work. A famous entrance exam for new cooks at fine dining restaurants is "make an omelet". Tailscale is, thankfully (and, to me, irritatingly), more like an omelet than a truffled hash.
- dastbe 5y agonobody goes to these restaurants for their omelets, though at the same time nobody should care how tailscale actually implements things so long as it works well. imo they’re not solving a notoriously hard problem, they’re realizing that they have the tiniest version of this problem (and that will most likely continue to be true). when that’s the case and you want to prioritize stuff that moves the needle like dev velocity, why do anything less simple? getting hung up on building scalable solutions far beyond what you need is somewhere between a fool and a big co employees game. far too often people introduce complexity way before it’s needed, and that almost always slows down your devs ability to actually fix bugs and make the product more reliable.
- tptacek 5y agoThey don't, but there are other "finesse" dishes that people do in fact go to restaurants for that are notable for their simplicity, and the fact that you can only pull them off if you have flawless technique. Japanese, in particular, is notorious for this.
- jimbokun 5y agoThere’s a manga about a food critic (Oishinbou [1]), where he takes another food critic to a hole in the wall Tokyo restaurant as an example of elite cooking. The chef-owner brings him his meal. Which is just a piece of cooked fish, rice, and miso soup. The critic is first almost insulted by such a simple meal. But then realizes the fish, rice and soup are prepared to perfection, and is amazed at the chef’s skill. [1] https://en.wikipedia.org/wiki/Oishinbo https://en.wikipedia.org/wiki/Oishinbo
- rablackburn 5y agoI loved this article, despite the negativity. A discussion that keeps recurring on HN is how you can get a lot done with simple, boring solutions. This is a perfect example of that in practice. What’s the simplest thing that’ll work? Do that, monitor the solution, and when you start hitting a limitation reevaluate what the next appropriate solution is. There are plenty of comments here deriding this as insanity when we have established best practices, but “best practices for me are not necessarily best practices thee”. People aren’t perfect, not every engineer comes fully-formed grokking the industry’s best solutions and their trade offs. This is what learning looks like. This is what an evolving system looks like. And this is how valuable software gets built every day. Who gives a fuck about scalability on a poc? “Let’s just put in a json file”. I Love it. Personally I start at the “use SQLite” step and go from there because SQLite is such amazing software. Thanks to this post I’m excited to try out Lifestream. You might rightly point out that Tailscale isn’t a side-project/poc anymore. But that’s exactly why they’re changing to SQLite and writing this post; this is their team learning to deal with larger problems and changing needs, and they’re sharing that journey with us. As long as they’re not _ignoring_ serious risks let them experiment. But try not to get too upset if they’re making different risk/effort trade-offs than you would. Even if it’s “objectively” the “wrong” thing. My rubric for technical sanity is: - are you taking (and testing) backups? - is your infra/data secure? - are you monitoring what’s happening? - can you recover from catastrophic failure in a acceptable time frame? - are you meeting your legal obligations? Cover those needs and you can be forgiven a lot suboptimal/experimental implementation details.
- ramraj07 5y agoThe question is, is this the simple solution? Perhaps I and others have been spoiled by cloud offerings but having set up RDS once and never having to worry much about the db architecture after that for years as we scale has maybe spoiled us.
- rablackburn 5y agoI’ll be honest and admit my own bias to cheaper, self-managed tooling. I’ll admit that RDS definitely meets the criteria of simple. From what I understand RDS works great and you’re happy to pay what feels like the tiny cost…right up until it doesn’t anymore. You’re paying Amazon a premium for them to manage the tedious stuff. But sometimes dealing with the tedious is what teaches you how the stack works and makes you jmuch more prepared to solve those more complex problems down the road. The uncomfortable fact is businesses finding themselves in situations where they experience vendor-lock-in just because they’ve let their capability wither away. Clearly David’s point about avoiding MySQL and PostgreSQL due to fears of “vendor lock-in” resonates with me, even if I personally think that’s not quite where the threat is. …for Postgres anyway. I’ve heard enough stories that I never want to touch an oracle product in my career.
- dang 5y agoRelated: Why I Built Litestream - https://news.ycombinator.com/item?id=26103776 https://news.ycombinator.com/item?id=26103776 - Feb 2021 (176 comments)
- 0xFFFE 5y agoTotally off topic. I am not an UI/UX designer but why can't the page scale to the size of the screen? Sure I could ctrl/cmd++ but is it hard to detect the screen size? Genuine question, not trying to be condescending.
- deleted 5y ago[deleted]
- dahfizz 5y agoThis should be a new logical fallacy: appeal to current year. Every time I hear someone say "it's $CURRENT_YEAR, we obviously need to implement $PARTISAN_POLICY" I wonder about what fringe politics will be "obviously correct" in 3022.
- deleted 5y ago[deleted]
- AnimalMuppet 5y agoSeconded. It's a logical fallacy because they are using the year as proof that we should adopt their policy. Except the year only proves what year it is - nothing else.
- billywhizz 5y agooh boy. is nobody allowed to make a little joke anymore. you should take yourself a lot less seriously my dude.
- AnimalMuppet 5y agoFirst: I am not "my dude" to you. And, completely apart from the database, it's a really valid point - not a joke at all. When someone says, "It's 2022. Why are we still arguing about X?" or some such, it means that they haven't been able to persuade the rest of the world about X on the merits of their argument, and so they want to short-circuit the argument by appearing "modern" or "up to date" or "on the right side of history". They want to win by persuading the other side to give up in embarrassment, not by virtue of the quality of their position. (In their own eyes, their position is probably self-evidently the only right and moral one, but that doesn't tell us much.) It's a way of trying to win by default, and it deserves nothing other than being regarded as "you lose by default".
- billywhizz 5y ago
- mathgladiator 5y agoI believe using a large JSON file is not half bad, but you do run into the problem of how do you index and query it in meaningful. I actually ran into this problem when building a game because a document doesn't provide a great model.. The relational model is... JUST... SO... GOOD. And, it is a shame that most of the relational systems are so complicated. A document within Adama (https://www.adama-platform.com/ https://www.adama-platform.com/) is basically a giant JSON file held within memory with clients connected via a WebSocket. I'm basically building my own indexing since I want the indexing to be reactive, and I've got exceptionally fun optimization problems. Litestream, in my opinion, is a great way to get started. Actually, it's beyond fantastic if you maintain the 1 server to 1 database because migrations become so easy. So, I applaud the team for taking this simple approach.
- tptacek 5y agoIf you have a "single source of truth", sqlite/Litestream seems like a pretty-near-optimal way of taking advantage of the relational model while keeping design simplicity. I don't know what the higher-level architecture of Tailscale is, but we have the same problems; we have a complicated "single source of truth" that takes the form of a Consul cluster, but sqlite makes an absolute ton of sense for us, because we can condense the Consul cluster down to a SQL schema and then ship it around our fleet. (we don't use Litestream right now; we're pushing the complexity up a layer in our design instead) A lot of people on this thread are looking down their noses at sqlite, but I think they're kind of beclowning themselves; the unreasonable effectiveness of sqlite has been a meme in infra dev for a couple years now. It's not a new idea. Lots of people are doing stuff like this. The funniest bit on this thread is the person saying they should use RDS, as if their infrastructure was just a big Rails app.
- statictype 5y agoIt would be interesting to known why a standard boring RDS setup wouldn’t solve their problem completely. In fact I would be more interested to understand that than the actual details of sqlite tailing. (I think the reason they gave was vendor lock-in, but apart from that, I didn’t understand why it wouldn’t be adequate)
- tartakovsky 5y agoApril fool’s?
- didip 5y agoSo… litestream is bidirectional? Nothing can go wrong when daemon A and daemon B write almost at the same time? Edit: I just found this tips and gotcha page: https://litestream.io/tips/ https://litestream.io/tips/
- benbjohnson 5y agoLitestream author here. I'm glad you were able to track down what you were looking for. For posterity, I'll add a response here. Litestream is a one-way, physical replication tool so you have a single primary node that continuously copies changes out to an external destination (S3, SFTP, etc). If you're looking to have multiple primaries then you can try using the SQLite session extension[1] and copying patch files. That's quite a bit more complicated though. [1]: https://www.sqlite.org/sessionintro.html https://www.sqlite.org/sessionintro.html
- bArray 5y ago(Ignoring the fact this may be an April fools joke.) > We’re going to put everything in a single file on disk again. For most small scale projects I'm just reading and writing lines to a file on disk. Sure, I could over-engineer some database, or I could get a minimal viable project off of the ground and worry about scaling later. Most projects run into issues long before scalability. I found myself recently wanting a lightweight key-value store for random structures. For this purpose, I created a toy single-file key-value store [1] in less than 256 lines of C. It's not quite ready for prime time, but it's incredibly easy to use. [1] https://gitlab.com/danbarry16/u-database https://gitlab.com/danbarry16/u-database
- ctopher7 5y agoa little bit oot, but just installed tailscale on my dev stack (ec2, bare metal at home and my macbook air). mindblown by its simplicity and "it just works". Use it for logging purposes (ELK is too heavy for free tier ec2) and CI/CD (again, too heavy for my free tier ec2)
- Razengan 5y agoWho would use it if it’s just for one year?
- amithgeorge 5y agoMaybe I missed something. How do you account for the VM downtime? The VM containing the service writing to the sqlite db, it can go down without warning. How do you mitigate that?
- nikita2206 5y agoIt doesn’t. Given that it periodically dumps the WAL to an S3 bucket for actual storage this sounds like async replication. Async replication is prone to lose last bits (last by time) of data when the node goes down. They’re probably more or less fine if the SQLite file is stored on the EBS though, it means they’ll just have to restart that node that failed or reattach EBS volume to a new healthy node. It does mean downtime though. On another note I’m using Tailscale and it’s a wonderful service and I get it for free.
- coderdd 5y agoI was pondering about using simple decentralized databases that are kept synchronized asynchronously. I arrived to a design where SQLite databases would be synchronized by Kafka. Kafka is really robust, and has a friendly semantics when configured to be in-order delivery. The catch is, you don't issue writes to SQLite anymore. You write Kafka messages, and have to prepare to resolve concurrency problems at read+write-to-db time. For example, user registration is not an atomic action anymore - you write a registration attempt message, and then it might race with a concurrent registration attempt from an other node (say due to retries and random load balancing). You resolve the race in message-reception time (doable due to stable order on multiple reader nodes). I expect this would work nice, but needs you to rethink all actions that used to be synchronous db writes.
- lifty 5y agoThere are more robust solutions out there for this architectural pattern. Look into dolt(dolthub) and noms.
- jmconfuzeus 5y agoThat was a fun read but I'm missing something here. I've never worked on large scale systems and I don't understand why they didn't choose My or Postgre SQL. Some comments say that it's because they're complicated but what exactly is complicated about them? Can someone with more experience explain to me what would have happened if, for example, they used PostgreSQL from the beginning? P.S Apologies if this is a stupid question. Edit: Looks like I hurt someone's feelings in this comment.
- hokus_p2 5y ago
- billywhizz 5y agoi don't think it's a stupid question and not sure why you getting downvoted. the earlier blog post from tailscale goes into more detail about how the system in question works: https://tailscale.com/blog/an-unlikely-database-migration/ https://tailscale.com/blog/an-unlikely-database-migration/ > Tailscale’s coordination server, our “control plane”, has become known as CONTROL. It’s currently a single Go process on a single VM. if they had used postgres/mysql or some other external RDBMS then that would have meant managing a whole other set of dependencies and doing at least an IPC/Pipe roundtrip for every single query instead of being able to use an embedded database like custom-built JSON db or SQLite which is only going to need in-process syscalls for reading/writing to disk/cache. so, this allows them to scale their queries vertically orders of magnitude more than an external db (SQLite can do an insane amount of simple read queries per second) as well as making their whole dev/test/release cycle simpler and easier to manage. that's my understanding anyway - hopefully it makes some sense.
- SodiumMerchant0 5y ago
- bob1029 5y agoThis is great. We've been using sqlite in production as our exclusive database storage solution for over half a decade. For non-trivial, multi-user systems too. Replication of data has always been a concern for us, with the current mitigation being periodic VM snapshots. In order to engage larger customers, we would need to tighten this up. Our next gen data storage technology leverages some dark magic from fintech. Append-only log, single writer, synchronous replication, etc. Most recent benchmarks with replication to 1 sync witness on same lan are in excess of 200k business objects created or updated per second.
- alophawen 5y agoNot commenting on tailscale, but for the state of databases. Sometimes boring is the right choice. PostgreSQL has worked for decades now, and seems to have regained much of the performance that MySQL once boasted. If you do this for the money, investing in tried and true (but boring) software should be the default solution. I watched with dread how the MongoDB fiasco played out a decade ago. Meanwhile, I kept using PostgreSQL and had a good ride.
- kfk 5y agoIt depends. I was in a demo call from Blaze, a CDP (Customer Data Platform), Segment.io competitor. Their underlying db seems to be MongoDB and for a CDP makes a lot of sense. In a CDP you collect all kind of data on n hard ids that identify 1 person interacting with your web assets, schemaless and json first is a lot easier to reason in this context than SQL. This is because you keep enriching your profile with additional attributes over time. Can PostgreSQL do this? Absolutely, but it’s not its main feature. Things are complex, especially in data, I think it’s important to evaluate use cases rather than going with generic assumptions like “use boring tech”.
- louwrentius 5y agoYou say 'sometimes', I think it should be the default position. When you want to deviate from the boring software and technology, there must be a very sound 'business case' for it. If you are building a planetary scale business, OK, maybe that warrants something more fancy. But how many companies really need that? I'm not limiting myself to the HN unicorns, but looking at the whole market.
- alophawen 5y agoI agree with you and I also wrote: > If you do this for the money, investing in tried and true (but boring) software should be the default solution. Meaning if you are just playing around, then by all means evaluate new and shiny stuff.
- unfocussed_mike 5y agoYep. IMHO MySQL becoming boring (fast, pretty reliable, non-compliant quirks being ironed out, decent backup tool availability, credible admin tools, JSON and CTE support) is one of the most important long term trends on the internet. Along with the parallel evolution of PHP.
- deleted 5y ago[deleted]
- thrower123 5y agoI'm getting half-remembered flashbacks to some of the distributed, self-replicating Lotus Notes applications I worked on very early in my career
- rr808 5y agoI loved filed based h2 as well. Is there any reason in 2022 to use h2 over sqlite?
- jhgb 5y agoProbably not if you can just jump to a single-file Firebird database, and open that one either in embedded (single-user) or server (multi-user) mode.
- throwdbaaway 5y agoIt has been almost 15 months since the previous blog post: > Unfortunately, if we want to move to running more than one CONTROL process simultaneously for high-availability and better release management, that means we no longer have exactly one process managing the data Do you guys still only have 1 CONTROL process?
- kall 5y agoAt first I was also a little confused by this architecture. I think that was because I was missing a clear picture of what they want to do. I think it is: - one leader that can write to the db - a ton of other hosts that can read from the "streamed" db and want to do so often/fast - hosts send their writes (if they have any?) to the leader via some api call (?) Is that right? If so, the solution makes a ton of sense. "Just use pg/MySQL" in that case would mean hundreds (thousands?) of read replicas, which doesn't sound fun, or every read over the network.
- tptacek 5y agoIt's replacing etcd, which has essentially that model.
- petethepig 5y agoI set up a similar system in the past with redis replicas — it was about 200 MBs of data that I needed to have on quick access to on ~ 300 servers around the world — it was easy to set up and worked really well.
- AtNightWeCode 5y agoApril fools joke? I do not get it.
- stavros 5y agoI love this. If it works, keep it until shortly before it stops working. The older I get the more love I have for simple solutions that are easy to work with (and change when you've grown out of them). These days I wince when I hear of overcomplicated solutions for an MVP that "will scale" at a nebulous point in the future, while we're going to be paying the complexity penalty every day for a benefit that may never come.