3 ms·

"This didn’t exactly come as a surprise to us, as for the entirety of Kagi’s life so far we have actually used the cheapest, single-core database available to u

"This didn’t exactly come as a surprise to us, as for the entirety of Kagi’s life so far we have actually used the cheapest, single-core database available to us on GCP!" Outages suck, but I love the fact that they are building such a lean product. Been paying for Kagi as a part of de-Google-ifying my use of online services and the experience so far (I wasn't impacted by this outage) has been great. A few years ago I built a global SaaS (first employee and SWE) in the weather space which was backed by a single DB, and while it had more than just 1 core (8 when I left from memory), I think a lot of developers reach for distributed DBs far too early. Modern hardware can do a lot, products like AWS Aurora are impressive, but they come with their own complexities (and MUCH higher costs).

z64 3y ago
Very cool! The deluge of cloud solutions are absolutely one of those things that can be a distraction from figuring out "What do I actually need the computer to do?". Internally I try to promote solutions with the fewest moving cloud-parts, and channel the quiet wisdom of those running services way larger than ours with something like an sqlite3 file that they rsync... I know they're out there. Not to downplay the feats of engineering of huge distributed solutions, but sometimes things can be that simple!
- layoric 3y ago
  Distracting is right! I watched your CrystalConf video and was happy to see the familiar Postgres + Redis combo :). I remember worrying about running out of legs with Redis (being single threaded), but with a combo of pipelining and changing the data structures used it ended up being the piece of infra that had the most headroom. Monitoring was probably the biggest value for outsourcing to another SaaS. I used Runscope, AWS dashboards and own elasticsearch and it was pretty cost effective for the API that was doing ~2M API calls a day. The other risk of cloud solutions is the crazy cost spikes. I remember consulting a partner, much larger, weather company on reducing their costs of a similar global weather product where they chose AWS DynamoDB as their main datastore, and their bill just for DynamoDB was twice our companies cloud bill. All because it was slightly "easier" to not think about compute requirements! Any ways, thanks for the postmortem, hopefully your channeling of quiet wisdom continues to branch out to others! :)

z64 3y ago

Very cool! The deluge of cloud solutions are absolutely one of those things that can be a distraction from figuring out "What do I actually need the computer to do?". Internally I try to promote solutions with the fewest moving cloud-parts, and channel the quiet wisdom of those running services way larger than ours with something like an sqlite3 file that they rsync... I know they're out there. Not to downplay the feats of engineering of huge distributed solutions, but sometimes things can be that simple!

layoric 3y ago
Distracting is right! I watched your CrystalConf video and was happy to see the familiar Postgres + Redis combo :). I remember worrying about running out of legs with Redis (being single threaded), but with a combo of pipelining and changing the data structures used it ended up being the piece of infra that had the most headroom. Monitoring was probably the biggest value for outsourcing to another SaaS. I used Runscope, AWS dashboards and own elasticsearch and it was pretty cost effective for the API that was doing ~2M API calls a day. The other risk of cloud solutions is the crazy cost spikes. I remember consulting a partner, much larger, weather company on reducing their costs of a similar global weather product where they chose AWS DynamoDB as their main datastore, and their bill just for DynamoDB was twice our companies cloud bill. All because it was slightly "easier" to not think about compute requirements! Any ways, thanks for the postmortem, hopefully your channeling of quiet wisdom continues to branch out to others! :)