12 ms·

Post-mortem for last week's incident at Kagi

346 points by leetrout 3y ago

vfxdev 3y ago
This incidcent also shows that no security audit was ever performed on this platform, as a DoS vulnerability would've been one of the first findings there. Makes you wonder what other vulnerabilities are in their infrastructure...
deleted 3y ago
[deleted]
renewiltord 3y ago
Interesting. The classic problem. You offer to not meter something and then someone will use it to max capacity. Then you’re forced to place a limit so that one user won’t hose everyone else.
- fotta 3y ago
  > We were later in contact with an account that we blocked who claimed they were using their account to perform automated scraping of our results, which is not something our terms allow for. I mean beyond that it was a user that was violating the TOS. This isn’t really a bait and switch scenario (although it could be reasonably construed as such).
- JumpCrisscross 3y ago
  > you’re forced to place a limit so that one user won’t hose everyone else Soft limits at the tail of usage comports with the term unlimited as it's commonly used. For Kagi, a rate limit derived from how quickly a human can type makes sense.
fwsgonzo 3y ago
It wasn't that long ago that I had heard about Kagi the first time. Now I use it every day, and the fact that I can pin cppreference.com to the top is just such a boon.
boomboomsubban 3y ago
One user running a scraper took the service down for seven hours? I know it's easy to sit on the outside and say they should have seen this coming, but how does nobody in testing go "what happens if a ton of searches happen?"
- z64 3y ago
  Hi there, this is Zac from Kagi. I just posted some other details here that might be of interest: https://news.ycombinator.com/item?id=39019936 https://news.ycombinator.com/item?id=39019936 TL;DR - we are a tiny, young team at the center, and everyone has a closet full of hats they wear. No dedicated SRE team yet. > "what happens if a ton of searches happen?" In fairness, you can checkout https://kagi.com/stats https://kagi.com/stats - "a lot of searches" is already happening, approaching 400k per day, and systems still operate with plenty of capacity day-to-day, in addition to some auto-scaling measures. The devil is in the details of some users exploting a pathological case. Our lack of experience (now rightfully gained) is knowing what organic or pathological traffic we could have predicted and simulated ahead of time. Load-simulating 20,000 users searching concurrently sounds like it would have been a sound experiment early on, and we did do some things resembling this. But considering this incident, it still would not have caught this issue. We have also had maybe 10 people run security scanners on our production services at this point that generated more traffic than this incident. It is extremely difficult to balance this kind of development when we also have features to build, and clearly we could do with more of it! As mentioned in my other post, we are looking to expand the team in the near term so that we are not spread so thin on these sorts of efforts. There is a lot that could be said in hindsight, but I hope that is a bit more transparent WRT how we ended up here.
  - smcleod 3y ago
    Zac, I think you’re doing great handling and communicating this. Keep up the great work and have fun learning while you’re at it!
  - rconti 3y ago
    What does "pathological" mean in this context?
fragmede 3y ago
That speaks volumes about the observability they have of their internal systems. It's easy for me to say they should have seen it sooner, but the right datadog dashboards and splunk queries should have made that clear as day much faster. Hopefully they take it as a learning experience and invest in better monitoring.
- z64 3y ago
  Hi there, I'm Zac, Kagi's tech lead / author of the post-mortem etc. This has 100% been a learning experience for us, but I can provide some finer context re: observability. Kagi is a small team. The number of staff we have capable of responding to an event like this is essentially 3 people, seated across 3 timezones. For myself and my right-hand dev, this is actually our very first step in our web careers - this is to say that we are not some SV vets who have seen it all already. To say that we have a lot to learn is a given, from building Kagi from nothing though, I am proud of how far we've come & where we're going. Observability is something we started taking more seriously in the past 6 months or so. We have tons of dashboards now, and alerts that go right to our company chat channels and ping relevant people. And as the primary owner of our DB, GCP's query insights are a godsend. During the incident both our monitoring went off, as well as query insights showing the "culprit" query - but, we could have monitoring in the world, and still lack the experience to interpret it and understand what the root cause is or most efficient action to mitigate is. In other words, we don't have the wisdom yet to not be "gaslit" by our own systems if we're not careful. Only in hindsight can I say that GCP's query insights was 100% on the money, and not some bug in application space. All said, our growth has enabled us to expand our team quite a bit now. We have had SRE consultations before, and intend to bring on more full or part-time support to help keep things moving forward.
  - ijhuygft776 3y ago
    Kagi, an efficient company. Thanks Zac and the rest of the team
  - jjtheblunt 3y ago
    I bet a silent majority are thinking "well done, Zac, all the same".
layoric 3y ago
"This didn’t exactly come as a surprise to us, as for the entirety of Kagi’s life so far we have actually used the cheapest, single-core database available to us on GCP!" Outages suck, but I love the fact that they are building such a lean product. Been paying for Kagi as a part of de-Google-ifying my use of online services and the experience so far (I wasn't impacted by this outage) has been great. A few years ago I built a global SaaS (first employee and SWE) in the weather space which was backed by a single DB, and while it had more than just 1 core (8 when I left from memory), I think a lot of developers reach for distributed DBs far too early. Modern hardware can do a lot, products like AWS Aurora are impressive, but they come with their own complexities (and MUCH higher costs).
- z64 3y ago
  Very cool! The deluge of cloud solutions are absolutely one of those things that can be a distraction from figuring out "What do I actually need the computer to do?". Internally I try to promote solutions with the fewest moving cloud-parts, and channel the quiet wisdom of those running services way larger than ours with something like an sqlite3 file that they rsync... I know they're out there. Not to downplay the feats of engineering of huge distributed solutions, but sometimes things can be that simple!
  - layoric 3y ago
    Distracting is right! I watched your CrystalConf video and was happy to see the familiar Postgres + Redis combo :). I remember worrying about running out of legs with Redis (being single threaded), but with a combo of pipelining and changing the data structures used it ended up being the piece of infra that had the most headroom. Monitoring was probably the biggest value for outsourcing to another SaaS. I used Runscope, AWS dashboards and own elasticsearch and it was pretty cost effective for the API that was doing ~2M API calls a day. The other risk of cloud solutions is the crazy cost spikes. I remember consulting a partner, much larger, weather company on reducing their costs of a similar global weather product where they chose AWS DynamoDB as their main datastore, and their bill just for DynamoDB was twice our companies cloud bill. All because it was slightly "easier" to not think about compute requirements! Any ways, thanks for the postmortem, hopefully your channeling of quiet wisdom continues to branch out to others! :)
jacob019 3y ago
If you're listening, Kagi, please add an à la carte plan for search. Maybe hide it behind the API options as not to disrupt your normal plans. I love the search and I'm happy to pay, but I'm cost sensitive now and it's the only way that I'm going to feel comfortable using it long-term.
- spiderice 3y ago
  They have a $5 for 300 searches option. Is that not what you're referring to?
  - JumpCrisscross 3y ago
    > Is that not what you're referring to? It sounds like they'd prefer a 2¢ per search option via API.
    - jacob019 3y ago
      Yes, exactly. Or 5 cents for the first 100 searches, 2 cents after, something like that.
      - binsquare 3y ago
        They do have the FastGPT api which is approx 1.5cents per api call
        lostlogin 3y ago
        So raise this to 5c for first 100 and 2c thereafter and we have a happy customer?
        jacob019 3y ago
        Not search.
  - BeetleB 3y ago
zilti 3y ago
[flagged]
- HaZeust 3y ago
  I'll bite; how do you figure?
  - zilti 3y ago
    GCP prices are ludicrously high. You're far better off to just get a VPS and self-host Postgres and your webserver. Which is what we did at my last startup; only thing we used GCP for was build servers, and only because we got a ton of free credits. We could've gotten a paid intern for the money we saved.
smcleod 3y ago
Good write up. I always appreciate Kagi’s honesty and transparency like this. Great product, great service.
muhammadusman 3y ago
I was one of the users that went and reported this issue on Discord. I love Kagi but I was a bit disappointed to see that their status page showed everything was up and running. I think that made me a bit uneasy and it shows their status pages are not given priority during incidents that are affecting real users. I hope in the future the status page is accurately updated. In the past, services I heavily rely on (e.g. Github), have updated their status pages immediately and this allows me to rest assured that people are aware of the issue and it's not an issue with my devices. When this happened with Kagi, I was looking up the nearest grocery stores open since we were getting snow later that day so it was almost like I got let down b/c I had to go to Google for this. I will continue using Kagi b/c 99.9% of the other time I've used it, it has been better than Google but I hope the authors of the post-mortem do mean it when they say they'll be moving their status page code to a different service/platform. And thanks again Zac for being transparent and writing this up. This is part of good engineering!
- Terretta 3y ago
  > In the past, services I heavily rely on (e.g. Github), have updated their status pages immediately Also in the past, other times GitHub has not updated its status page immediately.
- Neikius 3y ago
  Microsoft is notorious for their lax status page updates...
  - wiml 3y ago
    Is there anyone who isn't?
- lambdaba 3y ago
  I'm only replying to the praise here - I too, although I haven't fully switched, had a very enticing moment with Kagi when it returned a result that couldn't even be found on Google at any page in the results. This really sold me on Kagi and I've been going back and forth with some queries, but I have to say that between LLMs, Perplexity, and Google often answering my queries right on the search page, I just don't have that many queries left for Kagi. If Kagi would somehow merge with Perplexity, now that would be something.
blantonl 3y ago
At first, by what turned out to be a complete coincidence, the incident occurred at precisely the same time that we were performing an infrastructure upgrade to our VMs with additional RAM resources I can assure you that these "coincidences" happen all the time, and will cause you to question your very existence when you are troubleshooting them. And if you panic while questioning your very existence, you'll invariably push a hotfix that breaks something else and then you are in a world of hurt. \ Muphy's law is a cruel thing to sysadmins and developers.
- JohnMakin 3y ago
  Completely agree. I’ve triaged many outages with varying degrees of severity in my career so far, and the worst ones were always caused by someone panic-jumping onto some red herring rather than coming up with a sensible explanation as to why that would be a fix other than “it happened at the same time.” I have a saying I really like to throw around, which is “if you don’t know why/how you fixed it, you may not have”
  - hammyhavoc 3y ago
    I read "varying degrees of severity" as "varying degrees of sanity" and was all "omg, he just like me fr" until I reread it. ༼ ༎ຶ ᆺ ༎ຶ༽
  - tetha 3y ago
    > I have a saying I really like to throw around, which is “if you don’t know why/how you fixed it, you may not have” There is coincidental function, and planned function. If you just throw numbers at a configuration parameter until it works and then call it "fixed",it may be fixed, but that's just coincidental. If the system load changes, you can't really tell how to modify the value afterwards. On the other hand, if you can explain why a certain setting will fix it, that's planned function, and if something changes, you most likely know how to proceed from there.
  - oldandboring 3y ago
    > I have a saying I really like to throw around, which is “if you don’t know why/how you fixed it, you may not have” Thanks, I'm going to steal this. Similarly we say: you must be sure you fixed THE problem, not just A problem.
deleted 3y ago
[deleted]
Havoc 3y ago
Ironically the reports of instability got me make an account with them. It's been on my todo list but forgot about it
louthy 3y ago
This is something that every start-up company ends up going through at some point. I’ve been there and it’s painful! Sometimes you just don’t have enough time or resource to build the capabilities that would stop an issue like this. Sometimes you didn’t even think a particular issue could even happen and it comes and bites you. The transparency is important, the learning is important, but also (sometimes) the compensation is important. Kagi should consider giving some search credits for the time that we were unable to use the service. Especially as the real-time response was inadequate (as they freely admit). An outage for a paid-for service is not the same as an outage of a you’re-the-product service
refulgentis 3y ago
I didn't think it was possible to only buy a single core outside of an edge function.
3np 3y ago
Wouldn't some level of per-account rate-limiting make sense? Say, 1000 searches per hour? It's commendable and impressive that Kagi has apparently been able to get this far and perform this consistently without any account-level automated rate-limiting but the only alternative is an inevitable cat-and-mouse and whack-a-mole of cutting off paying customers who knowingly or not violate the ToS. Returning 429 with actionable messages makes it clear to users what they should expect. You obviously want the block-interval to be long enough to not cause too much additional churn on the database. Applying restrictions on IP-level when you can avoid it is just a world of frustration for everyone involved.
- eep_social 3y ago
  > we need to set some automated limits to help us enforce this. From analyzing our user’s usage, we have picked some limits that no good-faith user of Kagi should reasonably hit. > These new limits should already be in place by the time of this post, and we will monitor their impact and continue to tune them as needed.
precision1k 3y ago
Reminds me of a time I was running a proof-of-concept for a new networking tool at a customer site, and about two minutes after we got it running their entire network went down. We were in a sandboxed area so there was no way our product could of caused a network wide outage, but in my head I'm thinking: "there's no way, right. . . .RIGHT?!?!".
- tryauuum 3y ago
  what was the problem? some leaking abstractions?
_xnmw 3y ago
> This didn’t exactly come as a surprise to us, as for the entirety of Kagi’s life so far we have actually used the cheapest, single-core database available to us on GCP! Wow, love that you guys are keeping it lean. Have you considered something like PolyScale to handle sudden spikes in read load and squeeze more performance out of it?
sabujp 3y ago
We were later in contact with an account that we blocked who claimed they were using their account to perform automated scraping of our results, which is not something our terms allow for." Set QPS limits for every possible incoming RPC / API / HTTP request , especially public ones!
- leesalminen 3y ago
  So much this. I learned this the hard way. We had a search function with typeahead abilities. I had intentionally removed the rate limit from that endpoint to support fast typers. One day around 6AM, someone in Tennessee came into work and put their purse down on their keyboard. The purse depressed a single key and started hitting the API with each keystroke. Of course after 15 minutes of this the db became very unhappy. Then a web server crashed because the db was lagging too much. Cascading failures until that whole prod cluster crashed. Needless to say the rate limit was readded that day ;).
  - o11c 3y ago
    This is a reminder that "we want to support bursts" is much more common thing than "we want a higher ratelimit". Often multiple levels of bursts are reasonable (e.g. support 10 requests per minute, but only 100 requests per day; support 10 requests per user, but only 100 requests across all users). There are several ways to track history with just a couple variables (or, if you do have the history, but only accessing a couple of variables); the key observation is that you usually don't have to be exact, only put a bound on it. For history approximations in general, one thing I'm generally fond of is using an exponential moving average (often with λ=1/8 so it can be done with shifts `ema -= ema>>3; ema += datum>>3` and it's obvious overflow can't happen). You do have to be careful that you aren't getting hyperbolic behavior though; I'm not sure I would use this for a rate limiter in particular.
- AtNightWeCode 3y ago
  And a public endpoint is any Internet facing endpoint including the ones where the user needs to be logged in. People seems to forget that.
prh8 3y ago
Totally missed this incident, just want to say thank you to the Kagi team for an awesome product. Love the search and also Orion.
rwiggins 3y ago
Aaaahhh, it's crazy how much this incident resonates with me! I've personally handled this exact same kind of outage more times than I'd care to admit. And just like the fine folks at Kagi, I've fallen into the same rabbit hole (database connection pool health) and tried all the same mitigations - futilely throwing new instances at the problem, the belief that if I could just "reset" traffic it'd all be fixed, etc... It doesn't help that the usual saturation metrics (CPU%, IOPS, ...) for databases typically don't move very much during outages like these. You see high query latency, sure, but you go looking and think: "well, it still has CPU and IOPS headroom..." without realizing, as always, lock contention lurks. In my experience, 98% of the time, any weirdness with DB connection pools is a result of weirdness in the DB itself. Not sure what RDBMS Kagi's running, but I'd highly recommend graphing global I/O wait time (seconds per second) and global lock acquisition time (seconds per second) for the DB. And also query execution time (seconds per second) per (normalized) query. Add a CPU utilization chart and you've got a dashboard that will let you quickly identify most at-scale perf issues. Separately: I'm a bit surprised that search queries trigger RDBMS writes. I would've figured the RDBMS would only be used for things like user settings, login management, etc. I wonder if Kagi's doing usage accounting (e.g. incrementing a counter) in the RDBMS. That'd be an absolute classic failure mode at scale.
- WhitneyLand 3y ago
  I was wondering the same thing. They would have some writes indirectly due to searches, say if someone chooses to block a search result. They’re also going to have some history and analytics surely. But yeah it’s not obvious what should cause per search write lock contention…
  - rwiggins 3y ago
    You know, in retrospect, I think Kagi expects O(thousands) searches per month per user, so doing per-user usage accounting in the DB is fine -- thanks to row-level locking. Well, at least until you get a user who does 60k "in a short time period"... :-)
    - 3y ago
lolc 3y ago
I've been using it for a few weeks now and when it didn't load right away last week I was at a loss what to do. I wondered "what is Kagi and why can't my browser search anymore?" It's really well built to get out of your way and I'd all forgotten about it. Eventually I realized I could use another search engine. The bother! Before this post-mortem dropped I'd also forgotten the incident. Props to the team that doesn't make me think when I search! And my sympathy in this incident. It's rough when things coincide like that and cause you to look at the wrong metrics.
mightyham 3y ago
As much as people gush over Kagi on HN, I still have yet to actually try it because I cannot for the life of me get authentication to work. Even after immediately resetting my password, I get an "incorrect email or password" or "try again later" error on the login page. I've tried at least 3 times over the last few months with the same results each time. If such a fundamental part of a web service company's website is broken, it makes me weary of their competence.
- freediver 3y ago
  Have you tried contacting Kagi support to help debug the issue?
  - mightyham 3y ago
    No and I think it's kind of absurd that I would have to reach out to support staff so that I can get something as basic as account log in working correctly.
    - returningfory2 3y ago
      I'm confident that account login is working for most people. Potentially something weird has happened with you account and it requires manual intervention. This is exactly what support is for - handling weird situations.
    - saagarjha 3y ago
      I feel like it is a lot more absurd to not reach out to the resources that are equipped specifically to handle problems like yours.
      - mightyham 3y ago
        My point is that this is a service that I was curious about but have no real need for. I'm mostly satisfied by using Google or DDG. As a customer, it's pretty absurd that the onus is on me to spend a fairly significant amount of time contacting support in order to simply evaluate a niche product. Furthermore, the company is a tech company, so the fact that their authentication is bugged, seems like more than enough of a reason to not spend any more of my time evaluating their service. I literally can't think of any web based products I currently use which have, at any point, had bugged authentication.
thecleaner 3y ago
What I gather is this was a rate limiting issue. Rate limiting is a standard pattern for API platforms but I wonder how many consumer facing services implement it.
rmrf100 3y ago
Muphy's law
lopkeny12ko 3y ago
> While we do offer unlimited searches to our users, such volume was a clear abuse of our platform, against our terms of use. As such, we removed searching capability from the offending accounts. Don't advertise something as unlimited if it's not actually unlimited.
- hackernewds 3y ago
  confused how both it is advertised, and against ToS
- karlshea 3y ago
  Unlimited searches as a real user is not the same thing as running a bot against the service.
  - eviks 3y ago
    Real users can also run a bot for their real use cases, the correct way out is not to mislead
  - callalex 3y ago
    Just advertise it as 10,000 per day if that’s what it actually is. Any sane user will see a number like that and know they don’t have to worry about the limit.
  - lopkeny12ko 3y ago
    What difference does it make? So would it be "legitimate use" if I sat at a computer and manually clicked 100 times to do something, while it is "abuse" for me to write a script that automates exactly the same thing? Is it only considered abuse if the action isn't tedious and tiring for me as a user?
    - saagarjha 3y ago
      Yes. This is why a buffet that offers you unlimited food requires you to actually eat it, and not just pack it into boxes to take home.
sedatk 3y ago
I’m a paid user of Kagi, and having experienced downtime made me realize how much I took Google’s reliability for granted. Google has never gone down on me, maybe once in the last two decades. Losing access to your search engine is quite crippling. I LOVE Kagi, that’s why I pay for it, but experiencing downtime in my second month was quite off-putting. I love post-mortems, but I hope to never read them. :) That said, I hope this experience makes Kagi even more resilient and reliable.
- tsegers 3y ago
  As another paying user of Kagi I wonder what prevented you from using another search engine for the six hours that Kagi was unavailable. Search engines are not like your email provider or ISP in that you're locked in.
  - wiether 3y ago
    > I wonder what prevented you from using another search engine Well, seeing that Kagi was down, I tried to switch to Google by doing "!g xxx" which gave me the same result than my previous "xxx" search. I took me a few seconds to realize how stupid I was and then typing "google.com" in the address bar.
  - sedatk 3y ago
    The thing is, I never thought Kagi was down and thought that it must be a problem with my configuration or connection. That was how much I trusted in Kagi. I didn’t spend whole downtime online though.
- pbronez 3y ago
  100%. This outage and the (unrelated) bug on the new mobile safari extension were jarring. I definitely rely on snappy & ubiquitous to Kagi!
lostlogin 3y ago
I cracked up when I saw this. My search went bad and I assumed I’d broken something, tried a few things, took a break, tried a few more and it was working. I moved on. It wasn’t ever at my end.
dotnet00 3y ago
This was fun to see playing out, it made me realize how I didn't even think about how impressive it was that you guys (being a small team) were pushing out updates with very few bugs or visible downtime. I didn't mind the downtime, just gave me an excuse to take a break and focus on more casual coding.
WhereIsTheTruth 3y ago
If 60k requests hurts your infra, then your infra is poorly built and you cheap out on resources Also straight out blocking the account instead of rate-limiting? Yeah, poor infrastructure Kagi is aimed at power users, they should do and provide better
- andrewaylett 3y ago
  60k requests isn't a useful metric on its own. How long is that over, and what's their normal rate?
- saagarjha 3y ago
  You spend extra on resources and people lambast you for your bloated and expensive stack. You run it lean and people like you say you’re cheaping out.
LeoNatan25 3y ago
Ouch, a seven hour downtime for a paid search engine service. Maybe “every startup goes through this”, as some comment here stated, but not all startups are created equal. I wonder how much this incident will cost them long term.
AtNightWeCode 3y ago
I would put something like Cloudflare in front of a service like this. Bot protection may handle it and if not a technician can fix it in 5 minutes.
MyFedora 3y ago
> While we do offer unlimited searches to our users, such volume was a clear abuse of our platform, against our terms of use. Disclose that it's fair use upfront. Leveraging unlimited searches =/= Abuse. Never bait and switch unlimited to fair use, nobody reads fine print. Customers who pay for unlimited expect the service to scale the infrastructure to handle whatever they throw at it. I think that's a reasonable expectation since they paid for unlimited, but got limited.
- _cenw 3y ago
  the presence of a paid search API should be enough of a hint that a $10 account trying to read back the entire index is not going to be tolerated. https://help.kagi.com/kagi/api/search.html https://help.kagi.com/kagi/api/search.html though the lack of alerting and the lack of consideration for bad actors in general (just look for the kagi feedback thread on suicide results, you can kagi it) seems pretty consistent
  - nijave 3y ago
    Not necessarily. It's common to charge business users a different amount (>0) than personal use
  - xigoi 3y ago
    What does this have to do with the suicide debate?
joelcollinsdc 3y ago
Always scary when you learn you can be DOS’d by a single user that learned to use bash and a for loop.

346 points by leetrout 3y ago

This incidcent also shows that no security audit was ever performed on this platform, as a DoS vulnerability would've been one of the first findings there. Makes you wonder what other vulnerabilities are in their infrastructure...

deleted 3y ago

[deleted]

renewiltord 3y ago

Interesting. The classic problem. You offer to not meter something and then someone will use it to max capacity. Then you’re forced to place a limit so that one user won’t hose everyone else.

fotta 3y ago
> We were later in contact with an account that we blocked who claimed they were using their account to perform automated scraping of our results, which is not something our terms allow for. I mean beyond that it was a user that was violating the TOS. This isn’t really a bait and switch scenario (although it could be reasonably construed as such).
JumpCrisscross 3y ago
> you’re forced to place a limit so that one user won’t hose everyone else Soft limits at the tail of usage comports with the term unlimited as it's commonly used. For Kagi, a rate limit derived from how quickly a human can type makes sense.

fwsgonzo 3y ago

It wasn't that long ago that I had heard about Kagi the first time. Now I use it every day, and the fact that I can pin cppreference.com to the top is just such a boon.

boomboomsubban 3y ago

One user running a scraper took the service down for seven hours? I know it's easy to sit on the outside and say they should have seen this coming, but how does nobody in testing go "what happens if a ton of searches happen?"

z64 3y ago
Hi there, this is Zac from Kagi. I just posted some other details here that might be of interest: https://news.ycombinator.com/item?id=39019936 https://news.ycombinator.com/item?id=39019936 TL;DR - we are a tiny, young team at the center, and everyone has a closet full of hats they wear. No dedicated SRE team yet. > "what happens if a ton of searches happen?" In fairness, you can checkout https://kagi.com/stats https://kagi.com/stats - "a lot of searches" is already happening, approaching 400k per day, and systems still operate with plenty of capacity day-to-day, in addition to some auto-scaling measures. The devil is in the details of some users exploting a pathological case. Our lack of experience (now rightfully gained) is knowing what organic or pathological traffic we could have predicted and simulated ahead of time. Load-simulating 20,000 users searching concurrently sounds like it would have been a sound experiment early on, and we did do some things resembling this. But considering this incident, it still would not have caught this issue. We have also had maybe 10 people run security scanners on our production services at this point that generated more traffic than this incident. It is extremely difficult to balance this kind of development when we also have features to build, and clearly we could do with more of it! As mentioned in my other post, we are looking to expand the team in the near term so that we are not spread so thin on these sorts of efforts. There is a lot that could be said in hindsight, but I hope that is a bit more transparent WRT how we ended up here.
- smcleod 3y ago
  Zac, I think you’re doing great handling and communicating this. Keep up the great work and have fun learning while you’re at it!
- rconti 3y ago
  What does "pathological" mean in this context?

fragmede 3y ago

That speaks volumes about the observability they have of their internal systems. It's easy for me to say they should have seen it sooner, but the right datadog dashboards and splunk queries should have made that clear as day much faster. Hopefully they take it as a learning experience and invest in better monitoring.

z64 3y ago
Hi there, I'm Zac, Kagi's tech lead / author of the post-mortem etc. This has 100% been a learning experience for us, but I can provide some finer context re: observability. Kagi is a small team. The number of staff we have capable of responding to an event like this is essentially 3 people, seated across 3 timezones. For myself and my right-hand dev, this is actually our very first step in our web careers - this is to say that we are not some SV vets who have seen it all already. To say that we have a lot to learn is a given, from building Kagi from nothing though, I am proud of how far we've come & where we're going. Observability is something we started taking more seriously in the past 6 months or so. We have tons of dashboards now, and alerts that go right to our company chat channels and ping relevant people. And as the primary owner of our DB, GCP's query insights are a godsend. During the incident both our monitoring went off, as well as query insights showing the "culprit" query - but, we could have monitoring in the world, and still lack the experience to interpret it and understand what the root cause is or most efficient action to mitigate is. In other words, we don't have the wisdom yet to not be "gaslit" by our own systems if we're not careful. Only in hindsight can I say that GCP's query insights was 100% on the money, and not some bug in application space. All said, our growth has enabled us to expand our team quite a bit now. We have had SRE consultations before, and intend to bring on more full or part-time support to help keep things moving forward.
- ijhuygft776 3y ago
  Kagi, an efficient company. Thanks Zac and the rest of the team
- jjtheblunt 3y ago
  I bet a silent majority are thinking "well done, Zac, all the same".

layoric 3y ago

"This didn’t exactly come as a surprise to us, as for the entirety of Kagi’s life so far we have actually used the cheapest, single-core database available to us on GCP!" Outages suck, but I love the fact that they are building such a lean product. Been paying for Kagi as a part of de-Google-ifying my use of online services and the experience so far (I wasn't impacted by this outage) has been great. A few years ago I built a global SaaS (first employee and SWE) in the weather space which was backed by a single DB, and while it had more than just 1 core (8 when I left from memory), I think a lot of developers reach for distributed DBs far too early. Modern hardware can do a lot, products like AWS Aurora are impressive, but they come with their own complexities (and MUCH higher costs).

z64 3y ago
Very cool! The deluge of cloud solutions are absolutely one of those things that can be a distraction from figuring out "What do I actually need the computer to do?". Internally I try to promote solutions with the fewest moving cloud-parts, and channel the quiet wisdom of those running services way larger than ours with something like an sqlite3 file that they rsync... I know they're out there. Not to downplay the feats of engineering of huge distributed solutions, but sometimes things can be that simple!
- layoric 3y ago
  Distracting is right! I watched your CrystalConf video and was happy to see the familiar Postgres + Redis combo :). I remember worrying about running out of legs with Redis (being single threaded), but with a combo of pipelining and changing the data structures used it ended up being the piece of infra that had the most headroom. Monitoring was probably the biggest value for outsourcing to another SaaS. I used Runscope, AWS dashboards and own elasticsearch and it was pretty cost effective for the API that was doing ~2M API calls a day. The other risk of cloud solutions is the crazy cost spikes. I remember consulting a partner, much larger, weather company on reducing their costs of a similar global weather product where they chose AWS DynamoDB as their main datastore, and their bill just for DynamoDB was twice our companies cloud bill. All because it was slightly "easier" to not think about compute requirements! Any ways, thanks for the postmortem, hopefully your channeling of quiet wisdom continues to branch out to others! :)

jacob019 3y ago

If you're listening, Kagi, please add an à la carte plan for search. Maybe hide it behind the API options as not to disrupt your normal plans. I love the search and I'm happy to pay, but I'm cost sensitive now and it's the only way that I'm going to feel comfortable using it long-term.

spiderice 3y ago
They have a $5 for 300 searches option. Is that not what you're referring to?
- JumpCrisscross 3y ago
  > Is that not what you're referring to? It sounds like they'd prefer a 2¢ per search option via API.
  - jacob019 3y ago
    Yes, exactly. Or 5 cents for the first 100 searches, 2 cents after, something like that.
    - binsquare 3y ago
      They do have the FastGPT api which is approx 1.5cents per api call
      - lostlogin 3y ago
        So raise this to 5c for first 100 and 2c thereafter and we have a happy customer?
        jacob019 3y ago
        Not search.
- BeetleB 3y ago

zilti 3y ago

[flagged]

HaZeust 3y ago
I'll bite; how do you figure?
- zilti 3y ago
  GCP prices are ludicrously high. You're far better off to just get a VPS and self-host Postgres and your webserver. Which is what we did at my last startup; only thing we used GCP for was build servers, and only because we got a ton of free credits. We could've gotten a paid intern for the money we saved.

smcleod 3y ago

Good write up. I always appreciate Kagi’s honesty and transparency like this. Great product, great service.

muhammadusman 3y ago

I was one of the users that went and reported this issue on Discord. I love Kagi but I was a bit disappointed to see that their status page showed everything was up and running. I think that made me a bit uneasy and it shows their status pages are not given priority during incidents that are affecting real users. I hope in the future the status page is accurately updated. In the past, services I heavily rely on (e.g. Github), have updated their status pages immediately and this allows me to rest assured that people are aware of the issue and it's not an issue with my devices. When this happened with Kagi, I was looking up the nearest grocery stores open since we were getting snow later that day so it was almost like I got let down b/c I had to go to Google for this. I will continue using Kagi b/c 99.9% of the other time I've used it, it has been better than Google but I hope the authors of the post-mortem do mean it when they say they'll be moving their status page code to a different service/platform. And thanks again Zac for being transparent and writing this up. This is part of good engineering!

Terretta 3y ago
> In the past, services I heavily rely on (e.g. Github), have updated their status pages immediately Also in the past, other times GitHub has not updated its status page immediately.
Neikius 3y ago
Microsoft is notorious for their lax status page updates...
- wiml 3y ago
  Is there anyone who isn't?
lambdaba 3y ago
I'm only replying to the praise here - I too, although I haven't fully switched, had a very enticing moment with Kagi when it returned a result that couldn't even be found on Google at any page in the results. This really sold me on Kagi and I've been going back and forth with some queries, but I have to say that between LLMs, Perplexity, and Google often answering my queries right on the search page, I just don't have that many queries left for Kagi. If Kagi would somehow merge with Perplexity, now that would be something.

blantonl 3y ago

At first, by what turned out to be a complete coincidence, the incident occurred at precisely the same time that we were performing an infrastructure upgrade to our VMs with additional RAM resources I can assure you that these "coincidences" happen all the time, and will cause you to question your very existence when you are troubleshooting them. And if you panic while questioning your very existence, you'll invariably push a hotfix that breaks something else and then you are in a world of hurt. \ Muphy's law is a cruel thing to sysadmins and developers.

JohnMakin 3y ago
Completely agree. I’ve triaged many outages with varying degrees of severity in my career so far, and the worst ones were always caused by someone panic-jumping onto some red herring rather than coming up with a sensible explanation as to why that would be a fix other than “it happened at the same time.” I have a saying I really like to throw around, which is “if you don’t know why/how you fixed it, you may not have”
- hammyhavoc 3y ago
  I read "varying degrees of severity" as "varying degrees of sanity" and was all "omg, he just like me fr" until I reread it. ༼ ༎ຶ ᆺ ༎ຶ༽
- tetha 3y ago
  > I have a saying I really like to throw around, which is “if you don’t know why/how you fixed it, you may not have” There is coincidental function, and planned function. If you just throw numbers at a configuration parameter until it works and then call it "fixed",it may be fixed, but that's just coincidental. If the system load changes, you can't really tell how to modify the value afterwards. On the other hand, if you can explain why a certain setting will fix it, that's planned function, and if something changes, you most likely know how to proceed from there.
- oldandboring 3y ago
  > I have a saying I really like to throw around, which is “if you don’t know why/how you fixed it, you may not have” Thanks, I'm going to steal this. Similarly we say: you must be sure you fixed THE problem, not just A problem.

deleted 3y ago

[deleted]

Havoc 3y ago

Ironically the reports of instability got me make an account with them. It's been on my todo list but forgot about it

louthy 3y ago

This is something that every start-up company ends up going through at some point. I’ve been there and it’s painful! Sometimes you just don’t have enough time or resource to build the capabilities that would stop an issue like this. Sometimes you didn’t even think a particular issue could even happen and it comes and bites you. The transparency is important, the learning is important, but also (sometimes) the compensation is important. Kagi should consider giving some search credits for the time that we were unable to use the service. Especially as the real-time response was inadequate (as they freely admit). An outage for a paid-for service is not the same as an outage of a you’re-the-product service

refulgentis 3y ago

I didn't think it was possible to only buy a single core outside of an edge function.

3np 3y ago

Wouldn't some level of per-account rate-limiting make sense? Say, 1000 searches per hour? It's commendable and impressive that Kagi has apparently been able to get this far and perform this consistently without any account-level automated rate-limiting but the only alternative is an inevitable cat-and-mouse and whack-a-mole of cutting off paying customers who knowingly or not violate the ToS. Returning 429 with actionable messages makes it clear to users what they should expect. You obviously want the block-interval to be long enough to not cause too much additional churn on the database. Applying restrictions on IP-level when you can avoid it is just a world of frustration for everyone involved.

eep_social 3y ago
> we need to set some automated limits to help us enforce this. From analyzing our user’s usage, we have picked some limits that no good-faith user of Kagi should reasonably hit. > These new limits should already be in place by the time of this post, and we will monitor their impact and continue to tune them as needed.

precision1k 3y ago

Reminds me of a time I was running a proof-of-concept for a new networking tool at a customer site, and about two minutes after we got it running their entire network went down. We were in a sandboxed area so there was no way our product could of caused a network wide outage, but in my head I'm thinking: "there's no way, right. . . .RIGHT?!?!".

tryauuum 3y ago
what was the problem? some leaking abstractions?

_xnmw 3y ago

> This didn’t exactly come as a surprise to us, as for the entirety of Kagi’s life so far we have actually used the cheapest, single-core database available to us on GCP! Wow, love that you guys are keeping it lean. Have you considered something like PolyScale to handle sudden spikes in read load and squeeze more performance out of it?

sabujp 3y ago

We were later in contact with an account that we blocked who claimed they were using their account to perform automated scraping of our results, which is not something our terms allow for." Set QPS limits for every possible incoming RPC / API / HTTP request , especially public ones!

leesalminen 3y ago
So much this. I learned this the hard way. We had a search function with typeahead abilities. I had intentionally removed the rate limit from that endpoint to support fast typers. One day around 6AM, someone in Tennessee came into work and put their purse down on their keyboard. The purse depressed a single key and started hitting the API with each keystroke. Of course after 15 minutes of this the db became very unhappy. Then a web server crashed because the db was lagging too much. Cascading failures until that whole prod cluster crashed. Needless to say the rate limit was readded that day ;).
- o11c 3y ago
  This is a reminder that "we want to support bursts" is much more common thing than "we want a higher ratelimit". Often multiple levels of bursts are reasonable (e.g. support 10 requests per minute, but only 100 requests per day; support 10 requests per user, but only 100 requests across all users). There are several ways to track history with just a couple variables (or, if you do have the history, but only accessing a couple of variables); the key observation is that you usually don't have to be exact, only put a bound on it. For history approximations in general, one thing I'm generally fond of is using an exponential moving average (often with λ=1/8 so it can be done with shifts `ema -= ema>>3; ema += datum>>3` and it's obvious overflow can't happen). You do have to be careful that you aren't getting hyperbolic behavior though; I'm not sure I would use this for a rate limiter in particular.
AtNightWeCode 3y ago
And a public endpoint is any Internet facing endpoint including the ones where the user needs to be logged in. People seems to forget that.

prh8 3y ago

Totally missed this incident, just want to say thank you to the Kagi team for an awesome product. Love the search and also Orion.

rwiggins 3y ago

Aaaahhh, it's crazy how much this incident resonates with me! I've personally handled this exact same kind of outage more times than I'd care to admit. And just like the fine folks at Kagi, I've fallen into the same rabbit hole (database connection pool health) and tried all the same mitigations - futilely throwing new instances at the problem, the belief that if I could just "reset" traffic it'd all be fixed, etc... It doesn't help that the usual saturation metrics (CPU%, IOPS, ...) for databases typically don't move very much during outages like these. You see high query latency, sure, but you go looking and think: "well, it still has CPU and IOPS headroom..." without realizing, as always, lock contention lurks. In my experience, 98% of the time, any weirdness with DB connection pools is a result of weirdness in the DB itself. Not sure what RDBMS Kagi's running, but I'd highly recommend graphing global I/O wait time (seconds per second) and global lock acquisition time (seconds per second) for the DB. And also query execution time (seconds per second) per (normalized) query. Add a CPU utilization chart and you've got a dashboard that will let you quickly identify most at-scale perf issues. Separately: I'm a bit surprised that search queries trigger RDBMS writes. I would've figured the RDBMS would only be used for things like user settings, login management, etc. I wonder if Kagi's doing usage accounting (e.g. incrementing a counter) in the RDBMS. That'd be an absolute classic failure mode at scale.

WhitneyLand 3y ago
I was wondering the same thing. They would have some writes indirectly due to searches, say if someone chooses to block a search result. They’re also going to have some history and analytics surely. But yeah it’s not obvious what should cause per search write lock contention…
- rwiggins 3y ago
  You know, in retrospect, I think Kagi expects O(thousands) searches per month per user, so doing per-user usage accounting in the DB is fine -- thanks to row-level locking. Well, at least until you get a user who does 60k "in a short time period"... :-)
  - 3y ago

lolc 3y ago

I've been using it for a few weeks now and when it didn't load right away last week I was at a loss what to do. I wondered "what is Kagi and why can't my browser search anymore?" It's really well built to get out of your way and I'd all forgotten about it. Eventually I realized I could use another search engine. The bother! Before this post-mortem dropped I'd also forgotten the incident. Props to the team that doesn't make me think when I search! And my sympathy in this incident. It's rough when things coincide like that and cause you to look at the wrong metrics.

mightyham 3y ago

As much as people gush over Kagi on HN, I still have yet to actually try it because I cannot for the life of me get authentication to work. Even after immediately resetting my password, I get an "incorrect email or password" or "try again later" error on the login page. I've tried at least 3 times over the last few months with the same results each time. If such a fundamental part of a web service company's website is broken, it makes me weary of their competence.

freediver 3y ago
Have you tried contacting Kagi support to help debug the issue?
- mightyham 3y ago
  No and I think it's kind of absurd that I would have to reach out to support staff so that I can get something as basic as account log in working correctly.
  - returningfory2 3y ago
    I'm confident that account login is working for most people. Potentially something weird has happened with you account and it requires manual intervention. This is exactly what support is for - handling weird situations.
  - saagarjha 3y ago
    I feel like it is a lot more absurd to not reach out to the resources that are equipped specifically to handle problems like yours.
    - mightyham 3y ago
      My point is that this is a service that I was curious about but have no real need for. I'm mostly satisfied by using Google or DDG. As a customer, it's pretty absurd that the onus is on me to spend a fairly significant amount of time contacting support in order to simply evaluate a niche product. Furthermore, the company is a tech company, so the fact that their authentication is bugged, seems like more than enough of a reason to not spend any more of my time evaluating their service. I literally can't think of any web based products I currently use which have, at any point, had bugged authentication.

thecleaner 3y ago

What I gather is this was a rate limiting issue. Rate limiting is a standard pattern for API platforms but I wonder how many consumer facing services implement it.

rmrf100 3y ago

Muphy's law

lopkeny12ko 3y ago

> While we do offer unlimited searches to our users, such volume was a clear abuse of our platform, against our terms of use. As such, we removed searching capability from the offending accounts. Don't advertise something as unlimited if it's not actually unlimited.

hackernewds 3y ago
confused how both it is advertised, and against ToS
karlshea 3y ago
Unlimited searches as a real user is not the same thing as running a bot against the service.
- eviks 3y ago
  Real users can also run a bot for their real use cases, the correct way out is not to mislead
- callalex 3y ago
  Just advertise it as 10,000 per day if that’s what it actually is. Any sane user will see a number like that and know they don’t have to worry about the limit.
- lopkeny12ko 3y ago
  What difference does it make? So would it be "legitimate use" if I sat at a computer and manually clicked 100 times to do something, while it is "abuse" for me to write a script that automates exactly the same thing? Is it only considered abuse if the action isn't tedious and tiring for me as a user?
  - saagarjha 3y ago
    Yes. This is why a buffet that offers you unlimited food requires you to actually eat it, and not just pack it into boxes to take home.

sedatk 3y ago

I’m a paid user of Kagi, and having experienced downtime made me realize how much I took Google’s reliability for granted. Google has never gone down on me, maybe once in the last two decades. Losing access to your search engine is quite crippling. I LOVE Kagi, that’s why I pay for it, but experiencing downtime in my second month was quite off-putting. I love post-mortems, but I hope to never read them. :) That said, I hope this experience makes Kagi even more resilient and reliable.

tsegers 3y ago
As another paying user of Kagi I wonder what prevented you from using another search engine for the six hours that Kagi was unavailable. Search engines are not like your email provider or ISP in that you're locked in.
- wiether 3y ago
  > I wonder what prevented you from using another search engine Well, seeing that Kagi was down, I tried to switch to Google by doing "!g xxx" which gave me the same result than my previous "xxx" search. I took me a few seconds to realize how stupid I was and then typing "google.com" in the address bar.
- sedatk 3y ago
  The thing is, I never thought Kagi was down and thought that it must be a problem with my configuration or connection. That was how much I trusted in Kagi. I didn’t spend whole downtime online though.
pbronez 3y ago
100%. This outage and the (unrelated) bug on the new mobile safari extension were jarring. I definitely rely on snappy & ubiquitous to Kagi!

lostlogin 3y ago

I cracked up when I saw this. My search went bad and I assumed I’d broken something, tried a few things, took a break, tried a few more and it was working. I moved on. It wasn’t ever at my end.

dotnet00 3y ago

This was fun to see playing out, it made me realize how I didn't even think about how impressive it was that you guys (being a small team) were pushing out updates with very few bugs or visible downtime. I didn't mind the downtime, just gave me an excuse to take a break and focus on more casual coding.

WhereIsTheTruth 3y ago

If 60k requests hurts your infra, then your infra is poorly built and you cheap out on resources Also straight out blocking the account instead of rate-limiting? Yeah, poor infrastructure Kagi is aimed at power users, they should do and provide better

andrewaylett 3y ago
60k requests isn't a useful metric on its own. How long is that over, and what's their normal rate?
saagarjha 3y ago
You spend extra on resources and people lambast you for your bloated and expensive stack. You run it lean and people like you say you’re cheaping out.

LeoNatan25 3y ago

Ouch, a seven hour downtime for a paid search engine service. Maybe “every startup goes through this”, as some comment here stated, but not all startups are created equal. I wonder how much this incident will cost them long term.

AtNightWeCode 3y ago

I would put something like Cloudflare in front of a service like this. Bot protection may handle it and if not a technician can fix it in 5 minutes.

MyFedora 3y ago

> While we do offer unlimited searches to our users, such volume was a clear abuse of our platform, against our terms of use. Disclose that it's fair use upfront. Leveraging unlimited searches =/= Abuse. Never bait and switch unlimited to fair use, nobody reads fine print. Customers who pay for unlimited expect the service to scale the infrastructure to handle whatever they throw at it. I think that's a reasonable expectation since they paid for unlimited, but got limited.

_cenw 3y ago
the presence of a paid search API should be enough of a hint that a $10 account trying to read back the entire index is not going to be tolerated. https://help.kagi.com/kagi/api/search.html https://help.kagi.com/kagi/api/search.html though the lack of alerting and the lack of consideration for bad actors in general (just look for the kagi feedback thread on suicide results, you can kagi it) seems pretty consistent
- nijave 3y ago
  Not necessarily. It's common to charge business users a different amount (>0) than personal use
- xigoi 3y ago
  What does this have to do with the suicide debate?

joelcollinsdc 3y ago

Always scary when you learn you can be DOS’d by a single user that learned to use bash and a for loop.