5 ms·

Year old startup overloaded GitHub – Incident report

27 points by Henrik501 2y ago

deleted 2y ago
[deleted]
gnabgib 2y ago
Discussion yesterday (14 points, 8 comments) https://news.ycombinator.com/item?id=42645887 https://news.ycombinator.com/item?id=42645887
diggan 2y ago
Why is the title of the post-mortem "GitHub Outage"? It makes it sound like Lovable somehow brought down GitHub, when in reality it seems like they were rate-limited by GitHub for creating lots of repositories, then got their GitHub App completely blocked for breaching the Terms of Service. > Incident report for the GitHub outage on January 2-3, 2025 Writing it like that looks like you're pushing the blame on your downtime/outage to GitHub, like they're responsible for your application to be up, instead of taking full responsibility for it.
- mschuster91 2y ago
  > Writing it like that looks like you're pushing the blame on your downtime/outage to GitHub, like they're responsible for your application be up, instead of taking full responsibility for it. Well, they did what they were supposed to - they explicitly asked Github what they were up to, Github gave an explicit "we're ok with this, go ahead", and once Github sees that, whoops, it's causing errors they don't even bother to check if there are support tickets open with the customer, they just go and disable their access.
  - diggan 2y ago
    But that's true for any 3rd party you'd depend on. Everything will work until it doesn't. Doesn't mean you're less responsible for your project being down. A title like "GitHub caused our outage" would still make it clear the downtime wasn't the direct action of anyone on the team, yet still take responsibility over that it happened. Instead, labeling it "Incident report for the GitHub outage" just seems like straight up blaming someone else.
    - Mathnerd314 2y ago
      Well, Github explicitly took responsibility. The first action Github did once Lovable reached out for support was "reinstate our app and apologize for the issues it caused us and our users." And no, you are not responsible for every 3rd party service you use. Some services are unavoidable, some services are just nice-to-have, but if you can't trust a service to perform its advertised function, it is the service's fault.
      - diggan 2y ago
        > Well, Github explicitly took responsibility. The first action Github did once Lovable reached out for support was "reinstate our app and apologize for the issues it caused us and our users." No, GitHub won't ever take responsibility for what you have between you and your projects/apps/companies users. Nor did they do so in this case. If I use service X for doing Y, and I write an email asking if it's OK that I upload 1000s of files every day, even if they say OK today, but then next week turn around and say "Nah", I'm still responsible for my users, who trust me and my team. Service X has no responsibility towards those users at all. It sucks though, I agree with that. > And no, you are not responsible for every 3rd party service you use. Some services are unavoidable, some services are just nice-to-have, but if you can't trust a service to perform its advertised function, it is the service's fault. Besides DNS and BGP I suppose, what services are "unavoidable" exactly? Git hosting isn't some arcane distributed network technology needing years of reading/experience to understand, the CLI ships with a web ui you can basically copy-paste to have your own Git hosting. I'd say you are responsible for everything that you use and depend on. And if you think "Ah, I'll just chunk 10K repositories at GitHub a day, they say it's fine" and then don't have any plan in case GitHub suddenly says it isn't OK, you are responsible for the falloff if shit hits the fan.
        Mathnerd314 2y ago
        Well, the app store, for example. Sure, it is of course a good idea to comply with the app store policies to the extent possible, but ultimately there not much you can do to prevent Google or Apple saying "we don't like this app" and pulling it. For example with the UTM emulator. So how then can Google or Apple making such a decision be "your responsibility"? As another example, let's say you build a house in a hurricane-prone area. It's your responsibility to ensure the owner buys hurricane insurance, as mandated by law. It's not your responsibility to build a nuclear-bunker-grade house that is impervious to hurricanes. It is easy to point the finger and say "you should have thought of that", but in practice it is easier to deal with such catastrophes as they happen.
  - arccy 2y ago
    sounds like a pretty sane thing to do: github protects the majority of their customers from instability caused by a few. unless you're a super important partner, the people on call might never have heard of your little app and just decide that it's the only safe thing to do to protect the reliability of the system.
  - cudgy 2y ago
    If I read it correctly, it was a support person that provided them with assurance. Not an executive or vice president or manager or vp of sales. GitHub did not give them permission nor their approval; it was a single person in support department. Who relies on support people to determine the basis of their business when it’s obvious that they were concerned with the high usage rate and that it might cause problems for their customers?
    - kgeist 2y ago
      Oh yeah, support staff aren't always aware of everything. For example, with one of our latest features, we didn’t have time to add a UI option to disable the feature. The expectation was that support staff could disable it via their special admin panel upon user request. However, I accidentally discovered that when users asked to hide the feature, tech support told them it wasn’t possible! It turned out the tech support lead forgot to share that information with the team. As for the OP, they should have conducted load testing and implemented rate limits on their end, rather than blindly relying on someone’s word that GitHub was ready to handle all their product's load for free.
koreanguy 2y ago
[dead]
qeternity 2y ago
I don't really understand how a well funded startup like this, with something that is relatively trivial, yet critical to their product, decided to just shove it into GitHub.
- remram 2y ago
  Their product is the creation of Git repos. Putting it on the platform their customers want to use makes a lot of sense. They probably should have had a backup location from day 2 though, I agree. If nothing else, in case of a GitHub outage.
  - diggan 2y ago
    > Their product is the creation of Git repos. Putting it on the platform their customers want to use makes a lot of sense. Maybe I read the landing page very wrong, but it seems to be a "app building toolkit" of some sorts? Not just "creation of git repos". They could have made the GitHub repository creation happen when the user does some action, instead of at the stage of "create app" which probably every single user does at least once, even people with no intention of actually building apps. Or better yet, offer their own viewer for Git repositories they themselves host. It's not overly difficult, and the `git` CLI tools even ship with a web UI you can take inspiration from.
    - d3nj4l 2y ago
      If you want a GitHub like UI Forgejo is FOSS too.
  - cratermoon 2y ago
    What value does Lovable's product add, then?
    - Kwpolska 2y ago
      It doesn’t add value. It fills them with LLM-generated code.
  - 2y ago
aimazon 2y ago
Many mistakes were made by Lovable that they could be berated for but on a more positive note, there is a lesson for us all: if you're doing something that you're worried about being problematic (e.g: creating a large volume of GitHub repositories) reaching out is a good thing but it is important to understand who you are reaching out to. GitHub is a huge organization, front-line support is not likely to have intimate knowledge of how exactly the acceptable usage policy is enforced nor the permission to make agreements. The key when reaching out is to find someone who has authority on the subject. Ideally, GitHub's front-line support would have escalated to the appropriate person/team but that isn't always possible (maybe they don't know who, maybe they're having a bad day and forgot). If the answer you get seems too convenient, it is probably not correct.
- chrisgd 2y ago
  If I reach out and they can’t direct me to the right solution, that doesn’t seem like something I need to continue to solve for them. Seems too onerous.
  - gabeio 2y ago
    If it is the critical path for your project or worse for your job, seems like that’s just due diligence. > Seems to onerous. I would only say that about a non-critical personal project.
    - chrisgd 2y ago
      Fair points
V__ 2y ago
Besides the outage, isn't it kind of an insane setup to use GitHub to store every clients' project, especially with 10k new repos every day?
elicksaur 2y ago
Title is misleading. Clicked thinking this startup caused a GitHub outage. Getting a flag like this hardly “overloaded” anything. Alerts for these things usually trigger well below any actual risk to the system.
rikafurude21 2y ago
git is not github, and its kind of funny that the best these guys could come up with for storing git repos of text files was using github. any competent dev could come up with a solution in an afternoon. using github at all tells you quite a bit
temp8388344 2y ago
The same user who posted this (Henrik501) also posted a comment two days ago (their only HN comment so far) praising the Lovable team for their incident response: https://news.ycombinator.com/item?id=42646297 https://news.ycombinator.com/item?id=42646297 And now this post with an exaggerated title. Seems like they're shilling and trying to make Lovable sound like a product with such huge traction that it "even brought Github down". They keep making outlandish claims on social media too, like reaching $4m ARR in 5 weeks etc. This company is very suspicious.
- hiatus 2y ago
  The same username on Github follows only one person, Niklas Vatn, who appears to be a lovable employee. https://github.com/Henrik501?tab=following https://github.com/Henrik501?tab=following Looking more into it, Henrik Westerlund works on "growth" at lovable (via LinkedIn) and posted Lovable to Product Hunt https://www.producthunt.com/@henrik_westerlund https://www.producthunt.com/@henrik_westerlund. So it appears they are shilling.
  - temp8388344 2y ago
    Pathetic company built on lies and shady "growth hacks"
- kgeist 2y ago
  >praising the Lovable team for their incident response For 8 hours, no one is aware the service is down. It then takes ~3 more days to fix it. One of their first decisions is to 'make as much noise as possible on social media' (?), and every step seems to create additional problems (corrupt repos etc.) Nothing appears to be well thought out, the blog post reads like they weren't ready for this at all, panicked and chaotic decisions without understanding the tech stack on a deeper level (race conditions, rate limits etc.) Not a lot of confidence in the team behind a project that looks like nothing more than a glue between an LLM and a storage backend.
- elicksaur 2y ago
  I like that HN is so minimal, but obvious stuff like this makes me want to write a browser extension that lets me custom tag accounts for my own notes.
CamouflagedKiwi 2y ago
315,000 repositories + 10,000 per day? They were obviously correctly concerned this wouldn't be able to go on forever, hence the pre-emptive email, and of course they got the response saying it was okay, but I really feel like this is the kind of thing that's too dangerous to leave your company sitting on because sooner or later they were going to be told "no". It feels too much like it's found a point of arbitrage in Github's ToS, and indeed ended up causing problems. I suppose they did respond pretty fast, but if I were them I'd have liked to have had the S3 option in my back pocket earlier. Maybe I'm just being too risk-averse here...
- registeredcorn 2y ago
  If nothing else, I was a little surprised they didn't (or at least didn't mention) having a fail over plan in place already. Seems like "Prepare for the worst, hope for the best" would have been the logical game plan.
Kwpolska 2y ago
The title ("Year old startup overloaded GitHub – Incident report") is a misrepresentation. A coding-LLM-as-a-service startup got banned by GitHub for abusing it by creating thousands of repositories.
deleted 2y ago
[deleted]
kgeist 2y ago
Interestingly, currently the site has this: >We're currently investigating issues. Please stay tuned until this error banner has been updated. When I try to create a new project, it says "An unknown error occurred with the code sandbox" Something about S3-backed repos didn't work out? UPD. "Under maintenance: Lovable is currently not able to reach the cloud provider for our previews, fly.io" Now it's fly.io's problem, not GitHub's

27 points by Henrik501 2y ago

deleted 2y ago

[deleted]

Discussion yesterday (14 points, 8 comments) https://news.ycombinator.com/item?id=42645887 https://news.ycombinator.com/item?id=42645887

diggan 2y ago

Why is the title of the post-mortem "GitHub Outage"? It makes it sound like Lovable somehow brought down GitHub, when in reality it seems like they were rate-limited by GitHub for creating lots of repositories, then got their GitHub App completely blocked for breaching the Terms of Service. > Incident report for the GitHub outage on January 2-3, 2025 Writing it like that looks like you're pushing the blame on your downtime/outage to GitHub, like they're responsible for your application to be up, instead of taking full responsibility for it.

mschuster91 2y ago
> Writing it like that looks like you're pushing the blame on your downtime/outage to GitHub, like they're responsible for your application be up, instead of taking full responsibility for it. Well, they did what they were supposed to - they explicitly asked Github what they were up to, Github gave an explicit "we're ok with this, go ahead", and once Github sees that, whoops, it's causing errors they don't even bother to check if there are support tickets open with the customer, they just go and disable their access.
- diggan 2y ago
  But that's true for any 3rd party you'd depend on. Everything will work until it doesn't. Doesn't mean you're less responsible for your project being down. A title like "GitHub caused our outage" would still make it clear the downtime wasn't the direct action of anyone on the team, yet still take responsibility over that it happened. Instead, labeling it "Incident report for the GitHub outage" just seems like straight up blaming someone else.
  - Mathnerd314 2y ago
    Well, Github explicitly took responsibility. The first action Github did once Lovable reached out for support was "reinstate our app and apologize for the issues it caused us and our users." And no, you are not responsible for every 3rd party service you use. Some services are unavoidable, some services are just nice-to-have, but if you can't trust a service to perform its advertised function, it is the service's fault.
    - diggan 2y ago
      > Well, Github explicitly took responsibility. The first action Github did once Lovable reached out for support was "reinstate our app and apologize for the issues it caused us and our users." No, GitHub won't ever take responsibility for what you have between you and your projects/apps/companies users. Nor did they do so in this case. If I use service X for doing Y, and I write an email asking if it's OK that I upload 1000s of files every day, even if they say OK today, but then next week turn around and say "Nah", I'm still responsible for my users, who trust me and my team. Service X has no responsibility towards those users at all. It sucks though, I agree with that. > And no, you are not responsible for every 3rd party service you use. Some services are unavoidable, some services are just nice-to-have, but if you can't trust a service to perform its advertised function, it is the service's fault. Besides DNS and BGP I suppose, what services are "unavoidable" exactly? Git hosting isn't some arcane distributed network technology needing years of reading/experience to understand, the CLI ships with a web ui you can basically copy-paste to have your own Git hosting. I'd say you are responsible for everything that you use and depend on. And if you think "Ah, I'll just chunk 10K repositories at GitHub a day, they say it's fine" and then don't have any plan in case GitHub suddenly says it isn't OK, you are responsible for the falloff if shit hits the fan.
      - Mathnerd314 2y ago
        Well, the app store, for example. Sure, it is of course a good idea to comply with the app store policies to the extent possible, but ultimately there not much you can do to prevent Google or Apple saying "we don't like this app" and pulling it. For example with the UTM emulator. So how then can Google or Apple making such a decision be "your responsibility"? As another example, let's say you build a house in a hurricane-prone area. It's your responsibility to ensure the owner buys hurricane insurance, as mandated by law. It's not your responsibility to build a nuclear-bunker-grade house that is impervious to hurricanes. It is easy to point the finger and say "you should have thought of that", but in practice it is easier to deal with such catastrophes as they happen.
- arccy 2y ago
  sounds like a pretty sane thing to do: github protects the majority of their customers from instability caused by a few. unless you're a super important partner, the people on call might never have heard of your little app and just decide that it's the only safe thing to do to protect the reliability of the system.
- cudgy 2y ago
  If I read it correctly, it was a support person that provided them with assurance. Not an executive or vice president or manager or vp of sales. GitHub did not give them permission nor their approval; it was a single person in support department. Who relies on support people to determine the basis of their business when it’s obvious that they were concerned with the high usage rate and that it might cause problems for their customers?
  - kgeist 2y ago
    Oh yeah, support staff aren't always aware of everything. For example, with one of our latest features, we didn’t have time to add a UI option to disable the feature. The expectation was that support staff could disable it via their special admin panel upon user request. However, I accidentally discovered that when users asked to hide the feature, tech support told them it wasn’t possible! It turned out the tech support lead forgot to share that information with the team. As for the OP, they should have conducted load testing and implemented rate limits on their end, rather than blindly relying on someone’s word that GitHub was ready to handle all their product's load for free.

koreanguy 2y ago

[dead]

qeternity 2y ago

I don't really understand how a well funded startup like this, with something that is relatively trivial, yet critical to their product, decided to just shove it into GitHub.

remram 2y ago
Their product is the creation of Git repos. Putting it on the platform their customers want to use makes a lot of sense. They probably should have had a backup location from day 2 though, I agree. If nothing else, in case of a GitHub outage.
- diggan 2y ago
  > Their product is the creation of Git repos. Putting it on the platform their customers want to use makes a lot of sense. Maybe I read the landing page very wrong, but it seems to be a "app building toolkit" of some sorts? Not just "creation of git repos". They could have made the GitHub repository creation happen when the user does some action, instead of at the stage of "create app" which probably every single user does at least once, even people with no intention of actually building apps. Or better yet, offer their own viewer for Git repositories they themselves host. It's not overly difficult, and the `git` CLI tools even ship with a web UI you can take inspiration from.
  - d3nj4l 2y ago
    If you want a GitHub like UI Forgejo is FOSS too.
- cratermoon 2y ago
  What value does Lovable's product add, then?
  - Kwpolska 2y ago
    It doesn’t add value. It fills them with LLM-generated code.
- 2y ago

aimazon 2y ago

Many mistakes were made by Lovable that they could be berated for but on a more positive note, there is a lesson for us all: if you're doing something that you're worried about being problematic (e.g: creating a large volume of GitHub repositories) reaching out is a good thing but it is important to understand who you are reaching out to. GitHub is a huge organization, front-line support is not likely to have intimate knowledge of how exactly the acceptable usage policy is enforced nor the permission to make agreements. The key when reaching out is to find someone who has authority on the subject. Ideally, GitHub's front-line support would have escalated to the appropriate person/team but that isn't always possible (maybe they don't know who, maybe they're having a bad day and forgot). If the answer you get seems too convenient, it is probably not correct.

chrisgd 2y ago
If I reach out and they can’t direct me to the right solution, that doesn’t seem like something I need to continue to solve for them. Seems too onerous.
- gabeio 2y ago
  If it is the critical path for your project or worse for your job, seems like that’s just due diligence. > Seems to onerous. I would only say that about a non-critical personal project.
  - chrisgd 2y ago
    Fair points

V__ 2y ago

Besides the outage, isn't it kind of an insane setup to use GitHub to store every clients' project, especially with 10k new repos every day?

elicksaur 2y ago

Title is misleading. Clicked thinking this startup caused a GitHub outage. Getting a flag like this hardly “overloaded” anything. Alerts for these things usually trigger well below any actual risk to the system.

rikafurude21 2y ago

git is not github, and its kind of funny that the best these guys could come up with for storing git repos of text files was using github. any competent dev could come up with a solution in an afternoon. using github at all tells you quite a bit

temp8388344 2y ago

The same user who posted this (Henrik501) also posted a comment two days ago (their only HN comment so far) praising the Lovable team for their incident response: https://news.ycombinator.com/item?id=42646297 https://news.ycombinator.com/item?id=42646297 And now this post with an exaggerated title. Seems like they're shilling and trying to make Lovable sound like a product with such huge traction that it "even brought Github down". They keep making outlandish claims on social media too, like reaching $4m ARR in 5 weeks etc. This company is very suspicious.

hiatus 2y ago
The same username on Github follows only one person, Niklas Vatn, who appears to be a lovable employee. https://github.com/Henrik501?tab=following https://github.com/Henrik501?tab=following Looking more into it, Henrik Westerlund works on "growth" at lovable (via LinkedIn) and posted Lovable to Product Hunt https://www.producthunt.com/@henrik_westerlund https://www.producthunt.com/@henrik_westerlund. So it appears they are shilling.
- temp8388344 2y ago
  Pathetic company built on lies and shady "growth hacks"
kgeist 2y ago
>praising the Lovable team for their incident response For 8 hours, no one is aware the service is down. It then takes ~3 more days to fix it. One of their first decisions is to 'make as much noise as possible on social media' (?), and every step seems to create additional problems (corrupt repos etc.) Nothing appears to be well thought out, the blog post reads like they weren't ready for this at all, panicked and chaotic decisions without understanding the tech stack on a deeper level (race conditions, rate limits etc.) Not a lot of confidence in the team behind a project that looks like nothing more than a glue between an LLM and a storage backend.
elicksaur 2y ago
I like that HN is so minimal, but obvious stuff like this makes me want to write a browser extension that lets me custom tag accounts for my own notes.

CamouflagedKiwi 2y ago

315,000 repositories + 10,000 per day? They were obviously correctly concerned this wouldn't be able to go on forever, hence the pre-emptive email, and of course they got the response saying it was okay, but I really feel like this is the kind of thing that's too dangerous to leave your company sitting on because sooner or later they were going to be told "no". It feels too much like it's found a point of arbitrage in Github's ToS, and indeed ended up causing problems. I suppose they did respond pretty fast, but if I were them I'd have liked to have had the S3 option in my back pocket earlier. Maybe I'm just being too risk-averse here...

registeredcorn 2y ago
If nothing else, I was a little surprised they didn't (or at least didn't mention) having a fail over plan in place already. Seems like "Prepare for the worst, hope for the best" would have been the logical game plan.

Kwpolska 2y ago

The title ("Year old startup overloaded GitHub – Incident report") is a misrepresentation. A coding-LLM-as-a-service startup got banned by GitHub for abusing it by creating thousands of repositories.

deleted 2y ago

[deleted]

kgeist 2y ago

Interestingly, currently the site has this: >We're currently investigating issues. Please stay tuned until this error banner has been updated. When I try to create a new project, it says "An unknown error occurred with the code sandbox" Something about S3-backed repos didn't work out? UPD. "Under maintenance: Lovable is currently not able to reach the cloud provider for our previews, fly.io" Now it's fly.io's problem, not GitHub's