6 ms·

Same experience with Firebase. I just joined a team that has been using it. I've never worked with serverless before, and it boggles my mind how anyone thought

Same experience with Firebase. I just joined a team that has been using it. I've never worked with serverless before, and it boggles my mind how anyone thought it would be a good idea. The cold starts are horrendous. In one case, it's consistently taking about 7 seconds to return ~10K of data. I investigated the actual runtime of the function and it completes in about 20ms, so the only real bottleneck is the fucking cold start.

kaishiro 5y ago
I'm assuming by Firebase you mean Firebase Functions? We have a fairly complex infrastructure running off a combination of Firebase's RTDB, Firestore, and Google Cloud Functions and have never seen anywhere near what you're describing. Are you sure you're experiencing a 7 second "cold start", or is the invocation simply taking 7 seconds to run? Because the latter is far more easily explained.
- tenaciousDaniel 5y ago
  I've confirmed that the actual execution of the function itself takes ~30ms, and that the time to download is standard (~200ms). That only leaves the cold start; nothing else makes sense. edit: I even set up a node server on AppEngine itself, copied over the exact code to an endpoint, and it was taking 300-500ms tops (after the initial cold start of the AppEngine server).
earthboundkid 5y ago
7s cold start is weird. I think there has to be more to the story than that.
- alex_duf 5y ago
  I've seen 30s on AWS, so it's not that surprising. They have now improved it greatly though. And yet I still believe it's a great technology, as always it's a matter of putting it on the right use case. Message consumption from a queue or topic, low traffic and low criticality API are two great use cases.
  - zoul 5y ago
    Also, cases where you can cache the responses for some reasonable time and revalidate in the background.
  - deleted 5y ago
    [deleted]
  - tenaciousDaniel 5y ago
    Yeah I could see serverless being useful for those scenarios.
  - nuerow 5y ago
    > I've seen 30s on AWS No, you've seen 30s on a random implementation running on AWS. If you write your lambdas without knowing what you're doing then you can't blame the technology for being grossly misused by you. Case in point: developing AWS Lambdas for the JDK runtime. The bulk of the startup time is not the lambda at all but the way the lambda code is initialized. This means clients and auth and stuff. I've worked on JDK Lambdas where cold starts were close to 20s due to their choice of dependency injection framework (Guide, the bane of JDK lambdas) which was shaven down to half that number by simply migrating to Dagger. I've worked on other JDK lambdas which saw similar reductions in cold starts just by paying attention to how a Redis client was configured. Just keep in mind that cold start times represent the time it takes for your own code to initialize. This is not the lambda, but your own code. If you tell the lambda to needlessly run a lot of crap, you can't blame the lambda for actually doing what you set it to do.
    - tsimionescu 5y ago
      That sort of time could be easily reached with lambdas that require VPC access, where a new ENI needs to be provisioned for each lambda container. I don't think alive seen 30s, but could easily see 5-10s for this case. And since this is required to run an isolated DB that is not some other AWS service, it isn't that uncommon. I believe they have since improved start times in this scenario significantly more recently.
    - alex_duf 5y ago
- tenaciousDaniel 5y ago
  That's what I thought. We've spent weeks investigating this one function and tried everything that Firebase recommends in their docs. Nothing has worked.
- p0la 5y ago
  Same intuition. We run ~all our back on firebase cloud functions since Beta in 2017 and cold start has never been such a problem.
  - tenaciousDaniel 5y ago
    I'm extremely surprised to hear that. I know that there can be implementation differences, but on the level of application-code, this stuff is super simple. Create a javascript function then upload it. Not really much else to it, so I can't fathom what the difference is between your project and my own. We're not doing anything crazy, it's just a basic CRUD application with minimal data (entire DB is less than 100MB at this point). And yet we're seeing constant, constant lags of several seconds for almost every single request. I can't explain it.
ceras 5y ago
Not a fit for every scenario, but you can set a minimum number of instances for Firebase to keep ready. Should help reduce cold starts: https://firebase.google.com/docs/functions/tips#min https://firebase.google.com/docs/functions/tips#min
- tenaciousDaniel 5y ago
  Yeah I tried that, it doesn't work. Ultimately the provisioning algorithm is a black box.
  - gbourne 5y ago
    Why didn't the Firebase minInstances work for you? I found amazing performance benefits, but at a $ cost. I actually forgot[1] a function to set a minInstance and a user complained that this particular functionality was slow (compare to the rest of the site). However, it isn't cheap. You also want to be sure your code is optimized[2]. For example, don't require module packages unless the function needs it, else you're loading unnecessary packages. I usually set a global variable `let fetch;` and in the function that requires fetch initialize it `fetch = fetch ?? require("node-fetch);`. [1] https://www.ayrshare.com/a-firebase-cloud-functions-cold-start-solution/ https://www.ayrshare.com/a-firebase-cloud-functions-cold-sta... [2] https://firebase.google.com/docs/functions/tips https://firebase.google.com/docs/functions/tips
    - tenaciousDaniel 5y ago
      No idea why. Yeah we're trying all the usual techniques.

kaishiro 5y ago

I'm assuming by Firebase you mean Firebase Functions? We have a fairly complex infrastructure running off a combination of Firebase's RTDB, Firestore, and Google Cloud Functions and have never seen anywhere near what you're describing. Are you sure you're experiencing a 7 second "cold start", or is the invocation simply taking 7 seconds to run? Because the latter is far more easily explained.

tenaciousDaniel 5y ago
I've confirmed that the actual execution of the function itself takes ~30ms, and that the time to download is standard (~200ms). That only leaves the cold start; nothing else makes sense. edit: I even set up a node server on AppEngine itself, copied over the exact code to an endpoint, and it was taking 300-500ms tops (after the initial cold start of the AppEngine server).

earthboundkid 5y ago

7s cold start is weird. I think there has to be more to the story than that.

alex_duf 5y ago
I've seen 30s on AWS, so it's not that surprising. They have now improved it greatly though. And yet I still believe it's a great technology, as always it's a matter of putting it on the right use case. Message consumption from a queue or topic, low traffic and low criticality API are two great use cases.
- zoul 5y ago
  Also, cases where you can cache the responses for some reasonable time and revalidate in the background.
- deleted 5y ago
  [deleted]
- tenaciousDaniel 5y ago
  Yeah I could see serverless being useful for those scenarios.
- nuerow 5y ago
  > I've seen 30s on AWS No, you've seen 30s on a random implementation running on AWS. If you write your lambdas without knowing what you're doing then you can't blame the technology for being grossly misused by you. Case in point: developing AWS Lambdas for the JDK runtime. The bulk of the startup time is not the lambda at all but the way the lambda code is initialized. This means clients and auth and stuff. I've worked on JDK Lambdas where cold starts were close to 20s due to their choice of dependency injection framework (Guide, the bane of JDK lambdas) which was shaven down to half that number by simply migrating to Dagger. I've worked on other JDK lambdas which saw similar reductions in cold starts just by paying attention to how a Redis client was configured. Just keep in mind that cold start times represent the time it takes for your own code to initialize. This is not the lambda, but your own code. If you tell the lambda to needlessly run a lot of crap, you can't blame the lambda for actually doing what you set it to do.
  - tsimionescu 5y ago
    That sort of time could be easily reached with lambdas that require VPC access, where a new ENI needs to be provisioned for each lambda container. I don't think alive seen 30s, but could easily see 5-10s for this case. And since this is required to run an isolated DB that is not some other AWS service, it isn't that uncommon. I believe they have since improved start times in this scenario significantly more recently.
  - alex_duf 5y ago
tenaciousDaniel 5y ago
That's what I thought. We've spent weeks investigating this one function and tried everything that Firebase recommends in their docs. Nothing has worked.
p0la 5y ago
Same intuition. We run ~all our back on firebase cloud functions since Beta in 2017 and cold start has never been such a problem.
- tenaciousDaniel 5y ago
  I'm extremely surprised to hear that. I know that there can be implementation differences, but on the level of application-code, this stuff is super simple. Create a javascript function then upload it. Not really much else to it, so I can't fathom what the difference is between your project and my own. We're not doing anything crazy, it's just a basic CRUD application with minimal data (entire DB is less than 100MB at this point). And yet we're seeing constant, constant lags of several seconds for almost every single request. I can't explain it.

ceras 5y ago

Not a fit for every scenario, but you can set a minimum number of instances for Firebase to keep ready. Should help reduce cold starts: https://firebase.google.com/docs/functions/tips#min https://firebase.google.com/docs/functions/tips#min

tenaciousDaniel 5y ago
Yeah I tried that, it doesn't work. Ultimately the provisioning algorithm is a black box.
- gbourne 5y ago
  Why didn't the Firebase minInstances work for you? I found amazing performance benefits, but at a $ cost. I actually forgot[1] a function to set a minInstance and a user complained that this particular functionality was slow (compare to the rest of the site). However, it isn't cheap. You also want to be sure your code is optimized[2]. For example, don't require module packages unless the function needs it, else you're loading unnecessary packages. I usually set a global variable `let fetch;` and in the function that requires fetch initialize it `fetch = fetch ?? require("node-fetch);`. [1] https://www.ayrshare.com/a-firebase-cloud-functions-cold-start-solution/ https://www.ayrshare.com/a-firebase-cloud-functions-cold-sta... [2] https://firebase.google.com/docs/functions/tips https://firebase.google.com/docs/functions/tips
  - tenaciousDaniel 5y ago
    No idea why. Yeah we're trying all the usual techniques.