41 ms·

Lessons Operating a Serverless-Like Platform at Netflix, Part 2

178 points by diab0lic 9y ago

sparrish 9y ago
"These fine grained units also meant that the composition of the final application was much more distributed. ... As long as business and client metrics remained unaffected, per unit health was ignored." This was my big takeaway. I think this is the trajectory for monitoring in general as services become increasingly more distributed.
- eikenberry 9y ago
  This isn't so much a unique feature of serverless though, any self-healing cluster based system is better monitored this way.
cagenut 9y ago
I would be curious to see more detail about their warmup handler practices. Did they tune the JVM for it or is there some way to deterministically invoke the right JIT'ing.
- techwizrd 9y ago
  I'm not sure what Netflix is doing, but Alibaba has a neat JWarmup feature[0] in their AJDK that warms up the JVM and JITs based on recorded profiling data. 0: https://jcp.org/aboutJava/communityprocess/ec-public/materials/2017-02-14/Java_at_Alibaba.pdf https://jcp.org/aboutJava/communityprocess/ec-public/materia...
  - ibejoeb 9y ago
    Very neat, thanks. I didn't know they ran a custom jvm. About to go down a rabbit hole.
- mark242 9y ago
  The easiest thing to do on AWS is just fire off a CloudWatch event. When your Lambda function gets updated, or even on a set schedule, you can invoke your function with whatever arguments you need. Presumably you could cook in a method that warmed everything up, and call that method.
  - abalone 9y ago
    That's fine for functions that go cold, but what about additional containers that spin up when usage spikes? Don't those also suffer from cold start delays? I never hear people talk about that.
    - abritinthebay 9y ago
      Depends on the container/instances. There are cold start delays but you can get it down to a minute or two if you really optimize (but... well, most people don't need to).
    - wanghq 9y ago
      Those will still have cold start delay. A simple warm-up script only solves 0-to-1 problem, but not 1-to-n problem.
      - sozal 9y ago
        You may have a look at OpsGenie's Sirocco https://github.com/opsgenie/sirocco/tree/master/sirocco-warmup https://github.com/opsgenie/sirocco/tree/master/sirocco-warm...
        wanghq 9y ago
        Awesome! Thanks.
    - cnj 9y ago
      Yes, they also suffer from cold starts. When people talk about it, they usually measure the percentage of cold-started function calls, which mostly ends up being around 0.1% of all calls. In a sense, you have to have a good amount of baseline traffic to end up with that problem, which reduces the percentual impact. So the problem is less pronounced than with a function that is very rarely run, where a high percentage of function calls are cold. (In theory, the worst case where a CloudWatch trigger would be mostly useless is that a function is totally idle and removed from warm-state, and then gets a huge concurrent spike, is idle again, gets a spike etc. But that is not a realistic traffic pattern, I assume. It would also break with existing auto-scaling that would be even slower with bringing up new instances.)
kolanos 9y ago
If anyone is interested in learning more about serverless, AWS lambda, etc. There's a webinar this Thursday: https://read.iopipe.com/announcing-the-things-to-know-about-lambda-webinar-eb8a42266301 https://read.iopipe.com/announcing-the-things-to-know-about-...
mirko22 9y ago
I have hard time understanding "Serverless" part... It sounds cool, but there are bunch of servers there they are using. As far as I can manage to understand that name only has to do with pricing rather then actual servers, but then why is it not called something like "cheap" or "pay as you use"? Maybe I am missing the point here, but this is just a buzzword? (nano-services being the other one, much like micro-services, do we have pico-ones?)
- jasonlotito 9y ago
  It's not just a buzzword (though some might play it off as one), but the reality is, serverless has a meaning. And it's not that there aren't servers that are there to run your code. Rather, your code can be set to run in production and not be running on any server. It's just not running. It's only running when it needs to run. Another system sits there and when it has an event that comes in that needs to run your code, it starts up your code, passes your code the details, and lets your code run. Once that's done, your code is finished, and it shuts down. This is different from having code running on a server, or deploying your code to a server where it runs. You are paying for the server, regardless of whether it runs. That's what it means.
  - buro9 9y ago
    Serverless: 1) The code comes into existence when it is called, and ceases to exist after the request is handled 2) The server on which this happens is managed by someone else. To all intents and purposes there is no server on a logical diagram of your application
    - jasonlotito 9y ago
      Yep. Much more succinct than my explanation. Thanks!
      - buro9 9y ago
        No, thank you. My understanding was vague until I read your comment, I was just summing up that new understanding.
cromwellian 9y ago
Seems like the extreme version of this would be a simple annotation in client code to run it on the server, like the old <script runat...> in ASP, e.g. Java pseudo code @RunAt(Platform.SERVER) public Observable<List<Customer>> findCustomersByName(String name) { return // code to query DB and return a list of customers over the wire } // on client public void searchByCustomer(String name) { // findCustomersByName() and bind stream to UI } The idea being that the RPC is automatic here. The client code makes an async call, constructs an observable around deserialized result. The server code is pulled out by and spun up as a cloud function on demand. Of course this only makes sense if the code needs to run on the server (e.g. the results are big and need to run a lot of code, or need to run code that can't be trusted on the client).
- dualogy 9y ago
  > like the old <script runat...> in ASP Good memories! I remember how neat I found that as a script-kid back in the pre-.NET, VBScript-based ASP 3.0 days some 17 years ago. (MS Personal Web Server for the Win! The QBasic of the early web age.) But there wasn't any code-sharing and it was always clearly server-code, not client-code. Let alone the fact that only IE could run VBScript on the client..

178 points by diab0lic 9y ago

"These fine grained units also meant that the composition of the final application was much more distributed. ... As long as business and client metrics remained unaffected, per unit health was ignored." This was my big takeaway. I think this is the trajectory for monitoring in general as services become increasingly more distributed.

eikenberry 9y ago
This isn't so much a unique feature of serverless though, any self-healing cluster based system is better monitored this way.

cagenut 9y ago

I would be curious to see more detail about their warmup handler practices. Did they tune the JVM for it or is there some way to deterministically invoke the right JIT'ing.

techwizrd 9y ago
I'm not sure what Netflix is doing, but Alibaba has a neat JWarmup feature[0] in their AJDK that warms up the JVM and JITs based on recorded profiling data. 0: https://jcp.org/aboutJava/communityprocess/ec-public/materials/2017-02-14/Java_at_Alibaba.pdf https://jcp.org/aboutJava/communityprocess/ec-public/materia...
- ibejoeb 9y ago
  Very neat, thanks. I didn't know they ran a custom jvm. About to go down a rabbit hole.
mark242 9y ago
The easiest thing to do on AWS is just fire off a CloudWatch event. When your Lambda function gets updated, or even on a set schedule, you can invoke your function with whatever arguments you need. Presumably you could cook in a method that warmed everything up, and call that method.
- abalone 9y ago
  That's fine for functions that go cold, but what about additional containers that spin up when usage spikes? Don't those also suffer from cold start delays? I never hear people talk about that.
  - abritinthebay 9y ago
    Depends on the container/instances. There are cold start delays but you can get it down to a minute or two if you really optimize (but... well, most people don't need to).
  - wanghq 9y ago
    Those will still have cold start delay. A simple warm-up script only solves 0-to-1 problem, but not 1-to-n problem.
    - sozal 9y ago
      You may have a look at OpsGenie's Sirocco https://github.com/opsgenie/sirocco/tree/master/sirocco-warmup https://github.com/opsgenie/sirocco/tree/master/sirocco-warm...
      - wanghq 9y ago
        Awesome! Thanks.
  - cnj 9y ago
    Yes, they also suffer from cold starts. When people talk about it, they usually measure the percentage of cold-started function calls, which mostly ends up being around 0.1% of all calls. In a sense, you have to have a good amount of baseline traffic to end up with that problem, which reduces the percentual impact. So the problem is less pronounced than with a function that is very rarely run, where a high percentage of function calls are cold. (In theory, the worst case where a CloudWatch trigger would be mostly useless is that a function is totally idle and removed from warm-state, and then gets a huge concurrent spike, is idle again, gets a spike etc. But that is not a realistic traffic pattern, I assume. It would also break with existing auto-scaling that would be even slower with bringing up new instances.)

kolanos 9y ago

If anyone is interested in learning more about serverless, AWS lambda, etc. There's a webinar this Thursday: https://read.iopipe.com/announcing-the-things-to-know-about-lambda-webinar-eb8a42266301 https://read.iopipe.com/announcing-the-things-to-know-about-...

mirko22 9y ago

I have hard time understanding "Serverless" part... It sounds cool, but there are bunch of servers there they are using. As far as I can manage to understand that name only has to do with pricing rather then actual servers, but then why is it not called something like "cheap" or "pay as you use"? Maybe I am missing the point here, but this is just a buzzword? (nano-services being the other one, much like micro-services, do we have pico-ones?)

jasonlotito 9y ago
It's not just a buzzword (though some might play it off as one), but the reality is, serverless has a meaning. And it's not that there aren't servers that are there to run your code. Rather, your code can be set to run in production and not be running on any server. It's just not running. It's only running when it needs to run. Another system sits there and when it has an event that comes in that needs to run your code, it starts up your code, passes your code the details, and lets your code run. Once that's done, your code is finished, and it shuts down. This is different from having code running on a server, or deploying your code to a server where it runs. You are paying for the server, regardless of whether it runs. That's what it means.
- buro9 9y ago
  Serverless: 1) The code comes into existence when it is called, and ceases to exist after the request is handled 2) The server on which this happens is managed by someone else. To all intents and purposes there is no server on a logical diagram of your application
  - jasonlotito 9y ago
    Yep. Much more succinct than my explanation. Thanks!
    - buro9 9y ago
      No, thank you. My understanding was vague until I read your comment, I was just summing up that new understanding.

cromwellian 9y ago

Seems like the extreme version of this would be a simple annotation in client code to run it on the server, like the old <script runat...> in ASP, e.g. Java pseudo code @RunAt(Platform.SERVER) public Observable<List<Customer>> findCustomersByName(String name) { return // code to query DB and return a list of customers over the wire } // on client public void searchByCustomer(String name) { // findCustomersByName() and bind stream to UI } The idea being that the RPC is automatic here. The client code makes an async call, constructs an observable around deserialized result. The server code is pulled out by and spun up as a cloud function on demand. Of course this only makes sense if the code needs to run on the server (e.g. the results are big and need to run a lot of code, or need to run code that can't be trusted on the client).

dualogy 9y ago
> like the old <script runat...> in ASP Good memories! I remember how neat I found that as a script-kid back in the pre-.NET, VBScript-based ASP 3.0 days some 17 years ago. (MS Personal Web Server for the Win! The QBasic of the early web age.) But there wasn't any code-sharing and it was always clearly server-code, not client-code. Let alone the fact that only IE could run VBScript on the client..