4 ms·

latency - how do you build a platform with high performance if every customer request has a network of n services to traverse for every operation?

by platz 12y ago

Denzel 12y ago
You'll want to issue requests concurrently, use timeouts liberally, and implement fallbacks to cached/calculated data in the event that one of the services fails or misbehaves. Those are just a few guiding principles that will help when building a SOA (service-oriented architecture). Netflix has a great blog post on the structure of their platform [1]. They even open-sourced Hystrix [2], the library they use internally. [1] http://techblog.netflix.com/2012/02/fault-tolerance-in-high-volume.html http://techblog.netflix.com/2012/02/fault-tolerance-in-high-... [2] https://github.com/Netflix/Hystrix https://github.com/Netflix/Hystrix
yureka 12y ago
There are a lot of simple techniques that can be used to optimize services in a SOA (e.g. caching). Since each service follows a request/response format, it becomes easier to see where the majority of a request lifespan spends its time to help eliminate bottle necks. You'd be surprised how low the overall latency for a request can be in large companies like Amazon (< 100ms)
jfoutz 12y ago
what's your budget? Is 30ms fast enough to put a widget in a shopping cart? Let's just assert 15ms from your server to the customer, you've got 15ms to play with. Given this https://gist.github.com/jboner/2841832 https://gist.github.com/jboner/2841832 let's assume you get around 5ms for each service, say a couple big disk reads and some filtering. So if n=3, it doesn't seem to bad. How much can you do in parallel? Stuff like search, it seems like you could partition quite a bit hit a bunch of services (10?)in the first 5ms, then spend the remaining 10ms sorting results, just send your best effort. I'm (of course) assuming a customer request means there's a customer sitting there waiting and 30ms is good enough. Won't work for a game. Won't work for HFT. I guess you'd use the standard stuff, many layers of caching, efficient implementation of the specific services, good algorithms. Heck, you could play a game with docker and run the whole infrastructure on 1 big machine, i think those local sockets would be a lot quicker that the 500,000 ns of a datacenter read. I mean, if an l2 cache miss is a disaster for you, clearly this won't do anything for you. But how fast is fast enough? do you need to hit that target every single time(real time)? Can you be late 1 in a million requests? Are you sort of dismissing the idea out of hand or are you (aside from the silly title) really thinking through the consequences of implementing an application this way?
- platz 12y ago
  not dismissing - 30ms would be stellar. Although I think that sets the bar very high. Careful engineering can get it that low, but my experience is that some platforms aren't that fast, especially if you're not building a greenfield set of services from the ground up.
asinglenet 12y ago
If speed is your biggest concern, it is likely that you will build a system that has a lot of services which each have a shallow stack. Each service would not go very deep into the service stack before hitting the data it was looking for. PHP, for example, almost always works this way. Maybe the data would be in a cache, as suggested by other commenters here. You have more places to put caches if you think about your system as multiple services. If you are making a big system, you probably would not want to try to put all of the components in one monolithic place. I argued that this makes testing and release more complicated (and slower). But, I suspect it also affects runtime speed by making the stack deeper. You would be doing more switching to get your requests to the right components.

latency - how do you build a platform with high performance if every customer request has a network of n services to traverse for every operation?

by platz 12y ago

latency - how do you build a platform with high performance if every customer request has a network of n services to traverse for every operation?

Denzel 12y ago

You'll want to issue requests concurrently, use timeouts liberally, and implement fallbacks to cached/calculated data in the event that one of the services fails or misbehaves. Those are just a few guiding principles that will help when building a SOA (service-oriented architecture). Netflix has a great blog post on the structure of their platform [1]. They even open-sourced Hystrix [2], the library they use internally. [1] http://techblog.netflix.com/2012/02/fault-tolerance-in-high-volume.html http://techblog.netflix.com/2012/02/fault-tolerance-in-high-... [2] https://github.com/Netflix/Hystrix https://github.com/Netflix/Hystrix

yureka 12y ago

There are a lot of simple techniques that can be used to optimize services in a SOA (e.g. caching). Since each service follows a request/response format, it becomes easier to see where the majority of a request lifespan spends its time to help eliminate bottle necks. You'd be surprised how low the overall latency for a request can be in large companies like Amazon (< 100ms)

jfoutz 12y ago

what's your budget? Is 30ms fast enough to put a widget in a shopping cart? Let's just assert 15ms from your server to the customer, you've got 15ms to play with. Given this https://gist.github.com/jboner/2841832 https://gist.github.com/jboner/2841832 let's assume you get around 5ms for each service, say a couple big disk reads and some filtering. So if n=3, it doesn't seem to bad. How much can you do in parallel? Stuff like search, it seems like you could partition quite a bit hit a bunch of services (10?)in the first 5ms, then spend the remaining 10ms sorting results, just send your best effort. I'm (of course) assuming a customer request means there's a customer sitting there waiting and 30ms is good enough. Won't work for a game. Won't work for HFT. I guess you'd use the standard stuff, many layers of caching, efficient implementation of the specific services, good algorithms. Heck, you could play a game with docker and run the whole infrastructure on 1 big machine, i think those local sockets would be a lot quicker that the 500,000 ns of a datacenter read. I mean, if an l2 cache miss is a disaster for you, clearly this won't do anything for you. But how fast is fast enough? do you need to hit that target every single time(real time)? Can you be late 1 in a million requests? Are you sort of dismissing the idea out of hand or are you (aside from the silly title) really thinking through the consequences of implementing an application this way?

platz 12y ago
not dismissing - 30ms would be stellar. Although I think that sets the bar very high. Careful engineering can get it that low, but my experience is that some platforms aren't that fast, especially if you're not building a greenfield set of services from the ground up.

asinglenet 12y ago

If speed is your biggest concern, it is likely that you will build a system that has a lot of services which each have a shallow stack. Each service would not go very deep into the service stack before hitting the data it was looking for. PHP, for example, almost always works this way. Maybe the data would be in a cache, as suggested by other commenters here. You have more places to put caches if you think about your system as multiple services. If you are making a big system, you probably would not want to try to put all of the components in one monolithic place. I argued that this makes testing and release more complicated (and slower). But, I suspect it also affects runtime speed by making the stack deeper. You would be doing more switching to get your requests to the right components.