5 ms·

tl;dr 20 clusters, one of which (the one with a million writes) records only gps data, running an application in a container in an api-driven os virtualization

tl;dr 20 clusters, one of which (the one with a million writes) records only gps data, running an application in a container in an api-driven os virtualization package -- I like the part where 10% performance overhead is not a big deal while at the same time making a big deal about how mesos can save you 30% performance overhead. And notice how at first they had a group of clusters for completely separate things like API, storage, etc, but now with mesos they program against an entire datacenter by using a different specific API for separate things like APIs, storage, etc. Much... better. Also now with mesos they can run different kinds of services on the same machine. Kind of like containers. Kind of like separate applications on any kind of machine at all. "If, for example, one services uses a lot of CPU it matches well with a service that uses a lot of storage or memory, then these two services can be efficiently run on the same server. Machine utilization goes up." Because if I have a machine pegging 90% CPU, I totally want an i/o heavy job running on that machine, because interrupts are, like, imaginary, man. Yeah, i'm sure the machine's utilization goes up... right up to triple-digit load. "Agility is more important than performance." So the ability to automate rolling out a virtual machine and application in as abstracted a way as possible is more important than their 1-million-writes-per-second goal. I think I know why they don't find 10% loss to be a problem now... they can just keep spinning up machines until they reach their performance goal. 150 machines per DC to store ride-sharing data seems a bit much, but maybe there's a lot more going on behind the scenes than the few features on their app.

ddorian43 10y ago
All those microservices add overhead. They read that google does it this way (multiple apps in a server) and now they all must do it to fully utilize aws vps.
icebraining 10y ago
Because if I have a machine pegging 90% CPU, I totally want an i/o heavy job running on that machine, because interrupts are, like, imaginary, man. Yeah, i'm sure the machine's utilization goes up... right up to triple-digit load. Honest question: can't you dedicate some cores to handling the interrupts, leaving the others for the other processes?
- user5994461 10y ago
  As a matter of fact, yes... that's a "common" very advanced optimization. I've mostly seen it for load balancers. Pin the network card interrupt on one core and the load balancer on another.
  - sitkack 10y ago
    Or turn interrupts completely off and poll. If you always have work in the queue, polling is less intensive, not more.
    - user5994461 10y ago
      That makes no sensen. Polling means the core is running at 100% all the time.
      - ddorian43 10y ago
        Yes, but it's actually faster. Source: scylladb.
        user5994461 10y ago
        Still doesn't make sense in the context of a network card. There must be an interrupt to pre-empt the system or there's gonna be packet loss.
        SEJeff 10y ago
        Actually it does, and is quite measurable via standard tools like netperf. I have done this, but only because I've worked in Finance as a Linux monkey for the previous 9.5-ish years. I specialize in Linux tuning and reading far too much Linux kernel source code. That and big distributed systems. In fact, modern kernels support this quite extensively and the RHEL7 documentation[1] has great tips on it for someone new to it. You can tune an interface for max throughput via coalescing[2] on Linux via ethtool -C to change and ethtool -c to view current coalescing. Setting interrupt per packet helps with latency for certain workload types in addition to SO_BUSY_POLL[3] and the global or per-interface busy polling sysctl. However, interrupt per packet can trivially overwhelm CPUs if you don't isolate those cpus using things like isolcpus= on the linux grub commandline or cpu_exclusive=1 in a cpuset. RFS/RSS and NICs with multiple receive queues make this much easier to tune. It is guaranteed to use more power, but you can do idle=poll[4] on the Linux kernel boot command line to stay in the highest C state and help with network latency. You don't really always need an interrupt either as apps can DMA directly to and from the NIC <---> Application bypassing the CPU. How do you think RDMA works at a lower level? Finally, Linux "kernel bypass" networking aka 100% userspace tcp/ip stacks can be written to be truly interrupt-less take a look at Mellanox's VMA or Solar Flare's openonload. Heck, the default openonload config is interrupt-less. The more you know :) [1] https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Performance_Tuning_Guide/sect-Red_Hat_Enterprise_Linux-Performance_Tuning_Guide-Networking-Configuration_tools.html#sect-Red_Hat_Enterprise_Linux-Performance_Tuning_Guide-Configuration_tools-Configuring_interrupt_queues https://access.redhat.com/documentation/en-US/Red_Hat_Enterp... [2] https://en.wikipedia.org/wiki/Interrupt_coalescing https://en.wikipedia.org/wiki/Interrupt_coalescing [3] http://man7.org/linux/man-pages/man7/socket.7.html http://man7.org/linux/man-pages/man7/socket.7.html [4] https://access.redhat.com/articles/65410 https://access.redhat.com/articles/65410 EDIT: This is a fantastic and simple overview with actual netperf results showing how this helps with latency: http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/open-source-kernel-enhancements-paper.pdf http://www.intel.com/content/dam/www/public/us/en/documents/...
tveita 10y ago
> Because if I have a machine pegging 90% CPU, I totally want an i/o heavy job running on that machine, because interrupts are, like, imaginary, man. Yeah, i'm sure the machine's utilization goes up... right up to triple-digit load. They're clearly tracking their own latency and reliability and comparing it to their availability goals - there's numbers right in the article, so I would think they would notice this and handle it somehow. So I guess either they're lying in the presentation, or your sarcasm is targeted towards a problem you conjured up on you end for your personal satisfaction.

ddorian43 10y ago

All those microservices add overhead. They read that google does it this way (multiple apps in a server) and now they all must do it to fully utilize aws vps.

icebraining 10y ago

Because if I have a machine pegging 90% CPU, I totally want an i/o heavy job running on that machine, because interrupts are, like, imaginary, man. Yeah, i'm sure the machine's utilization goes up... right up to triple-digit load. Honest question: can't you dedicate some cores to handling the interrupts, leaving the others for the other processes?

user5994461 10y ago
As a matter of fact, yes... that's a "common" very advanced optimization. I've mostly seen it for load balancers. Pin the network card interrupt on one core and the load balancer on another.
- sitkack 10y ago
  Or turn interrupts completely off and poll. If you always have work in the queue, polling is less intensive, not more.
  - user5994461 10y ago
    That makes no sensen. Polling means the core is running at 100% all the time.
    - ddorian43 10y ago
      Yes, but it's actually faster. Source: scylladb.
      - user5994461 10y ago
        Still doesn't make sense in the context of a network card. There must be an interrupt to pre-empt the system or there's gonna be packet loss.
        SEJeff 10y ago
        Actually it does, and is quite measurable via standard tools like netperf. I have done this, but only because I've worked in Finance as a Linux monkey for the previous 9.5-ish years. I specialize in Linux tuning and reading far too much Linux kernel source code. That and big distributed systems. In fact, modern kernels support this quite extensively and the RHEL7 documentation[1] has great tips on it for someone new to it. You can tune an interface for max throughput via coalescing[2] on Linux via ethtool -C to change and ethtool -c to view current coalescing. Setting interrupt per packet helps with latency for certain workload types in addition to SO_BUSY_POLL[3] and the global or per-interface busy polling sysctl. However, interrupt per packet can trivially overwhelm CPUs if you don't isolate those cpus using things like isolcpus= on the linux grub commandline or cpu_exclusive=1 in a cpuset. RFS/RSS and NICs with multiple receive queues make this much easier to tune. It is guaranteed to use more power, but you can do idle=poll[4] on the Linux kernel boot command line to stay in the highest C state and help with network latency. You don't really always need an interrupt either as apps can DMA directly to and from the NIC <---> Application bypassing the CPU. How do you think RDMA works at a lower level? Finally, Linux "kernel bypass" networking aka 100% userspace tcp/ip stacks can be written to be truly interrupt-less take a look at Mellanox's VMA or Solar Flare's openonload. Heck, the default openonload config is interrupt-less. The more you know :) [1] https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Performance_Tuning_Guide/sect-Red_Hat_Enterprise_Linux-Performance_Tuning_Guide-Networking-Configuration_tools.html#sect-Red_Hat_Enterprise_Linux-Performance_Tuning_Guide-Configuration_tools-Configuring_interrupt_queues https://access.redhat.com/documentation/en-US/Red_Hat_Enterp... [2] https://en.wikipedia.org/wiki/Interrupt_coalescing https://en.wikipedia.org/wiki/Interrupt_coalescing [3] http://man7.org/linux/man-pages/man7/socket.7.html http://man7.org/linux/man-pages/man7/socket.7.html [4] https://access.redhat.com/articles/65410 https://access.redhat.com/articles/65410 EDIT: This is a fantastic and simple overview with actual netperf results showing how this helps with latency: http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/open-source-kernel-enhancements-paper.pdf http://www.intel.com/content/dam/www/public/us/en/documents/...

tveita 10y ago

> Because if I have a machine pegging 90% CPU, I totally want an i/o heavy job running on that machine, because interrupts are, like, imaginary, man. Yeah, i'm sure the machine's utilization goes up... right up to triple-digit load. They're clearly tracking their own latency and reliability and comparing it to their availability goals - there's numbers right in the article, so I would think they would notice this and handle it somehow. So I guess either they're lying in the presentation, or your sarcasm is targeted towards a problem you conjured up on you end for your personal satisfaction.