3 ms·

I have some related questions about the latency: 1. Where is the latency variance coming from? (Garbage collection? Network stack?) 2. Between which two points

I have some related questions about the latency: 1. Where is the latency variance coming from? (Garbage collection? Network stack?) 2. Between which two points are you measuring it? 3. What would the variance look like if using two processing nodes on different machines? 4. Is it possible to use something like LTTng with Pony to understand latency distribution better? All-in-all a pretty cool project!

spooneybarger 9y ago
1. I'm not sure what you are reffering to in terms of latency variance. What variances are you referring to? 2. We measure from the moment that we ingest the message to when the message is done being processed. If a Wallarooa application was receiving data over TCP and sending out new data also over TCP, it would be from the time that we read in the data to when we write the message out onto the outgoing socket. If a message were to be filtered and not creaye any output then the end point would be once its filtered. 3. We currently use TCP between nodes. The latency that gets added with a network hop in part depends on how much traffic is going over that hop. If you were processing a couple hundred thousand messages or so a seond, that additional latency is general 500 micros to 2 milliseconds. 4. I suppose. I don't believe anyone has tried. The metrics that we provide include not just end to end latencies but also the amount of time spent in a given computation, going over a given network hop and similiar insight. Wallaroo has a fairly low overhead so most time in an application is spent in user computations (the latency of which Wallaroo can't control) and any network hops. Thank you!
- snowAbstraction 9y ago
  Thanks for your answers. If you feel up tackling my first question, I have tried to clarify it: I was referring to statistical variance of latencies listed in the blog post: 50% 95% 99% 99.9% 99.99% 66µs .5ms .5ms .5ms 1 ms More concretely, what is the difference between when ingesting and processing an event takes 1 ms and it takes 66µs? Do you know what causes this?
  - spooneybarger 9y ago
    We read in large chunks of messages at a time and start processing them. Some messages wait longer in internal queues for CPU time to become available to process them. The benchmark we use is intentionally very light computationally. We did that because, if we had a heavy computation then the overhead from Wallaroo would be hidden.

spooneybarger 9y ago

1. I'm not sure what you are reffering to in terms of latency variance. What variances are you referring to? 2. We measure from the moment that we ingest the message to when the message is done being processed. If a Wallarooa application was receiving data over TCP and sending out new data also over TCP, it would be from the time that we read in the data to when we write the message out onto the outgoing socket. If a message were to be filtered and not creaye any output then the end point would be once its filtered. 3. We currently use TCP between nodes. The latency that gets added with a network hop in part depends on how much traffic is going over that hop. If you were processing a couple hundred thousand messages or so a seond, that additional latency is general 500 micros to 2 milliseconds. 4. I suppose. I don't believe anyone has tried. The metrics that we provide include not just end to end latencies but also the amount of time spent in a given computation, going over a given network hop and similiar insight. Wallaroo has a fairly low overhead so most time in an application is spent in user computations (the latency of which Wallaroo can't control) and any network hops. Thank you!

snowAbstraction 9y ago
Thanks for your answers. If you feel up tackling my first question, I have tried to clarify it: I was referring to statistical variance of latencies listed in the blog post: 50% 95% 99% 99.9% 99.99% 66µs .5ms .5ms .5ms 1 ms More concretely, what is the difference between when ingesting and processing an event takes 1 ms and it takes 66µs? Do you know what causes this?
- spooneybarger 9y ago
  We read in large chunks of messages at a time and start processing them. Some messages wait longer in internal queues for CPU time to become available to process them. The benchmark we use is intentionally very light computationally. We did that because, if we had a heavy computation then the overhead from Wallaroo would be hidden.