4 ms·

Wow, I didn't expect my project to get posted here. Thanks a lot! I can see that many people are drawing comparisons with tc/netmem, which speedbump is not int

by kffl 4y ago

Wow, I didn't expect my project to get posted here. Thanks a lot! I can see that many people are drawing comparisons with tc/netmem, which speedbump is not intended to replace, so I think it's best if I provide some additional context info regarding the use case that prompted me to develop this project. When setting up application metrics collection and visualization (i.e. via Prometheus + Grafana), I've often found myself trying to introduce artificial latency within the instrumented system for the purpose of generating more interesting timeseries data to test a given monitoring solution. Even when running load tests against an instrumented system, the data plotted on Grafana dashboards was often rather boring, making it difficult to catch bugs in PromQL queries due to lack of immediate visual feedback. I figured that one way of adding predictable variability to the instrumented application’s metrics, would be to introduce variable latency between it and its upstream services (i.e. a databases, message brokers or other services called synchronously). Let's say that you have instrumented your app using Prometheus client so that it collects latency of DB queries in a histogram and that you are now building a Grafana dashboard to visualize these metrics. If you knew that the DB query latency over time should form a sine wave with a period of 2 mins and amplitude of 10 ms, it would be much easier to validate the correctness of metrics collection and visualization (you would know what to look for on the latency histogram/grah). Speedbump allows me to achieve just that by introducing latency that is variable, yet predictable over time.

spockz 4y ago
Are you aware of a tool that supports or planning to add support for more advanced delays? E.g. let through the http response headers and then slowly drizzly the content? Or let the last byte be delayed? What about package drop at a certain point in the response? I’ve found some interesting behaviour in our applications when delay isn’t uniform during the response.
- kffl 4y ago
  That's an interesting use case. I am not aware of any tools that would allow for such content-aware proxying of HTTP traffic with artificial latency introduced in the response headers content parts separately. As of now, speedbump is protocol agnostic - it just proxies the data over TCP without analyzing it (I've used it to proxy HTTP, Postgres, Mongo and AMQP traffic). While the idea of delaying the last byte seems promising at the first glance, especially in the context of retaining protocol agnosticism, the issue is that the proxy doesn't really know which byte is the last one (unless it actually understands the traffic flowing through it - i.e. detecting end of the HTTP response). EDIT: You may want to look into echo-nginx-module, as it allows for adding timeouts between output flushed to the client. Perhaps it could be used in conjunction with NGINX's proxy module.
  - spockz 4y ago
    Thanks, sidecars, like envoy and linkerd-proxy, already do protocol detection even if it is just for metrics. Maybe their stack can be augmented to do latency and fault stimulation. For plain http nginx might be an option indeed.

by kffl 4y ago

spockz 4y ago

Are you aware of a tool that supports or planning to add support for more advanced delays? E.g. let through the http response headers and then slowly drizzly the content? Or let the last byte be delayed? What about package drop at a certain point in the response? I’ve found some interesting behaviour in our applications when delay isn’t uniform during the response.

kffl 4y ago
That's an interesting use case. I am not aware of any tools that would allow for such content-aware proxying of HTTP traffic with artificial latency introduced in the response headers content parts separately. As of now, speedbump is protocol agnostic - it just proxies the data over TCP without analyzing it (I've used it to proxy HTTP, Postgres, Mongo and AMQP traffic). While the idea of delaying the last byte seems promising at the first glance, especially in the context of retaining protocol agnosticism, the issue is that the proxy doesn't really know which byte is the last one (unless it actually understands the traffic flowing through it - i.e. detecting end of the HTTP response). EDIT: You may want to look into echo-nginx-module, as it allows for adding timeouts between output flushed to the client. Perhaps it could be used in conjunction with NGINX's proxy module.
- spockz 4y ago
  Thanks, sidecars, like envoy and linkerd-proxy, already do protocol detection even if it is just for metrics. Maybe their stack can be augmented to do latency and fault stimulation. For plain http nginx might be an option indeed.