4 ms·

Some time we'll get round to writing this up but there's a small customer of Cloudflare that gets a very high HTTP requests per second rate. It's a simple servi

Some time we'll get round to writing this up but there's a small customer of Cloudflare that gets a very high HTTP requests per second rate. It's a simple service (bit like a "what's my IP address" but not that) and it turns out that a quite popular hardware device hard-coded requests to this service and doesn't appear to cache the results and so it asks over and over and over again for the same information. We've contacted the manufacturer and I think it's been patched but the life time of installed equipment is long... Yesterday: over a billion HTTP requests...

Thiez 6y ago
Are you not tempted to just block the requests from these devices, and let the manufacturer take the loss? I imagine serving all those requests is costing real money.
- jgrahamc 6y ago
  It's not a TOS violation. It did cause us some ops pain at one point (they were getting hit with > 50,000rps concentrated in certain locations). But one of the reasons Cloudflare can operate our service is we have 3.2 million customers who are doing all sorts of stuff. We get so much stronger from that great variety of traffic.
  - pests 6y ago
    Right. I'm sure you can explain more but I've read due to peering agreements and the like having a lot of one-directional traffic can be a good thing.
RyJones 6y ago
This is similar to how Qualcomm's DNS servers got knocked off the air. An OEM shipped an update which would query a development TURN server we were running - once per connection, over millions of devices. It was a crazy day.
DaniloDias 6y ago
Could you be more explicit on the nature of the service? I’d like to explore mechanisms for tests that detect IoT devices that misbehave this way (and other ways as well). Your anecdote sounds interesting. Is it unrelated to time servers? Unrelated to internet connectivity tests?
- jgrahamc 6y ago
  One day we'll write this up. It is not time or connectivity.

Thiez 6y ago

Are you not tempted to just block the requests from these devices, and let the manufacturer take the loss? I imagine serving all those requests is costing real money.

jgrahamc 6y ago
It's not a TOS violation. It did cause us some ops pain at one point (they were getting hit with > 50,000rps concentrated in certain locations). But one of the reasons Cloudflare can operate our service is we have 3.2 million customers who are doing all sorts of stuff. We get so much stronger from that great variety of traffic.
- pests 6y ago
  Right. I'm sure you can explain more but I've read due to peering agreements and the like having a lot of one-directional traffic can be a good thing.

RyJones 6y ago

This is similar to how Qualcomm's DNS servers got knocked off the air. An OEM shipped an update which would query a development TURN server we were running - once per connection, over millions of devices. It was a crazy day.

DaniloDias 6y ago

Could you be more explicit on the nature of the service? I’d like to explore mechanisms for tests that detect IoT devices that misbehave this way (and other ways as well). Your anecdote sounds interesting. Is it unrelated to time servers? Unrelated to internet connectivity tests?

jgrahamc 6y ago
One day we'll write this up. It is not time or connectivity.