4 ms·

We had a similar issue and assumed it was script kiddies having fun. Turns out someone (raises hand) wrote a really bad microservice who's inefficient queries s

• saltminer 3y ago
  Reminds me of a support ticket I had to investigate recently. Performance metrics were taking a dive and triggered automated alerts - average response time jumped from 200ms to 2000, 8000, and eventually began approaching 15000, at which point requests were timing out all over the place. At first I was wondering if my recently-deployed MR was responsible, but upon further investigation, it was from one developer on another team doing two very dumb things with their application prototype: - Constantly retrying the same query with no filtering and setting the maximum allowed page size - Sending massive quantities of small queries for specific individuals (50-100 per second) which quickly hit the rate limit, and immediately resuming after the rate limit expired Some developer outreach was required...
• ilyt 3y ago
  We had that except it was our own frontend developers. We also had some actual attacks so we made a system that detect anomalies (like more than 50rps per IP) and raises alert. ...which was thwarted by frontend developers again, as they loaded few hundred tiny icons at once that triggered that alert routinely, and only thru http2 multiplexing their idiotic design patents haven't bitten them before.
• jopsen 3y ago
  Hehehe, entirely unsurprising :)