3 ms·
[I work at imgix and have helped lead the team on the production issues we face and gathering the details for this blog post] > Which piece of infrastructure?
by skuhn 10y ago
[I work at imgix and have helped lead the team on the production issues we face and gathering the details for this blog post]
> Which piece of infrastructure?
In this case it was a top-of-rack switch. That rack had a few of our external DNS resolvers on it. Normally this wouldn't be a problem, because we deploy services across multiple racks to prevent this kind of failure mode.
In this case it was a bit of a problem, because this was the first rack deployed at this site and it turned out to not conform to the standard configuration across the rest of the racks (because it was what we bootstrapped the site with).
So two things: it was tough to get into the device to troubleshoot it (we wound up using the serial port infrastructure deployed for this purpose), and it had a service impact even though it shouldn't have (we have since migrated DNS resolvers out of this rack and have scheduled a future maintenance to get this switch's configuration corrected and ugprade its OS).
> New traffic patterns
This is a bit tougher to dig into, but essentially it was a needle-in-a-haystack kind of situation. Customers can use any imgix URL API parameter they like -- some of these use cases get pretty complicated for the backend to handle. Think of the watermark parameter stacked with another URL that is also an imgix URL with another watermark parameter, five layers deep. These sorts of operations take a lot more rendering resources than a simple ?h=600&w=600 operation.
In this case, we observed an influx of these sorts of more difficult operations. We have various logs and metrics sprinkled throughout the system -- we use Prometheus, Kafka, heka, BigQuery, Grafana and a few other systems to collect and present the data we need to run the service. We also issue unique id's per request to track their path through the system. What we don't have -- and need to -- is one end-to-end view of a request's path through the system and the system's capacity and performance at each point in our stack.
It turned out that for some amount of the increased rendering traffic we saw, it wasn't that we suddenly got more requests. There were simply many new permutations for each original object. That lowers the CDN cache hit rate for a period of time.
The other thing that comes to mind (and I think I'm forgetting a third type here), is that some of our request parameters require normalization so that they can utilize the same cache object. Think of parameters like dpr, which is a floating point number but realistically is only useful up to a few decimal places. dpr=1.33 and dpr=1.3333333333 are actually the same image, but they would have different cache keys and require two renders that are effectively the same object.
We normalize the dpr parameter down to three significant digits. What we found is that this sort of normalization was necessary for another parameter as well.
- billyhoffman 10y agoWhy isn't this in your blog post? Or at least a separate "technical details" blog post linked from the announcement? When you have an incident, especially something that hurts your customers beyond just a service outage, you come clean, fast, and as completely transparent as possible.
- skuhn 10y ago[I work at imgix and have helped lead the team on the production issues we face and gathering the details for this blog post] We posted the blog post yesterday (Friday) afternoon, after getting the copy ready Thursday evening. Friday was spent acquiring the right customer e-mail list (a little harder than it sounds and something to improve in the future) and taking one last pass over the post (among other tasks we're all doing to actually operate the service, communicate to individual customers, etc.). It's super important to be open and honest and fast in your response, but it's also important to communicate correctly. The post in its current form is meant to be transparent and apologetic but also show that positive steps have been taken and results are being seen now. If we missed the mark on transparency, or honesty, or the hints of positivity, it wouldn't have been the right message for where we truly are at today. I'm completely open to writing a more in-depth technical post, but that would have compromised our timing to get this post out on Friday. That's why we went with the post in its current form -- anything else would have taken longer to get to our customers. It's something that I'll think about more this coming week and hopefully get out soon.
- mcherm 10y ago> When you have an incident, especially something that hurts your customers Beyond just a service outage, you come clean, fast, and as completely transparent as possible. Isn't that precisely what they are doing here? This looks to me like a case of a company doing an excellent job of clearly communicating about an incident, at least from the point where the post was first published.