3 ms·

I’m very interested in the post mortem about the specific race condition. Since the article mentions session ids returned to the wrong user my first guesses wo

I’m very interested in the post mortem about the specific race condition. Since the article mentions session ids returned to the wrong user my first guesses would be 1) something with thread-local storage saving the the auth info in the request handler, and then coroutines or async code that is overwriting that thread-local with info from a new request before the old request has finished. 2) messagequeue based micro service communication that was not properly checking if a received response belongs to the original request and thus returning mismatched data

londons_explore 6y ago
Some user request marked cachable when it should not have been...
gruez 6y ago
>1) something with thread-local storage saving the the auth info in the request handler, and then coroutines or async code that is overwriting that thread-local with info from a new request before the old request has finished. AFAIK github is written in ruby. Does it even have thread local storage?
- ghiculescu 6y ago
  Yes, at least in Rails you see this a bit: > Thread.current[:key] = value Typically you’d only have one request at a time on each thread though. So your examples feel unlikely. Looking forward to finding out what happened here.
deckard1 6y ago
> 1) something with thread-local storage saving the the auth info in the request handler, and then coroutines or async code that is overwriting that thread-local with info from a new request before the old request has finished. This would be my guess. This sounds like Ruby. But I've seen very similar bugs in Node, where user data is leaked across requests. The issue is typically the developer is using a global variable when they should be using the request context/storage. In Node there is an insidious brother of this bug as well. It's where the developer is initializing some bit of code (often a 3rd party library, or loading a file) and they are doing it per request when they should be doing it once globally at process startup. It's just about the opposite of the first bug I mentioned. The result is Node leaks memory on every request until it runs out of memory and the process dies. This bug can be an incredible pain in the ass to hunt down.

londons_explore 6y ago

Some user request marked cachable when it should not have been...

gruez 6y ago

ghiculescu 6y ago
Yes, at least in Rails you see this a bit: > Thread.current[:key] = value Typically you’d only have one request at a time on each thread though. So your examples feel unlikely. Looking forward to finding out what happened here.

deckard1 6y ago

> 1) something with thread-local storage saving the the auth info in the request handler, and then coroutines or async code that is overwriting that thread-local with info from a new request before the old request has finished. This would be my guess. This sounds like Ruby. But I've seen very similar bugs in Node, where user data is leaked across requests. The issue is typically the developer is using a global variable when they should be using the request context/storage. In Node there is an insidious brother of this bug as well. It's where the developer is initializing some bit of code (often a 3rd party library, or loading a file) and they are doing it per request when they should be doing it once globally at process startup. It's just about the opposite of the first bug I mentioned. The result is Node leaks memory on every request until it runs out of memory and the process dies. This bug can be an incredible pain in the ass to hunt down.