7 ms·
Ruby's Timeout is dangerous and Thread.raise is terrifying (2015)
- Alifatisk 2y agoSo what's the alternative to Timeout? Or does it mean that if I think I Timeout is the solution then my approach is wrong?
- masklinn 2y agoTimers / cooperative timeouts. AFAIK they’re the only way to ensure that effectful code will never end with a corrupted state.
- mijoharas 2y agoThe other answer is correct I believe, but I think the most common recommendation I've seen for rails boils down to "Use Timeout, and then kill and restart the process" (ref: [0], [1]), which obviously doesn't feel great for performance when a timeout in one thread can cause all other threads to need to be killed. [0] https://www.schneems.com/2017/02/21/the-oldest-bug-in-ruby-why-racktimeout-might-hose-your-server/ https://www.schneems.com/2017/02/21/the-oldest-bug-in-ruby-w... [1] https://github.com/zombocom/rack-timeout https://github.com/zombocom/rack-timeout (c.f. term_on_timeout)
- baggy_trough 2y agoFor web workers, use a single thread per process model, then kill the worker if there's a timeout.
- Asmod4n 2y ago(Almost) Every OS kernel has an API to handle timeouts. poll, select, kqueue, epoll etc. just use that.
- simonw 2y agoThe word "just" is doing a lot of work there!
- immibis 2y agoYou can replace every blocking call with a timed one. In Python you can just call settimeout on a socket. In C you might want to write your own recv_timeout function, but it's not that difficult.
- simonw 2y agoI have no idea how I would use settimeout with a socket to solve any of my problems that involve causing a piece of code to terminate early if it takes longer than a specific amount of time.
- immibis 2y agoWhat is the code doing? Most of the time it's waiting for a socket. If it's not doing that, do the equivalent for whatever it is waiting for. If it's CPU-bound, add interrupt checks.
- simonw 2y agoPlaces I've wanted to implement this in the past include: - Run a subprocess such as "rg" and terminate early if necessary - Run a SQLite SQL a query that errors if it takes more than a second - Same but for other DBs - MySQL, PostgreSQL, Elastic, Mongo etc - Execute a fragment of JavaScript in something like QuickJS with a time limit - Brute force some kind of algorithm with a time limit - using a library I did not write myself
- Asmod4n 2y agoIf you are using a piece of software which has a part that can take a long time to execute and doesn't allow it to set timeouts for long running operations then i would send a PR to solve that issue. When said software is closed source and doesn't have that feature then that company sells a problem and not a solution.
- ape4 2y agoEven Windows has WaitForMultipleObjects() https://learn.microsoft.com/en-us/windows/win32/api/synchapi/nf-synchapi-waitformultipleobjects https://learn.microsoft.com/en-us/windows/win32/api/synchapi...
- maxlybbert 2y agoIn .Net, use CancellationToken ( https://learn.microsoft.com/en-us/dotnet/api/system.threading.cancellationtoken?view=net-8.0 https://learn.microsoft.com/en-us/dotnet/api/system.threadin... , https://learn.microsoft.com/en-us/dotnet/api/system.threading.cancellationtokensource?view=net-8.0 https://learn.microsoft.com/en-us/dotnet/api/system.threadin... ). The two big benefits are (1) if you pass the cancellation token to the task that might be canceled, hopefully it will be obvious that the task might be canceled; and (2) the task that might be canceled has a way to find out it’s being canceled and, if that ever happens, it can clean up.
- Alifatisk 2y agoAnd in Rubys case?
- vidarh 2y agoDepends heavily how trusted/untrusted the process is. If you trust the process won't actively try to circumvent you, just fork and call alarm() as the first thing is often enough. You can cancel it, so it's not a precaution against malice. If you can't trust the process, fork, wait in the parent, and kill the child if it hasn't terminated. I don't think alarm() is exposed in Ruby, unless it's been added recently, but it's easy enough to add. However since fork + kill is more reliable anyway, it's often a better choice.
- dragonwriter 2y ago> Depends heavily how trusted/untrusted the process is. Timeout is for threads not processes, and not everything that would be done on a thread can be conveniently rewritten to be a separate process.
- vidarh 2y agoIf you want reliable timeouts, that is your tradeoff. There are too many awful failure scenarios to resolve with threads.
- regularfry 2y agoYou want something patterned like select. Structure the application around threads blocking on queues and avoiding exceptions entirely makes things far more sane.
- nijave 2y agoYou can manually check an "is timed out" object and ensure that you're configuring timeouts that are smaller on any blocking code (basically I/O) Say you're looping over 1m items, maybe every 1k you check to see if you're out of time and return or raise
- pjc50 2y agoI recently had to debug an issue due to the Java equivalent; the analysis is basically the same. Raising an exception anywhere including code that is not expecting it tends to cause subtle and unrepeatable bugs.
- brabel 2y agoBut you can't do that in Java. Did you use the Thread.stop() method they mentioned in the blog post (removed in Java 8)??
- oefrha 2y agoInterruptedException is better but still handled incorrectly/badly in user code all the time. Sort of like interrupting processes: one process dying won’t corrupt the state of another, but the other process can continue on with bad assumptions.
- hyperpape 2y agoThis is a minor inaccuracy in Julia's post, Thread.stop() is deprecated, but it has not been disabled or removed: https://docs.oracle.com/en%2Fjava%2Fjavase%2F17%2Fdocs%2Fapi%2F%2F/java.base/java/lang/Thread.html#stop https://docs.oracle.com/en%2Fjava%2Fjavase%2F17%2Fdocs%2Fapi...(). InterruptedException is better, but requires something closer to cooperative multi-tasking. If the thread is in a long-running loop that's not doing IO or it swallows the InterruptedException, you won't actually be able to kill the thread. At work, someone had implemented an interface that let you kill a thread prior to my joining the company, so I found out that the warnings are very real when someone did that and it left the application unable to fetch DB connections.
- philipwhiuk 2y agoThreadGroup.stop() was removed in 23, so maybe in the future it'll actually be removed.
- mijoharas 2y agoInteresting, there's a ruby feature request[0] for a safer thread api (this has been a known issue for a long time), and I just saw that it got assigned a couple of months ago. Maybe this'll get addressed in the next ruby version. [0] https://bugs.ruby-lang.org/issues/17849 https://bugs.ruby-lang.org/issues/17849
- simonw 2y agoI'm fascinated by this general problem - I see it as a close relative of the "sandbox" problem, where I want to be able to safely run user-defined untrusted operations. For timeouts, I want to be able to build web application features which are guaranteed to terminate early after eg 2s in a way that resets all resources related to the operation. The best way I've found of doing this so far involves proceses: run the time-bound operation in an entirely separate process and terminate it early if necessary. I do that with "rg" here: https://github.com/simonw/datasette-ripgrep/blob/a40161f6273301b7d03a14a43561ec554cb206f7/datasette_ripgrep/__init__.py#L9 https://github.com/simonw/datasette-ripgrep/blob/a40161f6273...
- dools 2y agoI recently had to impose a timeout on a dbus operation using python and the only way I could get it working reliably was with a subprocess. Everything I tried using threads was crazily unreliable.
- bheadmaster 2y ago> the only way I could get it working reliably was with a subprocess Totally agree. I've worked in the past on a multi-camera computer vision system in which I used video4linux to capture images from USB cameras which would sometimes misbehave and block on an ioctl() forever. Not even SIGALRM helped. The only mechanism that could be properly interrupt this situation was a Process, via SIGKILL, which caused OS to clean up all the resources afterwards. Eventually, we connected the cameras to a USB-controlled relay that mechanically replugged the cameras after a certain number of timeouts (ioctl freezes) were exceeded one after another.
- coryrc 2y ago> multi-camera computer vision system That sounds interesting too.
- rcxdude 2y agoThe general issue is isolation, for sure: you need to be able to cleanup the operation regardless of what state it is in when you abort it, and that means that anything you share with it needs to be designed for this. Threads in most OSs simply don't provide enough isolation to do this, two threads share enough resources that you can't effectively do this (in part because many other parts of the system assume that that threads are not an isolation barrier), wheras processes and the means of sharing things between processes are specifically designed so that this is possible.
- seanalltogether 2y agoOne of the things I've come to appreciate about Reactive Streams is the ability to catch a cancel event and handle it appropriately in your stream. A cancel event can be a timeout or a manual disposal of your async process, but whatever the case you are given the opportunity to deal with it. fun doLongRunning(): Single<Result> { return Single.create { emitter -> val longProcess = /* whatever */ emitter.onSuccess(longProcess_result) emitter.setCancellable { cleanup(longProcess) } } } doLongRunning() .timeout(5_secs) .subscribe({ result -> }, { error -> })
- wallmountedtv 2y agoWhat language is this? It looks like some sort of Java/JVM.
- taf2 2y agoThis the main reason I insist on curb - we need reliable http timeouts
- neon_electro 2y agoCan you share a reference to what curb is? I can only find a project containing Ruby `curl` bindings.
- endgame 2y agoHaskell seems to be the only language where throwing exceptions at another thread isn't a box of footguns. Anyone know why that is?
- WJW 2y agoI don't know why you think that it isn't. You can do so, but AFAIK almost no libraries are written with the expectation that their computations might be interrupted at any time by another thread. The best you can hope for is that you "only" interrupt some pure computation and not throw your exception into the middle of some delicately sequenced IO side effects.
- whateveracct 2y ago> AFAIK almost no libraries are written with the expectation that their computations might be interrupted at any time by another thread. Haskell libraries are written this way all the time. Exception-safe libraries with async-aware handling are pretty commonplace nowadays (partially thanks to safe-exceptions and unsafeio providing a nice API to deal with them.)
- kccqzy 2y agoAbsolutely false. At least not in my experience. Only novice Haskell programmers write code that can't handle computations being interrupted by another thread. A published and maintained Haskell library is unlikely to have these issues.
- masklinn 2y agoErlang / Elixir is almost above that: - the shared-nothing structuring means it’s harder to corrupt external - that processes can die at any moment is part of the culture - the supervision tree means the most likely case for killing an other process is a parent killing a misbehaving child, which they can then handle as if the child had faulted
- jerf 2y agoAnother key to Erlang's success here is the ports system. Basically "external" resources have a defined cleanup mechanism even if the process that created them dies. An open socket under the hood has its own thread it can use to clean up. With effort you can do this with your own things too through the linking mechanism, having some process watch another process using your resources and then have an execution thread that can clean up if the user spontaneously dies. Some of it too is just Erlang being too, err, for lack of a better word "weak" to have some of the problematic resources. Since Erlang has no "locks", you simply can't take a lock and then kill the thread responsible for unlocking it. You can construct such locks at a higher level, but Erlang's design tends to encourage other designs. For instance, if I were designing a higher-level lock, I'd design it to link to the process taking the lock and release it if it dies. This can still theoretically get you into trouble but especially if this is documented as the semantics of the lock, you have to work a lot harder at it. If you really worked at it you could manage to screw up your Erlang system with thread killing through your own lock implementation and such, but it still will be something you can recover from the REPL, you won't confuse BEAM itself. In an imperative language you'd be ranging from lucky to be able to reach in somehow and fix it to there simply being no way.
- jupp0r 2y agoI'm not surprised, the whole language (Ruby) and ecosystem favors "ease of use" (ie the ability to write code fast) over correctness in so many places. Subtle details that other languages would require you to handle in order to robustly handle non happy path conditions are often ignored, lumped together, return just nil, etc. It seems careless, but a large group of engineers love it because it gives the illusion that the world is simple.
- neonsunset 2y agoRuby is not alone in this. Go shares similar philosophy to just half-ass the implementation. Well, at least it's not as brittle and impossibly slow as Ruby.
- jupp0r 2y agoDo you have an example? My impression of the Go standard library is that they picked a pretty decent compromise of abstraction levels over the various OS facilities they use under the hood.
- wk_end 2y agoThe classic reference here is “I want off Mr. Golang’s Wild Ride”: https://fasterthanli.me/articles/i-want-off-mr-golangs-wild-ride https://fasterthanli.me/articles/i-want-off-mr-golangs-wild-...
- jupp0r 2y ago"Ruby is not alone in this" Agreed. My point was rather that this is not Ruby's only dangerous shortcut.
- byroot 2y agoIf you read TFA, you'd have seen this problem isn't exclusive to Ruby at all, several other languages have a similar API.
- jupp0r 2y ago
- llimllib 2y agoI hit a pretty brutal (and fun, if you are into this sort of thing) problem with Timeout 11 years ago: https://stackoverflow.com/questions/17237743/timeout-within-a-popen-works-but-popen-inside-a-timeout-doesnt https://stackoverflow.com/questions/17237743/timeout-within-...
- assbuttbuttass 2y agoRust has a similar problem, where any future can be dropped at any time in async code, effectively cancelling the future. This means all async code has to be ready to be unexpectedly cancelled at any await point, and implement any defensive cleanup code, etc Some more discussion of the problems this creates in Rust: https://without.boats/blog/asynchronous-clean-up/ https://without.boats/blog/asynchronous-clean-up/
- demurgos 2y agoIt's definitely an issue, but it's also an improvement to the situation in other languages as the await points were cancellation is possible are visible in the code. One solution would be linear types that can't be dropped, but interactions with generics and panics make it hard.
- pkolaczk 2y agoIf a future is canceled it will run its destructors, so cleanup will usually happen correctly even if the developer didn’t think about it. Connections and files will be closed, memory released, locks unlocked. There are exceptions to that of course, bacause not running something to completion may break business logic - but no language can protect from those kinds of errors. And cancellation can happen only in await points which makes it much easier to analyze than being interrupted at any place like in Java.
- jupp0r 2y agoThis is not a bug, it's a feature. You have the ability to handle errors at defined points. As reality is such that errors can happen at any of these points, being able to gracefully handle them is as good as it gets. If you want to compose error handling along the type-of-error axis or the await-point-axis, the language already provides tools for you to do so. Again, not being able to blatantly ignore these errors is a feature, not a bug. In NodeJS/Browser environments, you have the exact same behavior where promises you are awaiting can get rejected.
- asabil 2y agoThe underlying problem is lack of isolation, and this is an area where the Erlang process model shines: processes are fully isolated and you can send a signal externally to terminate any process you want.
- jonnycat 2y agoThis post got me curious about similar scenarios in Elixir, and despite working with Elixir every day, I'm a bit surprised by one of the results I found: # Recursive function that never terminates: iex> f = fn i, f -> if rem(i, 100000) == 0 do IO.inspect(i) end; f.(i+1, f) end # Start the function in a task with a 1 ms timeout iex> Task.async(fn -> f.(0, f) end) |> Task.await(1) My expectation here is that the task would output 0, then get killed when it hits the timeout. And I do get a timeout "exit" message logged with the child pid. But ALSO, the numbers keep printing as though the child task is still running! It appears to be specific to the configuration of the iex process but I'm not sure what it is - any Elixir/Erlang folks who can explain exactly what is happening here?
- davisp 2y agoI never learned Elixir, but my first guess is IO.inspect is sending a message that is printed by a different process. Then the prints after exit are just the IO process working through its mailbox. Alternatively, the await might killing the waiting process, not the process being waited on.
- jonnycat 2y agoGood thoughts, but the printing continues indefinitely, and the documentation for Task.await explicitly says the child process will be killed: "If the timeout is exceeded, then the caller process will exit. If the task process is linked to the caller process which is the case when a task is started with async, then the task process will also exit". Processes can be configured with the behavior you describe, but it's not the case with Task.await.
- dasyatidprime 2y agoTask.await tries to exit the calling process when the timeout hits, but IEx traps the exit in that process, so it doesn't terminate and thus the linked task process doesn't either, I think? If I do all of this wrapped in another task, rather than directly in IEx, then I observe the innermost process get terminated by the process link after the intervening one doesn't trap the exit. Relevant from https://hexdocs.pm/elixir/1.4.5/Task.html https://hexdocs.pm/elixir/1.4.5/Task.html, which you've probably already seen: > If the timeout is exceeded, await will exit; however, the task will continue to run. When the calling process exits, its exit signal will terminate the task if it is not trapping exits.
- constantin0 2y agoThe same kind of issue prevents Celery (a python task queue) "soft timeout" feature from working reliably. It throws an exception _anywhere_ in the worker process to signal the timeout, but in practice the exception is often eaten by too-generic except blocks in library code and never reaches user code. An interesting solution is structured concurrency because it introduces "raise-safe" points (the "await"s), but the ecosystem is not there yet.
- kgeist 2y ago>but in practice the exception is often eaten by too-generic except blocks in library code and never reaches user code Interesting, C# has a special case for this: >ThreadAbortException is a special exception that can be caught, but it will automatically be raised again at the end of the catch block.
- deleted 2y ago[deleted]
- nijave 2y agoI like Python but it's definitely a pain when it comes to request cancellation I saw a StackOverflow post about some code that only worked on Linux and not Windows and it turns out sleeps on Windows are implemented using an uninterruptible API (I think there are other APIs on Windows but the developers of Python picked the one they did since it was simple/low overhead) * Maybe not a perfect description but it was something about a non interruptible Windows API Not sure if they changed it but iirc Python was using a less precise timer on Windows that couldn't sleep less than 14-15ms as well
- deleted 2y ago[deleted]
- hun3 2y agoC has this problem too: see signal-safety(7). > * If a signal handler interrupts the execution of an unsafe function, and the handler terminates via a call to longjmp(3) or siglongjmp(3) and the program subsequently calls an unsafe function, then the behavior of the program is undefined. Also note that pthread_cancel() provides a way to block asynchronous cancellations via pthread_setcancelstate().
- jandrese 2y agoAm I weird that I've been programming in C for decades and have never used longjmp? For the longest time I thought it was some legacy function that was only used on ancient segmented memory systems, but it instead turns out to be a complete program flow destroyer that I can't see a good use for. I have seen some libraries that do error handling via the user passing a callback for a longjmp, but even that seems ill advised. Fortunately it is always optional.
- jeff-davis 2y agoLongjmp is used by Postgres for transaction aborts. With C, there's not really a better option available.
- eqvinox 2y agoThat's not from within signal handlers, though. (i.e. it relates to this specific longjmp discussion but not the root post re. exceptions on other threads.)
- nurple 2y agoReminds me of early in my career as I was struggling with turning errors in my code into useful messages to the user. I thought exceptions were so cool and that I could just pop them up the stack and voila `print(err.msg)`. It took me awhile to realize that good error reporting was part of the UX as much as any other feature I wanted to create for users, and that it deserved a seat at the table of the API interfaces. I think handling timeouts properly is exactly the same case; like TFA mentioned, there is no such thing as a safe/general way to externally terminate a thread of execution. If you want timeouts to be part of your UX, you need to build it just like any other feature.
- nijave 2y ago>general way to externally terminate a thread of execution There's really no safe way to cover all cases unless you create a really scoped down interface that are explicit in what they do. One use case might allow abrupt interruption with no cleanup (like in the case the OS is powering down) and another interrupt it might be imperative to cleanup (a long running program that can leak resources)
- o11c 2y agoSome notes about this: * This issue plays out differently for compiled languages vs interpreted languages, but does apply to both. In particular, in many interpreted languages, the unfixable (for user code, at least) issue is "what happens if I get an exception after constructing the object but before assigning it to a local/member variable"? For compiled languages a similar issue may be "what if it's still in a register", though they generally handle this. * Even for languages without dangerous thread APIs, it usually applies to MemoryError, and possibly others (e.g. signals, at least "interrupt" and "quit") as well (e.g. Python, despite having increasingly broken signal handling in general, still causes this for the main thread) * True asynchronous exceptions aren't actually necessary; it suffices to manually check for exceptions on every non-finite backward jump (that is, every non-unrollable loop iteration except the first), every (generally, uninterruptible) syscall (or call to native code), and every memory allocation. For some particular language design choices, you might also want to check every memory access, though this is unlikely to be the same. * This is much less of an issue in a language with destructors; try-finally, Python-style-with, try-with-resources, defer, etc. are all fatally flawed. It's still possible to write bad code in a good language if you try hard enough, but at least it is possible to write good code as well.
- eqvinox 2y ago> C++: std::threads are not interruptible. This is kinda half-true, since C++ inherits aspects of the C environment, in this case on POSIX systems pthreads async cancellation [https://pubs.opengroup.org/onlinepubs/7908799/xsh/pthread_cancel.html https://pubs.opengroup.org/onlinepubs/7908799/xsh/pthread_ca..., https://www.man7.org/linux/man-pages/man3/pthread_cancel.3.html https://www.man7.org/linux/man-pages/man3/pthread_cancel.3.h...] You can control this through pthread_setcancelstate(), and if it is enabled it will only trigger on specific documented places in C library / system API calls. It is vaguely saner than this generic "throw exception on another thread" concept, but still pretty insane IMHO. Haven't used it myself, not sure if there are circumstances/use cases where I'd enable it. (To be fair, a lot of other languages are built on top of the C library too and inherit this same behavior, so this isn't quite C++ specific.)
- flqn 2y agoC++ also has stop_token, so although they're not natively interruptable a mechanism for implementing it exists in the standard.
- CodeWriter23 2y agoYep. Multithreaded coding is hard.
- Sonata 2y agoThe approach taken to this problem by Cats Effect (a Scala concurrency library) is interesting. It allows cancellation of a fiber from outside, but let's blocks of code be marked as uncancelable. If a fiber is cancelled while executing one of these blocks, it will complete the block before cancelling. This protects against cancellation in between two operations which leaves the program in a broken state. The drawback of this approach is that the onus is on anybody writing code which might be cancelled to correctly mark the uncancelable regions.
- mrkeen 2y agoI don't think you can do better than this. There's a fundamental tension between 'stop now' and 'close your file handles before stopping'.
- shirogane86x 2y agoFrom how you describe it, it sounds like it's taken the same approach as Haskell's async exceptions + mask/uninterruptibleMask, which to me seems like one of the best solutions around, so props to them
- mrinterweb 2y agoAs mentioned in the article, this isn't limited to ruby. It just shows that the ruby timeout can be dangerous, which a great reminder to those who use it. It would be great if the danger was mentioned in ruby's Timeout class documentation. The same type of danger can be encountered with the linux `kill` command, depending on which signal you send, losing power, operating system terminate, etc. If you have state in flux, hard killing a process can have unexpected outcomes. I'd also like to say that just because something is dangerous, doesn't mean it should not be used. Just use it carefully. Think of it as dynamite. Dynamite is very useful in some cases, and always needs to be handled with care.
- dang 2y agoDiscussed at the time: Why Ruby’s Timeout is dangerous and Thread.raise is terrifying - https://news.ycombinator.com/item?id=10638629 https://news.ycombinator.com/item?id=10638629 - Nov 2015 (20 comments)
- notjoemama 2y ago> There is no way to safely interrupt an arbitrary block of code. Anything could be happening at the end of that 5 seconds. IKR!? Like, I'll be click-click-clicking away on my 1982 IBM PC/XT, I mean I'm REALLY hammering it, like REALLY leaning into the keystrokes. Then, all of a sudden, WHAM! Out pops some random lines of code; right at the end of my thread.abort block. I mean, D00000D. OMFG. This can happen anywhere! Anytime! How is any programmer supposed to know what their code is executing?! I guess I tend to believe, if you don't know what you are doing, then don't do it. If this is that big of an issue, maybe try real estate. Say it with me, "three-brrr", "two-brrr". Good. Now together... Also, "idempotent". Everyone keeps dancing around that. Not sure why.
- wewtyflakes 2y agoMaybe there is a nuance that I am missing, but having in mind that an exception can be thrown "in any of your code, regardless of whether it could have possibly raised an exception before" when writing code seems sane to me. It seems like the issue being raised is that people are expecting code snippets to be perfectly transactional, which also seems wrong.
- Too 2y agoYes. This is why you should always use context managers and finally blocks in languages with exceptions. Always assume that anything can fail at any time and accept that not everything can nor should be recovered from. On the other end of the spectrum you have go, where in fact every line does need to have their error conditions explicitly checked. How do they handle this situation? What is dangerous with finally-blocks is if they start consuming a lot of time, such that the timeout interruption itself makes the timeout miss its deadline. Usually you need 2 timeouts, one for the main operation and a second for the cleanup.
- chucke 2y agoBear in mind that this article is old. Since then, ruby support a [handle_interrupt mechanism not far from the method described as working in java](https://rubyapi.org/3.3/o/thread#method-c-handle_interrupt https://rubyapi.org/3.3/o/thread#method-c-handle_interrupt)