5 ms·
I'm fascinated by this general problem - I see it as a close relative of the "sandbox" problem, where I want to be able to safely run user-defined untrusted ope
by simonw 2y ago
I'm fascinated by this general problem - I see it as a close relative of the "sandbox" problem, where I want to be able to safely run user-defined untrusted operations.
For timeouts, I want to be able to build web application features which are guaranteed to terminate early after eg 2s in a way that resets all resources related to the operation.
The best way I've found of doing this so far involves proceses: run the time-bound operation in an entirely separate process and terminate it early if necessary.
I do that with "rg" here: https://github.com/simonw/datasette-ripgrep/blob/a40161f6273301b7d03a14a43561ec554cb206f7/datasette_ripgrep/__init__.py#L9 https://github.com/simonw/datasette-ripgrep/blob/a40161f6273...
- dools 2y agoI recently had to impose a timeout on a dbus operation using python and the only way I could get it working reliably was with a subprocess. Everything I tried using threads was crazily unreliable.
- bheadmaster 2y ago> the only way I could get it working reliably was with a subprocess Totally agree. I've worked in the past on a multi-camera computer vision system in which I used video4linux to capture images from USB cameras which would sometimes misbehave and block on an ioctl() forever. Not even SIGALRM helped. The only mechanism that could be properly interrupt this situation was a Process, via SIGKILL, which caused OS to clean up all the resources afterwards. Eventually, we connected the cameras to a USB-controlled relay that mechanically replugged the cameras after a certain number of timeouts (ioctl freezes) were exceeded one after another.
- coryrc 2y ago> multi-camera computer vision system That sounds interesting too.
- rcxdude 2y agoThe general issue is isolation, for sure: you need to be able to cleanup the operation regardless of what state it is in when you abort it, and that means that anything you share with it needs to be designed for this. Threads in most OSs simply don't provide enough isolation to do this, two threads share enough resources that you can't effectively do this (in part because many other parts of the system assume that that threads are not an isolation barrier), wheras processes and the means of sharing things between processes are specifically designed so that this is possible.
- hyperpape 2y agoEven processes are not safe if they modify state outside of themselves (files on the filesystem or whatever else). It seems to me that the root issue is that encapsulation becomes our enemy here. We want to be able to call some code and not care about the details of how it is implemented. But reliable cancellation requires some sort of design that lets us know exactly what side-effects code has, control them, and force them to be done in a way that's transactional, while still letting us cancel the operation at the right time. We don't want a black box. I suspect the two general solutions are: 1) elbow grease: manually inspecting the code that you're calling to understand whether it's safe to cancel and designing a bespoke sandbox: a thread in some cases, a process in others, re-architecting the code you're calling in others... 2) something about algebraic effects or capability systems that I can't speak to because I only have vague ideas how they work and haven't applied them in anger.
- ekimekim 2y ago> Even processes are not safe if they modify state outside of themselves (files on the filesystem or whatever else). All processes should be prepared for a sudden crash without corrupting state. Things get OOM killed, machines get unplugged, networks go offline. We have things like journals and write-ahead-logs and at-least-one messaging precisely because of these kinds of problems. If your process can't handle being terminated at any time then you have problems regardless of if it's wrapped in timeout code. Doing this for a block of code within a process is a lot harder, because a) there's generally more surface area of externally-observable state, and b) code does not normally need to be prepared to handle it.
- joosters 2y agoYour comment boils down to 'all code should be perfect'. Which is a lovely request, but doesn't really help. In particular, I'd challenge you to find one large program that handles OOM situations 100% correctly, especially since most code runs atop an OS configured to over-allocate memory. But even if it wasn't, I doubt there's any sizeable codebase that handles every memory allocation failure correctly and gracefully.
- immibis 2y agoYou can sandbox time limits if you firewall state. If nothing in the code being timed out can affect its caller other than raising a "thing has timed out" error, then timing out is safe. Process boundaries provide this firewall on most systems, but they don't have to be the only boundary. I once read about an academic language that had this feature - I don't remember what it was called. It had checked and unchecked exceptions like Java. Checked exceptions are thrown, declared and caught like regular exception control flow. Unchecked exceptions can be raised anywhere, and can only be caught in a way that creates a state firewall. I don't remember how that was enforced.
- nijave 2y agoIn such a system each component still needs to be interruptible. If you're waiting on a blocking operation like disk or network I/O, the caller might give up but the request will still hang around until completion at which point it gets thrown out. In the case of a blocked process, you may be able to force kill it at a system level but then you risk uncleaned up state (leaked connections/resources) For instance, this is the default behavior with Postgres (queries complete even if the connection is closed)
- everforward 2y agoI've become quite partial to Go's implementation. It uses a context.Context that may or may not have one of a few ways of communicating that whatever has this context should stop processing (I.e. a timeout, a deadline, or I believe one or two others). That context then has a .Done() method that returns a channel; when a value is written to that channel, whatever functions are using that context are expected to stop themselves at the soonest point that makes sense to them. Typically this is done inside a for loop in long-running processes. I.e. for something that copies, it looks like for { select { case <-ctx.Done(): // we should stop and return a timeout error or something default: // copy some number of bytes, or check if a network call is done } } It does require all of the involved functions to implement support for this, though I think most things do at this point. I wouldn't call a library high quality unless it supports context.Context for long-running operations. It gives library authors the ability to determine at what points their code can be interrupted, run cleanup code as part of the timeout, etc. > The best way I've found of doing this so far involves proceses: run the time-bound operation in an entirely separate process and terminate it early if necessary. This doesn't handle remote resources cleanly, does it? E.g. if I were to lock a Postgres table for a query, and that query times out, will that correctly unlock the table and close the client? Or e.g. lock files? I'm sure some of that can be handled very carefully by managing it in the main process, but that seems error prone.
- acaloiar 2y agoThe context done channels are clearly the way when dealing with all native Go code. Allthough to the grandparent's point, whne you're dealing with executables or libraries outside of your control, the only true way I know of to get a "handle" on them is to create a process, with its pid becoming your handle. In situations like image processing, conversion, video transcoding, document conversion, etc. you're often dealing with non-Go-native libraries (although this problem transcends language), and there's no way to time-bound processes. That is to say that you often need to consider the Halting Problem and putting time bounds and preemption around execution. So what I've had good success with is adding a process manager around those external processes, and when a timeout or deadline is missed, kill the pid. You can also give users controls to kill processes. Obviously there are considerations with resource cleanup and all sorts of negative consequences to this, depending on your use case, but it does provide options for time bounding and preempting things that are otherwise non-preemptable.
- wongarsu 2y agoIn cooperative settings, say when writing your own web server, .Net Core does this quite well with CancellationToken. Basically it's just a convenient synchronized bool that everyone passes to their callees and occasionally checks to see if they should abort what they are doing. Most async APIs have overloads that take an CancellationToken, so even those operations are cancelled as soon as you cancel the token. They are really useful to impose time limits and for making sure you stop processing a request after it was aborted by the user. And because there isn't much magic it isn't too hard to make sure you reset your resources. But that only works if you trust the code you are executing. If you don't you pretty much have to either use the primitives provided by your OS or run your own interpreter (LUA and WASM are popular for a reason)
- nijave 2y agoIn go there's contexts that do something similar .NET and go seem to fairly good around request cancellation facilities
- Sohcahtoa82 2y agoI've had a similar problem before. I wanted to add a "!calc" trigger to an IRC bot written in Python. Rather than implement the Shunting Yard Algorithm [0] to make sure PEMDAS was handled correctly, I wanted to use `eval` (With some extra steps with the AST library to make a whitelist of AST nodes to prevent code injection), but with Python supporting essentially infinite integer precision, someone writing "!calc 9 * 9 * 9 * 9 * 9 * 9 * [..]" would eat up my entire CPU. I solved it the same way. I execute it in a separate process and terminate it if it takes more than a couple seconds. [0] https://en.wikipedia.org/wiki/Shunting_yard_algorithm https://en.wikipedia.org/wiki/Shunting_yard_algorithm