the Python threading module does not seem to provide a means to cancel a running thread. There are many discussions on the web dealing with this issue and many solutions are offered, but none of them seems to be applicable to my situation, which is as follows:
I have a C library which does some very computationally intensive stuff. Some functions in this library might run for a long time, and the code in the library is optimized for speed. The library is used in C programs and also from a PyGTK GUI program. In the GUI, you can choose the computation parameters and than start the computation. It is run in a separate thread, which calls a function in the C library via ctypes.
Now it should be possible to cancel the computation from the GUI. In the C programs, I can just kill or cancel the thread (there is no cleanup to be done -- the thread does not use any system resources apart from CPU time). In Python, this is not possible. The solutions for this problem I found on the web usually recommend to have the thread regularly check some variable and exit if this variable indicates to do so. This would have to be included in the C library and would include quite a bit of code refactoring. The check cannot be included in the inner loop because it would lead to significant performance loss. Furthermore, I just do not want to mess up an elegant and efficient library design which works perfect when used from C just to make up for shortcomings in Python.
So do I really have to refactor my C library just because Python Thread objects lack a cancel method? Is there really no other way? And why on earth doesn't that cancel method exist? There *are* good reasons to cancel a thread, just google for "terminate a Python thread" for tons of examples.
On Nov 7, 6:04 pm, Sven Marnach <s...@pantoffel-wg.de> wrote:
> So do I really have to refactor my C library just because Python > Thread objects lack a cancel method? Is there really no other way?
It doesn't sound like the thread is communicating with the process much. Therefore:
1. Run the C code in a separate process, or 2. Create the thread from a C extension, maybe even straight from ctypes, and kill it from C or ctypes.
> And why on earth doesn't that cancel method exist? There *are* good > reasons to cancel a thread, just google for "terminate a Python > thread" for tons of examples.
Arguing that there are good reasons to allow killing threads isn't going to get you very far. The language developers already know killing a thread is useful, yet the disallowed it anyway. The drawbacks were judged too severe (it makes enforcing invariants pretty much impossible).
Carl Banks wrote: > Arguing that there are good reasons to allow killing threads isn't > going to get you very far. The language developers already know > killing a thread is useful, yet the disallowed it anyway. The > drawbacks were judged too severe (it makes enforcing invariants pretty > much impossible).
While outright thread cancellation is generally unsafe, it would be useful if there was a way to force another thread to unblock from a wait condition, like a blocking read, with an exception. This is, among other things, why control-C won't terminate some threaded programs.
Python 2.6 and 3 have some steps in this direction. There's "signal.set_wakeup_fd(fd)", and "siginterrupt()". But the real problem, that signals are associated only with the first thread, hasn't been addressed.
Question: if "signal.set_wakeup_fd(fd)" is used, and the thread waiting on "fd" is not the main thread, will a signal cause the waiting thread to get a read completion? Or is this another "first thread only" thing"?
On Nov 8, 4:27 am, Carl Banks <pavlovevide...@gmail.com> wrote:
> It doesn't sound like the thread is communicating with the process > much. Therefore:
There is quite a bit of communication -- the computation results are visulized while they are generated.
> 1. Run the C code in a separate process, or > 2. Create the thread from a C extension, maybe even straight from > ctypes, and kill it from C or ctypes.
The second option is a good idea. Thanks a lot, Carl!
> > And why on earth doesn't that cancel method exist? There *are* good > > reasons to cancel a thread, just google for "terminate a Python > > thread" for tons of examples.
> Arguing that there are good reasons to allow killing threads isn't > going to get you very far. The language developers already know > killing a thread is useful, yet the disallowed it anyway. The > drawbacks were judged too severe (it makes enforcing invariants pretty > much impossible).
I really don't get that. If the reason would be that it is too much work to implement, then I could accept it. But saying: We know it is useful, but we won't allow to do it, just does not seem reasonable. Thread cancellation might be generally unsafe, but there are cases when it is safe. It should be up to the user to decide it. There are many things that do harm if you don't use them correctly, and of course it would be a bad idea to remove all of them from Python.
Well, I won't complain too much. At least some volunteers created that great language and gave it away for free :)
>On Nov 8, 4:27 am, Carl Banks <pavlovevide...@gmail.com> wrote: >>It doesn't sound like the thread is communicating with the process >>much. Therefore:
>There is quite a bit of communication -- the computation results are >visulized while they are generated.
I'm curious how this visualization works, since earlier you said something to the affect that there were no shared resources. If you kill a thread and it had opened a window and was drawing on it, with most toolkits, you'll end up with a window stuck in your screen, won't you?
>[snip]
>I really don't get that. If the reason would be that it is too much >work to >implement, then I could accept it. But saying: We know it is useful, >but we >won't allow to do it, just does not seem reasonable. Thread >cancellation >might be generally unsafe, but there are cases when it is safe. It >should be >up to the user to decide it. There are many things that do harm if >you don't >use them correctly, and of course it would be a bad idea to remove all >of >them from Python.
The CPython philosophy sort of follows the guideline that you should be allowed to do bad stuff if you want, except when that bad stuff would crash the interpreter (clearly ctypes is an exception to this rule of thumb). I think this is the argument that has been applied in opposition to adding thread termination in the past, though I don't remember for sure.
« TerminateThread is a dangerous function that should only be used in the most extreme cases. You should call TerminateThread only if you know exactly what the target thread is doing, and you control all of the code that the target thread could possibly be running at the time of the termination. For example, TerminateThread can result in the following problems:
* If the target thread owns a critical section, the critical section will not be released. * If the target thread is allocating memory from the heap, the heap lock will not be released. * If the target thread is executing certain kernel32 calls when it is terminated, the kernel32 state for the thread's process could be inconsistent. * If the target thread is manipulating the global state of a shared DLL, the state of the DLL could be destroyed, affecting other users of the DLL. »
Antoine Pitrou wrote: > Le Sun, 08 Nov 2009 04:40:26 -0800, sven a écrit : >> I really don't get that. If the reason would be that it is too much >> work to >> implement, then I could accept it.
> It would probably be a lot of work and even then it would still be unsafe.
I'd argue against general thread cancellation. Inter-thread signals, though, have safety problems no worse than the first-thread only signals we have now. You're allowed to raise an exception in a signal handler, which is effectively thread cancellation.
So right now, you can kill the first thread from another thread.
> I'd argue against general thread cancellation. Inter-thread > signals, though, have safety problems no worse than the first-thread > only signals we have now. You're allowed to raise an exception > in a signal handler, which is effectively thread cancellation.
Can you give an example of such "cancellation"? In any case, this would be a side-effect of the current implementation, not officially supported behaviour.
On Nov 8, 2:50 pm, exar...@twistedmatrix.com wrote:
> I'm curious how this visualization works, since earlier you said > something to the affect that there were no shared resources. If you > kill a thread and it had opened a window and was drawing on it, with > most toolkits, you'll end up with a window stuck in your screen, won't > you?
The Python code passes an array to the C library which in the end contains the computation results. This array only contains some kind of counts which are incremented during the computation. The PyGTK GUI simply visualizes the current state of this array in regular intervals. The C library does not do any GUI stuff. Cancelling the thread really would not do any harm -- the thread only allocates memory on the stack, and the stack will vanish with the thread.
> The CPython philosophy sort of follows the guideline that you should be > allowed to do bad stuff if you want, except when that bad stuff would > crash the interpreter (clearly ctypes is an exception to this rule of > thumb).
Well, I am really glad about that exception. And following the hint of Carl, I will use it to call pthread_cancel or pthread_kill directly from the library. (Most certainly I also have to use ctypes to create my threads to be able to do this.)
Antoine Pitrou wrote: > John Nagle <nagle <at> animats.com> writes: >> I'd argue against general thread cancellation. Inter-thread >> signals, though, have safety problems no worse than the first-thread >> only signals we have now. You're allowed to raise an exception >> in a signal handler, which is effectively thread cancellation.
> Can you give an example of such "cancellation"? > In any case, this would be a side-effect of the current implementation, not > officially supported behaviour.
It's not only documented behavior, it's an example in the official documentation. See
Le Sun, 08 Nov 2009 21:04:06 -0800, John Nagle a écrit :
> Antoine Pitrou wrote: >> John Nagle <nagle <at> animats.com> writes: >>> I'd argue against general thread cancellation. Inter-thread >>> signals, though, have safety problems no worse than the first-thread >>> only signals we have now. You're allowed to raise an exception in a >>> signal handler, which is effectively thread cancellation.
>> Can you give an example of such "cancellation"? In any case, this would >> be a side-effect of the current implementation, not officially >> supported behaviour.
> It's not only documented behavior, it's an example in the official > documentation. See
Well, the only supported behaviour is to send signals to the main thread. Besides, it doesn't "cancel" the thread, it just raises an exception in it, which can be caught and silenced. Just try the following:
import signal, time
def handler(signum, frame): print 'Signal handler called with signal', signum raise IOError
# Set the signal handler and a 5-second alarm signal.signal(signal.SIGALRM, handler) signal.alarm(2)
|
