Package Scientific :: Package Threading :: Module TaskManager :: Class TaskManager
[hide private]
[frames] | no frames]

Class TaskManager



Parallel task manager for shared-memory multiprocessor machines

This class provides a rather simple way to profit from shared-memory multiprocessor machines by running several tasks in parallel. The calling program decides how many execution threads should run at any given time, and then feeds compute tasks to the task manager, who runs them as soon as possible without exceeding the maximum number of threads.

The major limitation of this approach lies in Python's Global Interpreter Lock. This effectively means that no more than one Python thread can run at the same time. Consequently, parallelization can only be achieved if the tasks to be parallelized spend significant time in C extension modules that release the Global Interpreter Lock.

Instance Methods [hide private]
  __init__(self, nthreads)
  _removeTask(self, task)
  _scheduler(self)
  runTask(self, function, args)
Run a task as soon as processing capacity becomes available
  terminate(self)
Wait until all tasks have finished

Method Details [hide private]

__init__(self, nthreads)
(Constructor)

 
Parameters:
  • nthreads (int) - the maximum number of compute threads that should run in parallel. Note: This does not include the main thread which generated and feeds the task manager!

_removeTask(self, task)

 

_scheduler(self)

 

runTask(self, function, args)

 

Run a task as soon as processing capacity becomes available
Parameters:
  • function (callable) - the function that will be executed as the body of the task
  • args (tuple) - the arguments that will be passed to function when it is called. An additional argument will be added at the end: a lock object that the task can use to get temporarily exclusive access to data shared with other tasks.

terminate(self)

 

Wait until all tasks have finished