Multi/single processing abstraction.
Execute functions hiding the information whether are executed in parallel, using multiprocessing.pool.Pool or serially.
The module provides a high level factory function to create and retrieve worker instances get_worker().
Todo
Warn the user when getting an existing name with the other options not set to the defaults
Add context manager?
>>> import pyhetdex.tools.processes as p
>>> def func(args):
... print(", ".join(args))
... return " ".join(args)
Run single processor jobs
>>> worker_sp = p.get_worker(name="singlep")
>>> job = worker_sp(func, ["from", "single", "processor"])
from, single, processor
>>> job
<pyhetdex.tools.processes.Result object at ...>
>>> job.get()
'from single processor'
>>> worker_sp.get_results()
['from single processor']
>>> worker_sp.close()
Or multiprocessor jobs
>>> worker_mp = p.get_worker(name="multip", multiprocessing=True)
>>> job = worker_mp(func, ["from", "multi", "processor"])
>>> job
<multiprocessing.pool.ApplyResult object at ...>
>>> job.get()
'from single processor'
>>> worker_mp.get_results()
['from multi processor']
>>> worker_mp.close()
Run some initialisation function when creating the multiprocessing pool
>>> def init_func(message):
... print(message)
>>> worker_mp_init = p.get_worker(name="multip_init", multiprocessing=True,
... initializer=init_func,
... initargs=("docstring",))
>>> worker_mp_init.close()
Alternatively, you can use the worker within a with statement
>>> def func1(args):
... return " ".join(args[::-1])
>>> with p.get_worker(name="inwith", multiprocessing=True) as wworker:
... wworker(func1, ["in", "with", "statement"])
... wworker.get_results()
['in with statement']
Returns a worker with the specified name.
At the first call with a given name, the worker is created using the remaining arguments. By default the worker is for a single process. Subsequent calls with the same name return always the same worker instance and the remaining options are ignored. This means that worker instance never need to be passed between different parts of the application.
Parameters: | name : string, optional
multiprocessing : bool, optional
always_wait : bool, optional
poolclass : class
kwargs : dictionary
|
---|---|
Returns: | worker: _Worker instance |
Although the class itself should be considered as private, the attributes and method listed below are public
Class to hide the details of the single or multiprocessor execution. The class declaration should be considered as private and should be created and retrieved through the get_worker() factory functions.
The instance can be used once in a with statement. Upon exiting, it close/terminate and join the pool, if multiprocessing is used.
Parameters: | multiprocessing : bool, optional
always_wait : bool, optional
poolclass : class, optional
kwargs : dictionary
|
---|
Apply func on args and kwargs (asynchronously if multiprocessing is on).
Parameters: | func: callable
args: list
kwargs: dict
|
---|---|
Returns: | job : AsyncResult or Result
|
Wait for all the processes to finish and return the results.
Parameters: | fail_safe : bool
|
---|---|
Returns: | list
|
list of AsyncResult or Result instances
Implements the same interface as multiprocessing.pool.AsyncResult and execute the function at instantiation time.
Used to abstract single/multi processor cases in _Worker and to postpone error handling.
Parameters: | func : callable
args : list
kwargs : dict
|
---|