How to execute multiple tasks in parallel in fabric
I know that by using -P switch or @parallel tag I can run task in parallel on multiple hosts.
I'm trying to execute multiple long running tasks in parallel on the same host:
@task
def task1():
# long running op
@task
def task2():
#long running op
@task
def task3():
#long running op
@task
def backup_all():
execute(task1)
execute(task2)
execute(task3)
How can I start task1, task2 and task3 in parallel on the same host by using fabric. I know that I can run multiple fab processes with different tasks, but I'm looking for a solution that involves fabric.
You have a number of ways to address this task. You could use bash/linux level job controls and run the tasks in the background with bg and then wait for them to complete. Documentation explaining this style of control mechanism.
If you still really wanted to use Fabric/Python for this, you'd likely need to use the job_queue that's already in the lib and sorta write your own queue to push these into, or read up on multiprocessing and just do some simple python forking. Though that is essentially all the job_queue is doing.
fabric runs one task per host. -P switch just runs task in parallel on different hosts. No parallelization on the same host.
You could parallelize commands manually eg, using xargs -P or GNU parallel .
Invoke (beta) claims the ability to run tasks in parallel. It extracts non-ssh parts from fabric . If it is installed on remote hosts then you could use fabric to invoke invoke command that would run tasks in parallel on the same host.
下一篇: 如何在结构中并行执行多个任务