QueueRunner plugin

QueueRunner plugin implements simple queue for task execution instead of starting threads for ongoing tasks.

For example, if number of threads 10, but task need to be executed on 20 hosts, threaded runner will start first 10 threads to run task for first 10 hosts, after that start another 10 threads to run task for remaining 10 hosts.

Above process works well for majority of cases, but using QueueRunner might be beneficial in certain situations, e.g. QueueRunner pros:

  • worker threads started only once saving some negligible CPU cycles

  • even if one of the hosts takes longer time to complete the task, threads will not stay idle and continue serving other hosts, that might reveal better execution time

QueueRunner Architecture

../_images/QueueRunner_v0.png

QueueRunner Sample Usage

Need to instruct Nornir to use QueueRunner on instantiation:

from nornir import InitNornir

NornirObj = InitNornir(
    runner={
        "plugin": "QueueRunner",
        "options": {
            "num_workers": 100
        }
    }
)

QueueRunner Reference

class nornir_salt.plugins.runners.QueueRunner.QueueRunner(num_workers: int = 20)

QueueRunner run tasks over each host using queue together with workers threads consuming work from work queue.

Instead of firing up num_workers threads for each batch of hosts, QueueRunner starts num_workers threads once and uses queue to submit tasks and obtain results.

Parameters

num_workers – number of threads to use