DataProcessor Plugin

Processor plugin to process task results.

DataProcessor Sample Usage

Code to demonstrate how to use DataProcessor plugin:

from nornir import InitNornir
from nornir_netmiko import netmiko_send_command
from nornir_salt import DataProcessor

nr = InitNornir(config_file="config.yaml")

nr_with_processor = nr.with_processors([
    DataProcessor(
        [
            {"fun": "match", "pattern": "interface.*"}
        ]
    )
])

nr_with_processor.run(
    task=netmiko_send_command,
    command_string="show run"
)

Filtering mini-query-language specification

lod_filter, key_filter and find key name may be appended with check type specifier suffix to instruct what type of check to execute with criteria against key value in a format key_name__<check_type>. For example key_name__glob would use glob pattern matching.

Check Type

Description

Criteria Type

Functions Support

glob

Glob case sensitive pattern matching

string

find, lod_filter, key_filter

re

Regular Expression pattern matching

string

find, lod_filter, key_filter

eq

Equals

integer, string

find, lod_filter, key_filter

ge

Greater than or equal to

integer

find, lod_filter, key_filter

gt

Greater than

integer

find, lod_filter, key_filter

le

Less than or equal to

integer

find, lod_filter, key_filter

lt

Less than

integer

find, lod_filter, key_filter

in

Check if value in a list or in comma separated string

list, string

find, lod_filter, key_filter

contains

Value contains string

string

find, lod_filter, key_filter

DataProcessor Class reference

nornir_salt.plugins.processors.DataProcessor.DataProcessor(dp=[])

DataProcessor can process data obtained from devices. It is capable of:

  • loading data to Python structure from json, yaml, xml, csv or python format

  • serializing structured data to text in json, yaml, xml, csv or python format

  • filtering structured or text data

  • flattening and un-flattening nested data

  • parsing textual data into structured format

Parameters

dp – (list) list of Data Processors function names to pass results through

dp argument can be of one of these types:

  • comma separated string of function names

  • list of function name strings and/or dictionaries with function details

Data processor function dictionary items should have this structure:

{
    "fun": function name [str],
    "k1": "v1", ... "kN": "vN"
}

Where:

  • fun - Reference Name of DataProcessor function to run

  • kN - Any additional key-word arguments to use with function

DataProcessor Functions reference

DataProcessor Functions help to process results after task completed.

DataProcessor Functions

Reference Name

Description

find

Function to dispatch data to one of the filtering functions.

flake

Flattens python dictionary and filters its keys using key_filter

flatten

Turn a nested structure into a flattened dictionary

jmespath

Query JSON string or structured data using JMESPath library

key_filter

Filter data dictionary top keys using provided patterns.

load_json

Load JSON string into python dictionary structure

load_xml

Load XML string into python dictionary structure

lod_filter

List of Dictionaries (LOD) filter function

match

Search for regex pattern in devices output

parse_ttp

Parses text output from device into structured data

path

Retrieves content from nested structured data at given path

run_ttp

Parse text output from device sorting results across TTP inputs

to_json

Transform Python structure to JSON formatted string

to_pprint

Transform Python structure to pprint formatted string

to_str

Transform Python structure to string without doing any formatting

to_yaml

Transform Python structure to YAML formatted string

unflatten

Turn flat dictionary produced by flatten function to a nested structure

xml_flake

Transform XML in a flattened dictionary and filter keys using key_filter

xml_flatten

Transform XML in a flattened python dictionary

xml_rm_ns

Removes all namespace information from an XML Element tree

xml_to_json

Transform XML string to JSON string

xpath

Perform xpath search/filtering of XML string using LXML library

Formatter functions

Format structured data to json, yaml etc. text string

to_str

nornir_salt.plugins.processors.DataProcessor.to_str(data, **kwargs)

Reference Name to_str

Function to transform Python structure to string without applying any formatting

Parameters
  • data – (structure) Python structure to transform

  • kwargs – (dict) ignored

Returns

string

to_json

nornir_salt.plugins.processors.DataProcessor.to_json(data, **kwargs)

Reference Name to_json

Function to transform Python structure to JSON formatted string.

Parameters
  • data – (structure) Python structure to transform

  • kwargs – (dict) additional kwargs for json.dumps method

Returns

JSON formatted string

to_pprint

nornir_salt.plugins.processors.DataProcessor.to_pprint(data, **kwargs)

Reference Name to_pprint

Function to transform Python structure to pretty print string using pprint module

Parameters
  • data – (structure) Python structure to transform

  • kwargs – (dict) additional kwargs for pprint.pformat method

Returns

pretty print formatted string

to_yaml

nornir_salt.plugins.processors.DataProcessor.to_yaml(data, **kwargs)

Reference Name to_yaml

Function to transform Python structure to YAML formatted string

Dependencies: requires PyYAML library - pip install pyyaml

Parameters
  • data – (structure) Python structure to transform

  • kwargs – (dict) additional kwargs for yaml.dump method

Returns

pretty print formatted string

Loader functions

Load json, yaml etc. text into python structured data

load_xml

nornir_salt.plugins.processors.DataProcessor.load_xml(data, py_dict=True, **kwargs)

Reference Name load_xml

Load XML string into python dictionary structure using xmltodict library.

Dependencies: requires LXML library - pip install lxml

Parameters
  • data – (str) XML formatted string

  • py_dict – (bool) if True (default), will transform structure returned by xmltodict to normal dictionary instead of OrderedDict

  • kwargs – (dict) any additional **kwargs for xmltodict.parse method

Returns

python dictionary

load_json

nornir_salt.plugins.processors.DataProcessor.load_json(data, **kwargs)

Reference Name load_json

Load JSON string into python dictionary structure using json library.

Parameters
  • data – (str) JSON formatted string

  • kwargs – (dict) any additional **kwargs for json.loads method

Returns

python dictionary

Transform functions

Take structured data and return transformed structured data

flatten

nornir_salt.plugins.processors.DataProcessor.flatten(data, parent_key='', separator='.', **kwargs)

Reference Name flatten

Turn a nested structure (combination of lists/dictionaries) into a flattened dictionary.

This function is useful to explore deeply nested structures such as XML output obtained from devices over NETCONF.

Another usecase is filtering of the keys in resulted dictionary, as they are the strings, glob or regex matching can be applied on them.

Parameters
  • data – nested data to flatten

  • parent_key – string to prepend to dictionary’s keys, used by recursion

  • separator – string to separate flattened keys

Returns

flattened structure

Based on Stackoverflow answer: https://stackoverflow.com/a/62186053/12300761

All credits for the idea to https://github.com/ScriptSmith

Sample usage:

flatten({'a': 1, 'c': {'a': 2, 'b': {'x': 5, 'y' : 10}}, 'd': [1, 2, 3] })

>> {'a': 1, 'c.a': 2, 'c.b.x': 5, 'c.b.y': 10, 'd.0': 1, 'd.1': 2, 'd.2': 3}

unflatten

nornir_salt.plugins.processors.DataProcessor.unflatten(data, separator='.', **kwargs)

Reference Name unflatten

Turn flat dictionary produced by flatten function to a nested structure

Parameters
  • data – flattened dictionary

  • separator – string to split flattened keys

Returns

nested structure

List indexes must follow in order, for example this flattened dictionary:

{
    "5.a.b.0.c": 1,
    "5.a.b.1.c": 2,
    "10.a.b.0.c": 3,
    "10.a.b.1": 4,
}

Will produce unexpected results due to indexes not following numerical order:

[{'a': {'b': [{'c': 1}]}},
 {'a': {'b': [{'c': 2}]}},
 {'a': {'b': [{'c': 3}]}},
 {'a': {'b': [4]}}]}

xml_to_json

nornir_salt.plugins.processors.DataProcessor.xml_to_json(data, **kwargs)

Reference name xml_to_json

Dependencies: requires LXML library - pip install lxml

Function to transform XML string to JSON string.

Steps are:

  1. Transform XML to Python dictionary using xmltodict by calling load_xml function

  2. Serialize Python dictionary to JSON string by calling to_json function

Parameters
  • data – (str) XML formatted string

  • kwargs – (dict) kwargs to use with to_json function

Returns

JSON formatted string

xml_flatten

nornir_salt.plugins.processors.DataProcessor.xml_flatten(data, **kwargs)

Reference name xml_flatten

Dependencies: requires LXML library - pip install lxml

Function to transform XML in a flattened python dictionary representation

Steps are:

  1. Transform XML to Python dictionary using xmltodict by calling load_xml function

  2. Flatten python dictionary calling flatten function

Parameters
  • data – (str) XML formatted string

  • kwargs – (dict) kwargs to use with flatten function

Returns

flattened python dictionary

xml_rm_ns

nornir_salt.plugins.processors.DataProcessor.xml_rm_ns(data, recover=True, ret_xml=True, **kwargs)

Reference Name xml_rm_ns

Namespace clean up code taken from: https://github.com/jeremyschulman/xml-tutorial/blob/master/strip-namespaces.md

This function removes all namespace information from an XML Element tree so that a Caller can then use the xpath function without having to deal with the complexities of namespaces.

Dependencies: requires LXML library - pip install lxml

Parameters
  • data – (str) XML formatted string

  • recover – (bool) if True (default) uses etree.XMLParser(recover=True) to parse XML, can help to recover bad XML

  • ret_xml – (bool) indicates what to return, default is True

  • kwargs – (dict) any additional **kwargs are ignored

Returns

XML string with no namespaces if ret_xml is True, etree.Element otherwise

path

nornir_salt.plugins.processors.DataProcessor.path_(data, path, **kwargs)

Reference name path_

Function to retrieve content from nested structured data at given path.

Parameters
  • path – (str, list) dot separated path to result or list of path items

  • data – (dict) data to get results from

Returns

results at given path

Sample data:

{
    "VIP_cfg": {
        "1.1.1.1": {
            "config_state": "dis",
            "services": {
                "443": {
                    "https": [
                        {"real_port": "443"}
                    ],
                }
            }
        }
    }
}

With path "VIP_cfg.'1.1.1.1'.services.443.https.0.real_port" will return 443

Filter functions

Filter or search structured or text data

xpath

nornir_salt.plugins.processors.DataProcessor.xpath(data, expr, rm_ns=False, recover=False, **kwargs)

Reference Name xpath

Function to perform xpath search/filtering of XML string using LXML library.

Dependencies: requires LXML library - pip install lxml

Parameters
  • data – (str) XML formatted string

  • expr – (str) xpath expression to use

  • rm_ns – (bool) default is False, if True removes namespace from XML string using xml_rm_ns function

  • recover – (bool) default is False, if True uses etree.XMLParser(recover=True) while loading XML from string

  • kwarg – (dict) **kwargs to use for LXML etree.xpath method

Returns

XML filtered string

key_filter

nornir_salt.plugins.processors.DataProcessor.key_filter(data, pattern=None, checks_required=True, **kwargs)

Reference Name key_filter

Function to filter data dictionary top keys using provided patterns.

Parameters
  • data – (dictionary) Python dictionary

  • kwargs – (dict) any additional kwargs are key and value pairs, where key name is arbitrary and used to indicate check type following Filtering mini-query-language specification and value is the criteria to check. Default check type is glob case sensitive pattern matching.

  • pattern – (str) pattern to use for filtering

  • checks_required – (bool) if True (default) returns empty dictionary if no checks provided, returns all otherwise

Returns

filtered python dictionary

Default logic is key name must pass any of the criteria provided.

Sample usage:

key_filter(
    data=data_dictionary,
    pattern="1234*",
    pattern__glob="abc*",
    pattern2__glob="*abc*",
    pattern__re="abc.*",
)

Filtered dictionary key name must satisfy at least one of the matching criteria.

flake

nornir_salt.plugins.processors.DataProcessor.flake(data, **kwargs)

Reference name flake

FLAKE - Flattened Key Filter

Function to transform Python structure in a flattened python dictionary representation and filter its keys using key_filter function.

Steps are:

  1. Transform data to flattened Python dictionary using flatten function

  2. Filter Python dictionary keys using key_filter function

Parameters
  • data – (dict or list) structured data

  • kwargs – (dict) kwargs to use with key_filter function

Returns

flattened and filtered python dictionary

xml_flake

nornir_salt.plugins.processors.DataProcessor.xml_flake(data, pattern, **kwargs)

Reference name xml_flake

XML FLAKE - XML Flattened Key Filter

Function to transform XML in a flattened python dictionary representation and filter its keys using key_filter function

Steps are:

  1. Transform XML to flattened Python dictionary using xml_flatten function

  2. Filter Python dictionary keys using key_filter function

Parameters
  • data – (str) XML formatted string

  • kwargs – (dict) kwargs to use with key_filter function

Returns

flattened and filtered python dictionary

match

nornir_salt.plugins.processors.DataProcessor.match(data, pattern, before=0, **kwargs)

Reference name match

Function to search for regex pattern in devices output, similar to network devices include/match pipe statements.

Parameters
  • data – multiline string to search in

  • pattern – pattern to search for, glob (default) or regex

  • before – number of lines before match to include in results

Returns

filtered string

lod_filter

nornir_salt.plugins.processors.DataProcessor.lod_filter(data, pass_all=True, strict=True, checks_required=True, **kwargs)

Reference Name lod_filter

List of Dictionaries (LOD) filter function.

Iterates over list of dictionaries and returns dictionary items that have value of key matched by glob pattern.

Patterns are case sensitive.

Key value converted to string to perform match check.

Parameters
  • data – (list) list of dictionaries to search in

  • kwargs – (dict) any additional kwargs are key and value pairs, where key is a name of the dictionary key to search for and value is the criteria to check. Default check type is glob case sensitive pattern matching.

  • pass_all – (bool) if True (default) logic is AND - dictionary must pass ALL checks, if False logic is ANY

  • strict – (bool) if True (default) invalidates list dictionary item if no criteria key found in dictionary

  • checks_required – (bool) if True (default) returns empty list if no checks provided, returns all otherwise

Returns

filtered list of dictionaries

jmespath

nornir_salt.plugins.processors.DataProcessor.jmespath(data, expr, **kwargs)

Reference name jmespath

JMESPath is a query language for JSON

Function that uses JMESPath library to filter and search structured data.

Requires JMESPath package installed on the minion.

Parameters
  • data – (struct or str) list or dictionary structure or JSON formatted string

  • expr – (str) jmespath query expression

Returns

(struct) query results

find

nornir_salt.plugins.processors.DataProcessor.find(data, path=None, use_jmespath=False, use_xpath=False, **kwargs)

Reference name find

Function to dispatch data to one of the filtering functions based on data type.

Parameters
  • data – (list, dict, str) data to search in

  • use_jmespath – (bool) default is False, if True uses jmespath library

  • use_xpath – (bool) default is False, if True use lxml library xpath

  • path – (str) dot separated path or list of path items to results within data to search in; if use_jmespath is True, path must be jmespath query expression; if use_xpath is True, path must be lxml library xpath expression;

Returns

search results

Searching dispatch rules:

  • if use_jmespath is True uses jmespath function with path provided

  • if use_xpath is True uses xpath function with path provided

Next, if supplied data is a list or dictionary and path provided calls path_ function to narrow down results before dispatching them further:

  • if result type is a list uses lod_filter

  • if result type is a dictionary uses key_filter

  • if result type is a string uses match function

Parse functions

Produce structured data out of text by parsing it.

parse_ttp

nornir_salt.plugins.processors.DataProcessor.parse_ttp(data: str, template: str, ttp_kwargs={}, res_kwargs={}, **kwargs)

Reference name parse_ttp

Dependencies: requires TTP library - pip install ttp

Function to parse text output from device and return structured data

Parameters
  • data – (str) string to parse

  • template – (str) TTP template string or reference to ttp:// templates

  • ttp_kwargs – (dict) dictionary to use while instantiating TTP parse object

  • res_kwargs – (dict) dictionary to use with result method

Returns

parsed structure

run_ttp

nornir_salt.plugins.processors.DataProcessor.run_ttp(data, template, ttp_kwargs={}, res_kwargs={}, task=None, remove_tasks=True, **kwargs)

Reference name run_ttp

Dependencies: requires TTP library - pip install ttp

Function to parse text output from device sorting results across TTP inputs based on commands values.

Parameters
  • data – (str) Nornir MultiResult object

  • template – (str) TTP template string or reference to ttp:// templates

  • ttp_kwargs – (dict) dictionary to use while instantiating TTP parse object

  • res_kwargs – (dict) dictionary to use with result method

  • remove_tasks – (bool) if set to True and data is MultiResult object will remove other task results

  • task – (obj) Nornir Task object, used to form parsing results when data is MultiResult object

Returns

parsed structure

Provided Nornir MultiResult object processed by sorting task results across TTP Template inputs to parse. After parsing, all other tasks’ results removed from MultiResult object and parsing results appended to it. If remove_tasks set to False, other tasks results not removed.

Sample usage:

import pprint
from nornir import InitNornir
from nornir_salt import ResultSerializer, DataProcessor, netmiko_send_commands

nr = InitNornir(config_file="nornir_config.yaml")

# define TTP template with inputs having commands attributes
template = '''
<input name="arp">
commands = ["show arp"]
</input>

<input name="version">
commands = ["show version"]
</input>

<group name="arp_cache*" input="arp">
Internet  {{ ip }}   {{ age }}   {{ mac }}  ARPA   {{ interface }}
</group>

<group name="facts.version" input="version">
Cisco IOS XE Software, Version {{ iose_xe_version }}
</group>
'''

# add data processor with run_ttp function
nr_with_dp = nr.with_processors([DataProcessor(
    [{"fun": "run_ttp", "template": template}]
)])

# run task; commands for task will be dynamically populated by DataProcessor
# run_ttp function with commands defined within TTP template inputs
result = nr_with_dp.run(task=netmiko_send_commands)

# serialize results to dictionary
dict_result = ResultSerializer(result)

pprint.pprint(dict_result)

Running above code should print results:

{'hostname': {'run_ttp': [[{'arp_cache': [{'age': '2',
                            'interface': 'GigabitEthernet1',
                            'ip': '10.10.20.28',
                            'mac': '0050.56bf.f0be'},
                           {'age': '-',
                            'interface': 'GigabitEthernet1',
                            'ip': '10.10.20.48',
                            'mac': '0050.56bf.9379'}]},
            {'facts': {'version': {'iose_xe_version': '16.09.03'}}}]]}}

Misc functions

Various functions with miscellaneous or utility purpose.

add_commands_from_ttp_template

nornir_salt.plugins.processors.DataProcessor.add_commands_from_ttp_template(task, template, **kwargs)

Function to extract commands from TTP template and add them to task params. Used in conjunction with run_ttp parsing function.

This function called at task_started point, allowing to update task’s params dictionary with commands to collect from devices prior to further executing the task.

Commands extracted from TTP Template inputs that may contain definitions of commands list.

Dependencies: requires TTP library - pip install ttp.

Parameters
  • task – (obj) Nornir task object

  • template – (str) TTP Template string