Get started with NSO automation by understanding fundamental concepts.
Loading...
Loading...
Loading...
Loading...
Loading...
Learn how NSO keeps a record of its managed devices using CDB.
Cisco NSO is a network automation platform that supports a variety of uses. This can be as simple as a configuration of a standard-format hostname, which can be implemented in minutes. Or it could be an advanced MPLS VPN with custom traffic-engineered paths in a Service Provider network, which might take weeks to design and code.
Regardless of complexity, any network automation solution must keep track of two things: intent and network state.
The Configuration Database (CDB) built into NSO was designed for this exact purpose:
Firstly, the CDB will store the intent, which describes what you want from the network. Traditionally we call this intent a network service since this is what the network ultimately provides to its users.
Secondly, the CDB also stores a copy of the configuration of the managed devices, that is, the network state. Knowledge of the network state is essential to correctly provision new services. It also enables faster diagnosis of problems and is required for advanced functionality, such as self-healing.
This section describes the main features of the CDB and explains how NSO stores data there. To help you better understand the structure of the CDB, you will also learn how to add your data to it.
The CDB is a dedicated built-in storage for data in NSO. It was built from the ground up to efficiently store and access network configuration data, such as device configurations, service parameters, and even configuration for NSO itself. Unlike traditional SQL databases that store data as rows in a table, the CDB is a hierarchical database, with a structure resembling a tree. You could think of it as somewhat like a big XML document that can store all kinds of data.
There are a number of other features that make the CDB an excellent choice for a configuration store:
Fast lightweight database access through a well-defined API.
Subscription (“push”) mechanism for change notification.
Transaction support for ensuring data consistency.
Rich and extensible schema based on YANG.
Built-in support for schema and associated data upgrade.
Close integration with NSO for low-maintenance operation.
To speed up operations, CDB keeps a configurable amount of configuration data in RAM, in addition to persisting it to disk (see CDB Persistence for details). The CDB also stores transient operational data, such as alarms and traffic statistics. By default, this operational data is only kept in RAM and is reset during restarts, however, the CDB can be instructed to persist it if required.
The automatic schema update feature is useful not only when performing an actual upgrade of NSO itself, it also simplifies the development process. It allows individual developers to add and delete items in the configuration independently.
Additionally, the schema for data in the CDB is defined with a standard modeling language called YANG. YANG (RFC 7950, https://tools.ietf.org/html/rfc7950) describes constraints on the data and allows the CDB to store values more efficiently.
All of the data stored in the CDB follows the data model provided by various YANG modules. Each module usually comes as one or more files with a .yang extension and declares a part of the overall model.
NSO provides a base set of YANG modules out of the box. They are located in $NCS_DIR/src/ncs/yang if you wish to inspect them. These modules are required for proper system operation.
All other YANG modules are provided by packages and extend the base NSO data model. For example, each Network Element Driver (NED) package adds the required nodes to store the configuration for that particular type of device. In the same way, you can store your custom data in the CDB by providing a package with your own YANG module.
However, the CDB can't use the YANG files directly. The bundled compiler, ncsc, must first transform a YANG module into a final schema (.fxs) file. The reason is that internally and in the programming APIs NSO refers to YANG nodes with integer values instead of names. This conserves space and allows for more efficient operations, such as switch statements in the application code. The .fxs file contains this mapping and needs to be recreated if any part of the YANG model changes. The compilation process is usually started from the package Makefile by the make command.
Ensure that:
No previous NSO or netsim processes are running. Use the ncs --stop and ncs-netsim stop commands to stop them if necessary.
NSO Local Install with a fresh runtime directory has been created by the ncs-setup --dest ~/nso-lab-rundir or similar command.
The environment variable NSO_RUNDIR points to this runtime directory, such as set by the export NSO_RUNDIR=~/nso-lab-rundir command. It enables the below commands to work as-is, without additional substitution needed.
The easiest way to add your data fields to the CDB is by creating a service package. The package includes a YANG file for the service-specific data, which you can customize. You can create the initial package by simply invoking the ncs-make-package command. This command also sets up a Makefile with the code for compiling the YANG model.
Use the following command to create a new package:
The command line switches instruct the command to compile the YANG file and place the package in the right location.
Now start the NSO process if it is not running already and connect to the CLI:
Next, instruct NSO to load the newly created package:
Once the package loading process is completed, you can verify the data model from your package was incorporated into NSO. Use the show command, which now supports an additional parameter:
This command tells you that NSO knows about the extended data model but there is no actual data configured for it yet.
More interestingly, you are now able to add custom entries to the configuration. First, enter the CLI configuration mode:
Then add an arbitrary entry under my-data-entries:
What is more, you can also set a dummy IP address:
However, if you try to use something different from a dummy, you will get an error. Likewise, if you try to assign a dummy a value that is not an IP address. How did NSO learn about this dummy value?
If you assumed from the YANG file, you are correct. YANG files provide the schema for the CDB and that dummy value comes from the YANG model in your package. Let's take a closer look.
Exit the configuration mode and discard the changes by typing abort:
Open the YANG file in an editor or list its contents from the CLI with the following command:
At the start of the output, you can see the module my-data-entries, which contains your data model. By default, the ncs-make-package gives it the same name as the package. You can check that this module is indeed loaded:
The list my-data-entries statement, located a bit further down in the YANG file, allowed you to add custom entries before. And near the end of the output, you can find the leaf dummy definition, with IPv4 as the type. This is the source of information that enables NSO to enforce a valid IP address as the value.
NSO uses YANG to structure and enforce constraints on data that it stores in the CDB. YANG was designed to be extensible and handle all kinds of data modeling, which resulted in a number of language features that helped achieve this goal. However, there are only four fundamental elements (node types) for describing data:
leaf nodes
leaf-list nodes
container nodes
list nodes
You can then combine these elements into a complex, tree-like structure, which is why we refer to individual elements as nodes (of the data tree). In general, YANG separates nodes into those that hold data (leaf, leaf-list) and those that hold other nodes (container, list).
A leaf contains simple data such as an integer or a string. It has one value of a particular type and no child nodes. For example:
This code describes the structure that can hold a value of a hostname (of some device). A leaf node is used because the hostname only has a single value, that is, the device has one (canonical) hostname. In the NSO CLI, you set a value of a leaf simply as:
A leaf-list is a sequence of leaf nodes of the same type. It can hold multiple values, very much like an array. For example:
This code describes a data structure that can hold many values, such as a number of domain names. In the CLI, you can assign multiple values to a leaf-list with the help of square bracket syntax:
leaf and leaf-list describe nodes that hold simple values. As a model keeps expanding, having all data nodes on the same (top) level can quickly become unwieldy. A container node is used to group related nodes into a subtree. It has only child nodes and no value. A container may contain any number of child nodes of any type (including leafs, lists, containers, and leaf-lists). For example:
This code defines the concept of a server administrator. In the CLI, you first select the container before you access the child nodes:
Similarly, a list defines a collection of container-like list entries that share the same structure. Each entry is like a record or a row in a table. It is uniquely identified by the value of its key leaf (or leaves). A list definition may contain any number of child nodes of any type (leafs, containers, other lists, and so on). For example:
This code defines a list of users (of which there can be many), where each user is uniquely identified by their name. In the CLI, lists take an additional parameter, the key value, to select a single entry:
To set a value of a particular list entry, first specify the entry, then the child node, like so:
Combining just these four fundamental YANG node types, you can build a very complex model that describes your data. As an example, the model for the configuration of a Cisco IOS-based network device, with its myriad features, is created with YANG. However, it makes sense to start with some simple models, to learn what kind of data they can represent and how to alter that data with the CLI.
Ensure that:
No previous NSO or netsim processes are running. Use the ncs --stop and ncs-netsim stop commands to stop them if necessary.
NSO Local Install with a fresh runtime directory has been created by the ncs-setup --dest ~/nso-lab-rundir or similar command.
The environment variable NSO_RUNDIR points to this runtime directory, such as set by the export NSO_RUNDIR=~/nso-lab-rundir command. It enables the below commands to work as-is, without additional substitution needed.
You can add custom data models to NSO by using packages. So, you will build a package to hold the YANG module that represents your model. Use the following command to create a package (if you are building on top of the previous showcase, the package may already exist and will be updated):
Change the working directory to the directory of your package:
You will place the YANG model into the src/yang/my-test-model.yang file. In a text editor, create a new file and add the following text at the start:
The first line defines a new module and gives it a name. In addition, there are two more statements required: the namespace and prefix. Their purpose is to help avoid name collisions.
Add a statement for each of the four fundamental YANG node types (leaf, leaf-list, container, list) to the my-test-model.yang model.
Also, add the closing bracket for the module at the end:
Remember to finally save the file as my-test-model.yang in the src/yang/ directory of your package. It is a best practice for the name of the file to match the name of the module.
Having completed the model, you must compile it into an appropriate (.fxs) format. From the text editor first, return to the shell and then run the make command in the src/ subdirectory of your package:
The compiler will report if there are errors in your YANG file, and you must fix them before continuing.
Next, start the NSO process and connect to the CLI:
Finally, instruct NSO to reload the packages:
Enter the configuration mode by using the config command and test out how to set values for the data nodes you have defined in the YANG model:
host-name leaf
domains leaf-list
server-admin container
user-info list
Use the ? and TAB keys to see the possible completions.
Now feel free to go back and experiment with the YANG file to see how your changes affect the data model. Just remember to rebuild and reload the package after you make any changes.
Adding a new YANG module to the CDB enables it to store additional data, however, there is nothing in the CDB for this module yet. While you can add configuration with the CLI, for example, there are situations where it makes sense to start with some initial data in the CDB already. This is especially true when a new instance starts for the first time and the CDB is empty.
In such cases, you can bootstrap the CDB data with XML files. There are various uses for this feature. For example, you can implement some default “factory settings” for your module or you might want to pre-load data when creating a new instance for testing.
In particular, some of the provided examples use the CDB init files mechanism to save you from typing out all of the initial configuration commands by hand. They do so by creating a file with the configuration encoded in the XML format.
When starting empty, the CDB will try to initialize the database from all XML files found in the directories specified by the init-path and db-dir settings in ncs.conf (please see ncs.conf(5) in Manual Pages for exact details). The loading process scans the files with the .xml suffix and adds all the data in a single transaction. In other words, there is no specified order in which the files are processed. This happens early during start-up, during the so-called start phase 1, described in Starting NSO.
The content of the init file does not need to be a complete instance document but can specify just a part of the overall data, very much like the contents of the NETCONF edit-config operation. However, the end result of applying all the files must still be valid according to the model.
It is a good practice to wrap the data inside a config element, as it gives you the option to have multiple top-level data elements in a single file while it remains a valid XML document. Otherwise, you would have to use separate files for each of them. The following example uses the config element to fit all the elements into a single file.
There are many ways to generate the XML data. A common approach is to dump existing data with the ncs_load utility or the display xml filter in the CLI. All of the data in the CDB can be represented (or exported, if you will) in XML. This is no coincidence. XML was the main format for encoding data with NETCONF when YANG was created and you can trace the origin of some YANG features back to XML.
Implement basic automation with Python.
You can manipulate data in the CDB with the help of XML files or the UI, however, these approaches are not well suited for programmatic access. NSO includes libraries for multiple programming languages, providing a simpler way for scripts and programs to interact with it. The Python Application Programming Interface (API) is likely the easiest to use.
This section will show you how to read and write data using the Python programming language. With this approach, you will learn how to do basic network automation in just a few lines of code.
The environment setup that happens during the sourcing of the ncsrc file also configures the PYTHONPATH environment variable. It allows the Python interpreter to find the NSO modules, which are packaged with the product. This approach also works with Python virtual environments and does not require installing any packages.
Since the ncsrc file takes care of setting everything up, you can directly start the Python interactive shell and import the main ncs module. This module is a wrapper around a low-level C _ncs module that you may also need to reference occasionally. Documentation for both of the modules is available through the built-in help() function or separately in the HTML format.
If the import ncs statement fails, please verify that you are using a supported Python version and that you have sourced the ncsrc beforehand.
Generally, you can run the code from the Python interactive shell but we recommend against it. The code uses nested blocks, which are hard to edit and input interactively. Instead, we recommend you save the code to a file, such as script.py, which you can then easily run and rerun with the python3 script.py command. If you would still like to interactively inspect or alter the values during the execution, you can use the import pdb; pdb.set_trace() statements at the location of interest.
With NSO, data reads and writes normally happen inside a transaction. Transactions ensure consistency and avoid race conditions, where simultaneous access by multiple clients could result in data corruption, such as reading half-written data. To avoid this issue, NSO requires you to first start a transaction with a call to ncs.maapi.single_read_trans() or ncs.maapi.single_write_trans(), depending on whether you want to only read data or read and write data. Both of them require you to provide the following two parameters:
user: The username (string) of the user you wish to connect as
context: Method of access (string), allowing NSO to distinguish between CLI, web UI, and other types of access, such as Python scripts
These parameters specify security-related information that is used for auditing, access authorization, and so on. Please refer to AAA infrastructure for more details.
As transactions use up resources, it is important to clean up after you are done using them. Using a Python with code block will ensure that cleanup is automatically performed after a transaction goes out of scope. For example:
In this case, the variable t stores the reference to a newly started transaction. Before you can actually access the data, you also need a reference to the root element in the data tree for this transaction. That is, the top element, under which all of the data is located. The ncs.maagic.get_root() function, with transaction t as a parameter, achieves this goal.
Once you have the reference to the root element, say in a variable named root, navigating the data model becomes straightforward. Accessing a property on root selects a child data node with the same name as the property. For example, root.nacm gives you access to the nacm container, used to define fine-grained access control. Since nacm is itself a container node, you can select one of its children using the same approach. So, the code root.nacm.enable_nacm refers to another node inside nacm, called enable-nacm. This node is a leaf, holding a value, which you can print out with the Python print() function. Doing so is conceptually the same as using the show running-config nacm enable-nacm command in the CLI.
There is a small difference, however. Notice that in the CLI the enable-nacm is hyphenated, as this is the actual node name in YANG. But names must not include the hyphen (minus) sign in Python, so the Python code uses an underscore instead.
The following is the full source code that prints the value:
As you can see in this example, it is necessary to import only the ncs module, which automatically imports all the submodules. Depending on your NSO instance, you might also notice that the value printed is True, without any quotation marks. As a convenience, the value gets automatically converted to the best-matching Python type, which in this case is a boolean value (True or False).
Moreover, if you start a read/write transaction instead of a read-only one, you can also assign a new value to the leaf. Of course, the same validation rules apply as using the CLI and you need to explicitly commit the transaction if you want the changes to persist. A call to the apply() method on the transaction object t performs this function. Here is an example:
You can access a YANG list node like how you access a leaf. However, working with a list more resembles working with Python dict than a list, even though the name would suggest otherwise. The distinguishing feature is that YANG lists have keys that uniquely identify each list item. So, lists are more naturally represented as a kind of dictionary in Python.
Let's say there is a list of customers defined in NSO, with a YANG schema such as:
To simplify the code, you might want to assign the value of root.customers.customer to a new variable our_customers. Then you can easily access individual customers (list items) by their id. For example, our_customers['ACME'] would select the customer with id equal to ACME. You can check for the existence of an item in a list using the Python in operator, for example, 'ACME' in our_customers. Having selected a specific customer using the square bracket syntax, you can then access the other nodes of this item.
Compared to dictionaries, making changes to YANG lists is quite a bit different. You cannot just add arbitrary items because they must obey the YANG schema rules. Instead, you call the create() method on the list object and provide the value for the key. This method creates and returns a new item in the list if it doesn't exist yet. Otherwise, the method returns the existing item. And for item removal, use the Python built-in del function with the list object and specify the item to delete. For example, del our_customers['ACME'] deletes the ACME customer entry.
In some situations, you might want to enumerate all of the list items. Here, the list object can be used with the Python for syntax, which iterates through each list item in turn. Note that this differs from standard Python dictionaries, which iterate through the keys. The following example demonstrates this behavior.
Now let's see how you can use this knowledge for network automation.
No previous NSO or netsim processes are running. Use the ncs --stop and ncs-netsim stop commands to stop them if necessary.
Leveraging one of the examples included with the NSO installation allows you to quickly gain access to an NSO instance with a few devices already onboarded. The getting-started/developing-with-ncs set of examples contains three simulated routers that you can configure.
Navigate to the 0-router-network directory with the following command.
You can prepare and start the routers by running the make and netsim commands from this directory.
With the routers running, you should also start the NSO instance that will allow you to manage them.
In case the ncs command reports an error about an address already in use, you have another NSO instance already running that you must stop first (ncs --stop).
Before you can use Python to configure the router, you need to know what to configure. The simplest way to find out how to configure the DNS on this type of router is by using the NSO CLI.
In the CLI, you can verify that the NSO is managing three routers and check their names with the following command:
To make sure that the NSO configuration matches the one deployed on routers, also perform a sync-from action.
Let's say you would like to configure the DNS server 192.0.2.1 on the ex1 router. To do this by hand, first enter the configuration mode.
Then navigate to the NSO copy of the ex1 configuration, which resides under the devices device ex1 config path, and use the ? and TAB keys to explore the available configuration options. You are looking for the DNS configuration.
...
Once you have found it, you see the full DNS server configuration path: devices device ex1 config sys dns server.
As an alternative to using the CLI approach to find this path, you can also consult the data model of the router in the packages/router/src/yang/ directory.
As you won't be configuring ex1 manually at this point, exit the configuration mode.
Instead, you will create a Python script to do it, so exit the CLI as well.
You will place the script into the ex1-dns.py file.
In a text editor, create a new file and add the following text at the start.\
The root variable allows you to access configuration in the NSO, much like entering the configuration mode on the CLI does.
Next, you will need to navigate to the ex1 router. It makes sense to assign it to the ex1_device variable, which makes it more obvious what it refers to and easier to access in the script.
In NSO, each managed device, such as the ex1 router, is an entry inside the device list. The list itself is located in the devices container, which is a common practice for lists. The list entry for ex1 includes another container, configwhere the copy of ex1 configuration is kept. Assign it to the ex1_config variable.
Alternatively, you can assign to ex1_config directly, without referring to ex1_device, like so:
This is the equivalent of using devices device ex1 config on the CLI.
For the last part, keep in mind the full configuration path you found earlier. You have to keep navigating to reach the server list node. You can do this through the sys and dns nodes on the ex1_config variable.
DNS configuration typically allows specifying multiple servers for redundancy and is therefore modeled as a list. You add a new DNS server with the create() method on the list object.
Having made the changes, do not forget to commit them with a call to apply() or they will be lost.
Alternatively, you can use the dry-run parameter with the apply_params() to, for example, preview what will be sent to the device.
Lastly, add a simple print statement to notify you when the script is completed.
Save the script file as ex1-dns.py and run it with the python3 command.
You should see Done! printed out. Then start the NSO CLI to verify the configuration change.
Finally, you can check the configured DNS servers on ex1 by using the show running-config command.
If you see the 192.0.2.1 address in the output, you have successfully configured this device using Python!
The code in this chapter is intentionally kept simple to demonstrate the core concepts and lacks robustness in error handling. In particular, it is missing the retry mechanism in case of concurrency conflicts as described in Handling Conflicts.
Perhaps you've wondered about the unusual name of Python ncs.maagic module? It is not a typo but a portmanteau of the words Management Agent API (MAAPI) and magic. The latter is used in the context of so-called magic methods in Python. The purpose of magic methods is to allow custom code to play nicely with the Python language. An example you might have come across in the past is the __init__() method in a class, which gets called whenever you create a new object. This one and similar methods are called magic because they are invoked automatically and behind the scenes (implicitly).
The NSO Python API makes extensive use of such magic methods in the ncs.maagic module. Magic methods help this module translate an object-based, user-friendly programming interface into low-level function calls. In turn, the high-level approach to navigating the data hierarchy with ncs.maagic objects is called the Python Maagic API.
Get started with service development using a simple example.
The device YANG models contained in the Network Element Drivers (NEDs) enable NSO to store device configurations in the CDB and expose a uniform API to the network for automation, such as by Python scripts. The concept of NSO services builds on top of this network API and adds the ability to store service-specific parameters with each service instance.
This section introduces the main service building blocks and shows you how to build one yourself.
Network automation includes provisioning and de-provisioning configuration, even though the de-provisioning part often doesn't get as much attention. It is nevertheless significant since leftover, residual configuration can cause hard-to-diagnose operational problems. Even more importantly, without proper de-provisioning, seemingly trivial changes may prove hard to implement correctly.
Consider the following example. You create a simple script that configures a DNS server on a router, by adding the IP address of the server to the DNS server list. This should work fine for initial provisioning. However, when the IP address of the DNS server changes, the configuration on the router should be updated as well.
Can you still use the same script in this case? Most likely not, since you need to remove the old server from the configuration and add the new one. The original script would just add the new IP address after the old one, resulting in both entries on the device. In turn, the device may experience slow connectivity as the system periodically retries the old DNS IP address and eventually times out.
The following figure illustrates this process, where a simple script first configures the IP address 192.0.2.1 (“.1”) as the DNS server, then later configures 192.0.2.8 (“.8”), resulting in a leftover old entry (“.1”).
In such a situation, the script could perhaps simply replace the existing configuration, by removing all existing DNS server entries before adding the new one. But is this a reliable practice? What if a device requires an additional DNS server that an administrator configured manually? It would be overwritten and lost.
In general, the safest approach is to keep track of the previous changes and only replace the parts that have changed. This, however, is a lot of work and nontrivial to implement yourself. Fortunately, NSO provides such functionality through the FASTMAP algorithm, which is used when deploying services.
The other major benefit of using NSO services for automation is the service interface definition using YANG, which specifies the name and format of the service parameters. Many new NSO users wonder why use a service YANG model when they could just use the Python code or templates directly. While it might be difficult to see the benefits without much prior experience, YANG allows you to write better, more maintainable code, which simplifies the solution in the long run.
Many, if not most, security issues and provisioning bugs stem from unexpected user input. You must always validate user input (service parameter values) and YANG compels you to think about that when writing the service model. It also makes it easy to write the validation rules by using a standardized syntax, specifically designed for this purpose.
Moreover, the separation of concerns into the user interface, validation, and provisioning code allows for better organization, which becomes extremely important as the project grows. It also gives NSO the ability to automatically expose the service functionality through its APIs for integration with other systems.
For these reasons, services are the preferred way of implementing network automation in NSO.
As you may already know, services are added to NSO with packages. Therefore, you need to create a package if you want to implement a service of your own. NSO ships with an ncs-make-package utility that makes creating packages effortless. Adding the --service-skeleton python option creates a service skeleton, that is, an empty service, which you can tailor to your needs. As the last argument, you must specify the package name, which in this case is the service name. The command then creates a new directory with that name and places all the required files in the appropriate subdirectories.
The package contains the two most important parts of the service:
the service YANG model and
the service provisioning code also called the mapping logic.
Let's first look at the provisioning part. This is the code that performs the network configuration necessary for your service. The code often includes some parameters, for example, the DNS server IP address or addresses to use if your service is in charge of DNS configuration. So, we say that the code maps the service parameters into the device parameters, which is where the term mapping logic originates from. NSO, with the help of the NED, then translates the device parameters to the actual configuration. This simple tree-to-tree mapping describes how to create the service and NSO automatically infers how to update, remove, or re-deploy the service, hence the name FASTMAP.
How do you create the provisioning code and where do you place it? Is it similar to a stand-alone Python script? Indeed, the code is mostly the same. The main difference is that now you don't have to create a session and a transaction yourself because NSO already provides you with one. Through this transaction, the system tracks the changes to the configuration made by your code.
The package skeleton contains a directory called python. It holds a Python package named after your service. In the package, the ServiceCallbacks class (the main.py file) is used for provisioning code. The same file also contains the Main class, which is responsible for registering the ServiceCallbacks class as a service provisioning code with NSO.
Of the most interest is the cb_create() method of the ServiceCallbacks class:
NSO calls this method for service provisioning. Now, let's see how to evolve a stand-alone automation script into a service. Suppose you have Python code for DNS configuration on a router, similar to the following:
Taking into account the cb_create() signature and the fact that the NSO manages the transaction for a service, you won't need the transaction and root variable setup. The NSO service framework already takes care of setting up the root variable with the right transaction. There is also no need to call apply() because NSO does that automatically.
You only have to provide the core of the code (the middle portion in the above stand-alone script) to the cb_create():
You can run this code by adding the service package to NSO and provisioning a service instance. It will achieve the same effect as the stand-alone script but with all the benefits of a service, such as tracking changes.
In practice, all services have some variable parameters. Most often parameter values change from service instance to service instance, as the desired configuration is a little bit different for each of them. They may differ in the actual IP address that they configure or in whether the switch for some feature is on or off. Even the DNS configuration service requires a DNS server IP address, which may be the same across the whole network but could change with time if the DNS server is moved elsewhere. Therefore, it makes sense to expose the variable parts of the service as service parameters. This allows a service operator to set the parameter value without changing the service provisioning code.
With NSO, service parameters are defined in the service model, written in YANG. The YANG module describing your service is part of the service package, located under the src/yang path, and customarily named the same as the package. In addition to the module-related statements (description, revision, imports, and so on), a typical service module includes a YANG list, named after the service. Having a list allows you to configure multiple service instances with slightly different parameter values. For example, in a DNS configuration service, you might have multiple service instances with different DNS servers. The reason is, that some devices, such as those in the Demilitarized Zone (DMZ), might not have access to the internal DNS servers and would need to use a different set.
The service model skeleton already contains such a list statement. The following is another example, similar to the one in the skeleton:
Along with the description, the service specifies a key, nameto uniquely identify each service instance. This can be any free-form text, as denoted by its type (string). The statements starting with tailf: are NSO-specific extensions for customizing the user interface NSO presents for this service. After that come two lines, the uses and ncs:servicepoint, which tells NSO this is a service and not just some ordinary list. At the end, there are two parameters defined, device and server-ip.
NSO then allows you to add the values for these parameters when configuring a service instance, as shown in the following CLI transcript:
Finally, your Python script can read the supplied values inside the cb_create() method via the provided service variable. This variable points to the currently-provisioning service instance, allowing you to use code such as service.server_ip for the value of the server-ip parameter.
No previous NSO or netsim processes are running. Use the ncs --stop and ncs-netsim stop commands to stop them if necessary.
NSO Local Install with a fresh runtime directory has been created by the ncs-setup --dest ~/nso-lab-rundir or a similar command.
The environment variable NSO_RUNDIR points to this runtime directory, such as set by the export NSO_RUNDIR=~/nso-lab-rundir command. It enables the below commands to work as-is, without additional substitution needed.
The getting-started/developing-with-ncs set of examples contains three simulated routers that you can use for this scenario. The 0-router-network directory holds the data necessary for starting the routers and connecting them to your NSO instance.
First, change the current working directory:
From this directory, you can start a fresh set of routers by running the following make command:
The routers are now running. The required NED package and a CDB initialization file ncs-cdb/ncs_init.xmlwere also added to your NSO instance. The latter contains connection details for the routers and will be automatically loaded on the first NSO start.
In case you're not using a fresh working directory, you may need to use the ncs_load command to load the file manually. Older versions of the system may also be missing the above make target, which you can add to the Makefile yourself:
You create a new service package with the ncs-make-package command. Without the --dest option, the package is created in the current working directory. Normally you run the command without this option, as it is shorter. For NSO to find and load this package, it has to be placed (or referenced via a symbolic link) in the packages subfolder of the NSO running directory.
Change the current working directory before creating the package:
You need to provide two parameters to ncs-make-package. The first is the --service-skeleton python option, which selects the Python programming language for scaffolding code. The second parameter is the name of the service. As you are creating a service for DNS configuration, dns-config is a fitting name for it. Run the final, full command:
If you look at the file structure of the newly created package, you will see it contains a number of files.
The package-meta-data.xml describes the package and tells NSO where to find the code. Inside the python folder is a service-specific Python package, where you add your own Python code (to main.py file). There is also a README file that you can update with the information relevant to your service. The src folder holds the source code that you must compile before you can use it with NSO. That's why there is also a Makefile that takes care of the compilation process. In the yang subfolder is the service YANG module. The templates folder can contain additional XML files, discussed later. Lastly, there's the test folder where you can put automated testing scripts, which won't be discussed here.
While you can always hard-code the desired parameters, such as the DNS server IP address, in the Python code, it means you have to change the code every time the parameter value (the IP address) changes. Instead, you can define it as an input parameter in the YANG file. Fortunately, the skeleton already has a leaf called a dummy that you can rename and use for this purpose.
Open the dns-config.yang, located inside dns-config/src/yang/, in a text or code editor and find the following line:
Replace the word dummy with the word dns-server, save the file, and return to the shell. Run the make command in the dns-config/src folder to compile the updated YANG file.
In a text or code editor, open the main.py file, located inside dns-config/python/dns_config/. Find the following snippet:
Right after the self.log.info() call, read the value of the dns-server parameter into a dns_ip variable:
Mind the 8 spaces in front to make sure that the line is correctly aligned. After that, add the code that configures the ex1 router:
Here, you are using the dns_ip variable that contains the operator-provided IP address instead of a hard-coded value. Also, note that there is no need to check if the entry for this DNS server already exists in the list.
In the end, the cb_create() method should look like the following:
Save the file and let's see the service in action!
Start the NSO from the running directory:
Then, start the NSO CLI:
If you have started a fresh NSO instance, the packages are loaded automatically. Still, there's no harm in requesting a package reload anyway:
As you will be making changes on the simulated routers, make sure NSO has their current configuration with the devices sync-from command.
Now you can test out your service package by configuring a service instance. First, enter the configuration mode.
Configure a test instance and specify the DNS server IP address:
The easiest way to see configuration changes from the service code is to use the commit dry-run command.
The output tells you the new DNS server is being added in addition to an existing one already there. Commit the changes:
Finally, change the IP address of the DNS server:
With the help of commit dry-run observe how the old IP address gets replaced with the new one, without any special code needed for provisioning.
The DNS configuration example intentionally performs very little configuration, a single line really, to focus on the service concepts. In practice, services can become more complex in two different ways. First, the DNS configuration service takes the IP address of the DNS server as an input parameter, supplied by the operator. Instead, the provisioning code could leverage another system, such as an IP Address Management (IPAM), to get the required information. In such cases, you have to add additional logic to your service code to generate the parameters (variables) to be used for configuration.
Second, generating the configuration from the parameters can become more complex when it touches multiple subsystems or spans across multiple devices. An example would be a service that adds a new VLAN, configures an IP address and a DHCP server, and adds the new route to a routing protocol. Or perhaps the service has to be duplicated on two separate devices for redundancy.
An established approach to the second challenge is to use a templating system for configuration generation. Templates separate the process of constructing parameter values from how they are used, adding a degree of flexibility and decoupling. NSO uses XML-based configuration (config) templates, which you can invoke from provisioning code or link directly to services. In the latter case, you don't even have to write any Python code.
XML templates are snippets of configuration, similar to the CDB init files, but more powerful. Let's see how you could implement the DNS configuration service using a template instead of navigating the YANG model with Python.
While it is possible to write an XML template from scratch, it has to follow the target YANG model. Fortunately, the NSO CLI can help with generating most parts of the template from changes to the currenly open transaction. First, you'll need a sample instance with the desired configuration. As you are configuring the DNS server on a router and the ex1 device already has one configured, you can reuse that one. Otherwise, you might configure one by hand, using the CLI. You do that by displaying the existing configuration in the format of an XML template and saving it to a file, by piping it through the display xml-template and save filters, as shown here:
The file structure of a package usually contains a templates folder and that is where the template belongs. When loading packages, NSO will scan this folder and process any .xml files it finds as templates.
Of course, a template with hard-coded values is of limited use, as it would always produce the exact same configuration. It becomes a lot more useful with variable substitution. In its simplest form, you define a variable value in the provisioning (Python) code and reference it from the XML template, by using curly braces and a dollar sign: {$VARIABLE}. Also, many users prefer to keep the variable name uppercased to make it stand out more from the other XML elements in the file. For example, in the template XML file for the DNS service, you would likely replace the IP address 192.0.2.1 with the variable {$DNS_IP} to control its value from the Python code.
You apply the template by creating a new ncs.template.Template object and calling its apply() method. This method takes the name of the XML template as the first parameter (no trailing .xml), and an object of type ncs.template.Variables as the second parameter. Using the Variables object, you provide values for the variables in the template.
Variables in a template can take a more complex form of an XPath expression, where the parameter for the Template constructor comes into play. This parameter defines the root node (starting point) when evaluating XPath paths. Use the provided service variable, unless you specifically need a different value. It is what the so-called template-based services use as well.
Template-based services are no-code, pure template services that only contain a YANG model and an XML template. Since there is no code to set the variables, they must rely on XPath for the dynamic parts of the template. Such services still have a YANG data model with service parameters, that XPath can access. For example, if you have a parameter leaf defined in the service YANG file by the name dns-server, you can refer to its value with the {/dns-server} code in the XML template.
Likewise, you can use the same XPath in a template of a Python service. Then you don't have to add this parameter to the variables object but can still access its value in the template, saving you a little bit of Python code.
No previous NSO or netsim processes are running. Use the ncs --stop and ncs-netsim stop commands to stop them if necessary.
NSO local install with a fresh runtime directory has been created by the ncs-setup --dest ~/nso-lab-rundir or similar command.
The environment variable NSO_RUNDIR points to this runtime directory, such as set by the export NSO_RUNDIR=~/nso-lab-rundir command. It enables the below commands to work as-is, without additional substitution needed.
The getting-started/developing-with-ncs set of examples contains three simulated routers that you can use for this scenario. The 0-router-network directory holds the data necessary for starting the routers and connecting them to your NSO instance.
First, change the current working directory:
From this directory, you can start a fresh set of routers by running the following make command:
The routers are now running. The required NED package and a CDB initialization file, ncs-cdb/ncs_init.xml, were also added to your NSO instance. The latter contains connection details for the routers and will be automatically loaded on the first NSO start.
In case you're not using a fresh working directory, you may need to use the ncs_load command to load the file manually. Older versions of the system may also be missing the above make target, which you can add to the Makefile yourself:
The DNS configuration service that you are implementing will have three parts: the YANG model, the service code, and the XML template. You will put all of these in a package named dns-config. First, navigate to the packages subdirectory:
Then, run the following command to set up the service package:
In case you are building on top of the previous showcase, the package folder may already exist and will be updated.
You can leave the YANG model as is for this scenario but you need to add some Python code that will apply an XML template during provisioning. In a text or code editor open the main.py file, located inside dns-config/python/dns_config/, and find the definition of the cb_create() function:
You will define one variable for the template, the IP address of the DNS server. To pass its value to the template, you have to create the Variables object and add each variable, along with its value. Replace the body of the cb_create() function with the following:
The template_vars object now contains a value for the DNS_IP template variable, to be used with the apply() method that you are adding next:
Here, the first argument to apply() defines the template to use. In particular, using dns-config-tpl, you are requesting the template from the dns-config-tpl.xml file, which you will be creating shortly.
This is all the Python code that is required. The final, complete cb_create method is as follows:
The most straightforward way to create an XML template is by using the NSO CLI. Return to the running directory and start the NSO:
The --with-package-reload option will make sure that NSO loads any added packages and save a packages reload command on the NSO CLI.
Next, start the NSO CLI:
As you are starting with a new NSO instance, first invoke the sync-from action.
Next, make sure that the ex1 router already has an existing entry for a DNS server in its configuration.
Pipe the command through the display xml-template and save CLI filters to save this configuration as an XML template. According to the Python code, you need to create a template file dns-config-tpl.xml. Use packages/dns-config/templates/dns-config-tpl.xml for the full file path.
At this point, you have created a complete template that will provision the 10.2.3.4 as the DNS server on the ex1 device. The only problem is, that the IP address is not the one you have specified in the Python code. To correct that, open the dns-config-tpl.xml file in a text editor and replace the line that reads <address>10.2.3.4</address> with the following:
The only static part left in the template now is the target device and it's possible to parameterize that, too. The skeleton, created by the ncs-make-package command, already contains a node device in the service YANG file. It is there to allow the service operator to choose the target device to be configured.
One way to use the device service parameter is to read its value in the Python code and then set up the template parameters accordingly. However, there is a simpler way with XPath. In the template, replace the line that reads <name>ex1</name> with the following:
The XPath expression inside the curly braces instructs NSO to get the value for the device name from the service instance's data, namely the node called device. In other words, when configuring a new service instance, you have to add the device parameter, which selects the router for provisioning. The final XML template is then:
Remember to save the template file and return to the NSO CLI. Because you have updated the service code, you have to redeploy it for NSO to pick up the changes:
Alternatively, you could call the packages reload command, which does a full reload of all the packages.
Next, enter the configuration mode:
As you are using the device node in the service model for target router selection, configure a service instance for the ex2 router in the following way:
Finally, using the commit dry-run command, observe the ex2 router being configured with an additional DNS server.
As a bonus for using an XPath expression to a leaf-list in the service template, you can actually select multiple router devices in a single service instance and they will all be configured.
Next Steps
Build your own applications in NSO.
Services provide the foundation for managing the configuration of a network. But this is not the only aspect of network automation. A holistic solution must also consider various verification procedures, one-time actions, monitoring, and so on. This is quite different from managing configuration. NSO helps you implement such automation use cases through a generic application framework.
This section explores the concept of services as more general NSO applications. It gives an overview of the mechanisms for orchestrating network automation tasks that require more than just configuration provisioning.
You have seen two different ways in which you can make a configuration change on a network device. With the first, you make changes directly on the NSO copy of the device configuration. The Device Manager picks up the changes and propagates them to the affected devices.
The purpose of the Device Manager is to manage different devices uniformly. The Device Manager uses the Network Element Drivers (NEDs) to abstract away the different protocols and APIs towards the devices. The NED contains a YANG data model for a supported device. So, each device type requires an appropriate NED package that allows the Device Manager to handle all devices in the same, YANG-model-based way.
The second way to make configuration changes is through services. Here, the Service Manager adds a layer on top of the Device Manager to process the service request and enlists the help of service-aware applications to generate the device changes.
The following figure illustrates the difference between the two approaches.
The Device Manager and the Service Manager are tightly integrated into one transactional engine, using the CDB to store data. Another thing the two managers have in common is packages. Like Device Manager uses NED packages to support specific devices, Service Manager relies on service packages to provide an application-specific mapping for each service type.
However, a network application can consist of more than just a configuration recipe. For example, an integrated service test action can verify the initial provisioning and simplify troubleshooting if issues arise. A simple test might run the ping command to verify connectivity. Or an application could only monitor the network and not produce any configuration at all. That is why NSO actually uses an approach where an application chooses what custom code to execute for specific NSO events.
NSO allows augmenting the base functionality of the system by delegating certain functions to applications. As the communication must happen on demand, NSO implements a system of callbacks. Usually, the application code registers the required callbacks on start-up, and then NSO can invoke each callback as needed. A prime example is a Python service, which registers the cb_create() function as a service callback that NSO uses to construct the actual configuration.
In a Python service skeleton, callback registration happens inside a class Main, found in main.py:
In this code, the register_service() method registers the ServiceCallbacks class to receive callbacks for a service. The first argument defines which service that is. In theory, a single class could even handle service callbacks for multiple services but that is not a common practice.
On the other hand, it is also possible that no code registered a callback for a given service. This is quite often a result of a misspelling or a bug in the code that causes the application code to crash. In these situations, NSO presents an error if you try to use the service:
This error refers to the concept of a service point. Service points are declared in the service YANG model and allow NSO to distinguish ordinary data from services. They instruct NSO to invoke FASTMAP and the service callbacks when a service instance is being provisioned. That means the service skeleton YANG file also contains a service point definition, such as the following:
Service point therefore links the definition in the model with custom code. Some methods in the code will have names starting with cb_, for instance, the cb_create() method, letting you know quickly that they are an implementation of a callback.
NSO implements additional callbacks for each service point, that may be required in some specific circumstances. Most of these callbacks perform work outside of the automatic change tracking, so you need to consider that before using them. The section Service Callbacks offers more details.
As well as services, other extensibility options in NSO also rely on callbacks and callpoints, a generalized version of a service point. Two notable examples are validation callbacks, to implement additional validation logic to that supported by YANG, and custom actions. The section Overview of Extension Points provides a comprehensive list and an overview of when to use each.
In summary, you implement custom behavior in NSO by providing the following three parts:
A YANG model directing NSO to use callbacks, such as a service point for services.
Registration of callbacks, telling NSO to call into your code at a given point.
The implementation of each callback with your custom logic.
This way, an application in NSO can implement all the required functionality for a given use case (configuration management and otherwise) by registering the right callbacks.
The most common way to implement non-configuration automation in NSO is using actions. An action represents a task or an operation that a user of the system can invoke on demand, such as downloading a file, resetting a device, or performing some test.
Like configuration elements, actions must also be defined in the YANG model. Each action is described by the action YANG statement that specifies what are its inputs and outputs, if any. Inputs allow a user of the action to provide additional information to the action invocation, while outputs provide information to the caller. Actions are a form of a Remote Procedure Call (RPC) and have historically evolved from NETCONF RPCs. It's therefore unsurprising that with NSO you implement both in a similar manner.
Let's look at an example action definition:
The first thing to notice in the code is that, just like services use a service point, actions use an actionpoint. It is denoted by the tailf:actionpoint statement and tells NSO to execute a callback registered to this name. As discussed, the callback mechanism allows you to provide custom action implementation.
Correspondingly, your code needs to register a callback to this action point, by calling the register_action(), as demonstrated here:
The MyTestAction class, referenced in the call, is responsible for implementing the actual action logic and should inherit from the ncs.dp.Action base class. The base class will take care of calling the cb_action() class method when users initiate the action. The cb_action() is where you put your own code. The following code shows a trivial implementation of an action, that checks whether its input contains the string “NSO”:
The input and output arguments contain input and output data, respectively, which matches the definition in the action YANG model. The example shows the value of a simple Python in string check that is assigned to an output value.
The name argument has the name of the called action (such as my-test), to help you distinguish which action was called in the case where you would register the same class for multiple actions. Similarly, an action may be defined on a list item and the kp argument contains the full keypath (a tuple) to an instance where it was called.
Finally, the uinfo contains information on the user invoking the action and the trans argument represents a transaction, that you can use to access data other than input. This transaction is read-only, as configuration changes should normally be done through services instead. Still, the action may need some data from NSO, such as an IP address of a device, which you can access by using trans with the ncs.maagic.get_root() function and navigate to the relevant information.
If, for any reason, your action requires a new, read-write transaction, please also read through NSO Concurrency Model to learn about the possible pitfalls.
Further details and the format of the arguments can be found in the NSO Python API reference.
The last thing to note in the above action code definition is the use of the decorator @Action.action. Its purpose is to set up the function arguments correctly, so variables such as input and output behave like other Python Maagic objects. This is no different from services, where decorators are required for the same reason.
No previous NSO or netsim processes are running. Use the ncs --stop and ncs-netsim stop commands to stop them if necessary.
NSO local install with a fresh runtime directory has been created by the ncs-setup --dest ~/nso-lab-rundir or similar command.
The environment variable NSO_RUNDIR points to this runtime directory, such as set by the export NSO_RUNDIR=~/nso-lab-rundir command. It enables the below commands to work as-is, without additional substitution needed.
One of the most common uses of NSO actions is automating network and service tests but they are also a good choice for any other non-configuration task. Being able to quickly answer questions, such as how many network ports are available (unused) or how many devices currently reside in a given subnet, can greatly simplify the network planning process. Coding these computations as actions in NSO makes them accessible on-demand to a wider audience.
For this scenario, you will create a new package for the action, however actions can also be placed into existing packages. A common example is adding a self-test action to a service package.
First, navigate to the packages subdirectory:
Create a package skeleton with the ncs-make-package command and the --action-example option. Name the package count-devices, like so:
This command creates a YANG module file, where you will place a custom action definition. In a text or code editor open the count-devices.yang file, located inside count-devices/src/yang/. This file already contains an example action which you will remove. Find the following line (after module imports):
Delete this line and all the lines following it, to the very end of the file. The file should now resemble the following:
To model an action, you can use the action YANG statement. It is part of the YANG standard from version 1.1 onward, requiring you to also define yang-version 1.1 in the YANG model. So, add the following line at the start of the module, right before namespace statement:
Note that in YANG version 1.0, actions used the NSO-specific tailf:action extension, which you may still find in some YANG models.
Now, go to the end of the file and add a custom-actions container with the count-devices action, using the count-devices-action action point. The input is an IP subnet and the output is the number of devices managed by NSO in this subnet.
Also, add the closing bracket for the module at the end:
Remember to finally save the file, which should now be similar to the following:
The action code is implemented in a dedicated class, that you will put in a separate file. Using an editor, create a new, empty file count_devices_action.py in the count-devices/python/count_devices/ subdirectory.
At the start of the file, import the packages that you will need later on and define the action class with the cb_action() method:
Then initialize the count variable to 0 and construct a reference to the NSO data root, since it is not part of the method arguments:
Using the root variable, you can iterate through the devices managed by NSO and find their (IPv4) address:
If the IP address comes from the specified subnet, increment the count:
Lastly, assign the count to the result:
Your custom Python code is ready; however, you still need to link it to the count-devices action. Open the main.py from the same directory in a text or code editor and delete all the content already in there.
Next, create a class called Main that inherits from the ncs.application.Application base class. Add a single class method setup() that takes no additional arguments.
Inside the setup() method call the register_action() as follows:
This line instructs NSO to use the CountDevicesAction class to handle invocations of the count-devices-action action point. Also, import the CountDevicesAction class from the count_devices_action module.
The complete main.py file should then be similar to the following:
With all of the code ready, you are one step away from testing the new action, but to do that, you will need to add some devices to NSO. So, first, add a couple of simulated routers to the NSO instance:
Before the packages can be loaded, you must compile them:
You can start the NSO now and connect to the CLI:
Finally, invoke the action:
You can use the show devices list command to verify that the result is correct. You can alter the address of any device and see how it affects the result. You can even use a hostname, such as localhost.
NSO supports a number of extension points for custom callbacks:
Each extension point in the list has a corresponding YANG extension that defines to which part of the data model the callbacks apply, as well as the individual name of the call point. The name is required during callback registration and helps distinguish between multiple uses of the extension. Each extension generally specifies multiple callbacks, however, you often need to implement only the main one, e.g. create for services or action for actions.
In addition, NSO supports some specific callbacks from internal systems, such as the transaction or the authorization engine, but these have very narrow use and are in general not recommended.
Services and actions are examples of something that happens directly as a result of a user (or other northbound agent) request. That is, a user takes an active role in starting service instantiation or invoking an action. Contrast this to a change that happens in the network and requires the orchestration system to take some action. In this latter case, the system monitors the notifications that the network generates, such as losing a link, and responds to the new data.
NSO provides out-of-the-box support for the automation of not only notifications but also changes to the operational and configuration data, using the concept of kickers. With kickers, you can watch for a particular change to occur in the system and invoke a custom action that handles the change.
The kicker system is further described in Kicker.
Services, actions, and other features all rely on callback registration. In Python code, the class responsible for registration derives from the ncs.application.Application. This allows NSO to manage the application code as appropriate, such as starting and stopping in response to NSO events. These events include package load or unload and NSO start or stop events.
While the Python package skeleton names the derived class Main, you can choose a different name if you also update the package-meta-data.xml file accordingly. This file defines a component with the name of the Python class to use:
When starting the package, NSO reads the class name from package-meta-data.xml, starts the Python interpreter, and instantiates a class instance. The base Application class takes care of establishing communication with the NSO process and calling the setup and teardown methods. The two methods are a good place to do application-specific initialization and cleanup, along with any callback registrations you require.
The communication between the application process and NSO happens through a dedicated control socket, as described in the section called IPC Ports in Administration. This setup prevents a faulty application from bringing down the whole system along with it and enables NSO to support different application environments.
In fact, NSO can manage applications written in Java or Erlang in addition to those in Python. If you replace the python-class-name element of a component with java-class-name in the package-meta-data.xml file, NSO will instead try to run the specified Java class in the managed Java VM. If you wanted to, you could implement all of the same services and actions in Java, too. For example, see Service Actions to compare Python and Java code.
Regardless of the programming language you use, the high-level approach to automation with NSO does not change, registering and implementing callbacks as part of your network application. Of course, the actual function calls (the API) and other specifics differ for each language. The NSO Python VM, NSO Java VM, and Embedded Erlang Applications cover the details. Even so, the concepts of actions, services, and YANG modeling remain the same.
As you have seen, everything in NSO is ultimately tied to the YANG model, making YANG knowledge such a valuable skill for any NSO developer.
NSO uses socket communication to coordinate work with applications, such as a Python or Java service. In addition to the control socket, NSO uses a number of worker sockets to process individual requests: performing service mapping or executing an action, for example. We collectively call these data provider applications, since the data provider protocol underpins all of them.
The communication with data provider applications is subject to timeouts in order to manage the execution time of requests. These are defined in section /ncs-config/api in ncs.conf:
ncs-config/api/action-timeout
ncs-config/api/query-timeout
ncs-config/api/new-session-timeout
ncs-config/api/connect-timeout
For executing actions invoked by the clients, NSO uses action-timeout to ensures the response from data provider is received within the given time. If the data provider fails to do so within the stipulated timeout, NSO will kill the worker sockets executing the actions and trigger the abort action defined in cb_abort() without restarting the NSO VMs. The following code shows a trivial implementation of an abort action callback:
There are some important points worth noting for action timeout:
An action callback that times out in one user instance will not affect the result of an action callback in another user instance. This is because NSO executes actions using multiple worker sockets, and an action timeout will only terminate the worker socket executing that specific action.
Implementing your own abort action callback in cb_abort allows you to handle actions that are timing out. If cb_abort is not defined, NSO cannot trigger the abort action during a timeout, preventing it from unlocking the action for a user session. Consequently, you must wait for the action callback to finish before attempting it again.
For NSO operational data queries, NSO uses query-timeout to ensure the data provider return operational data within the given time. If the data provider fails to do so within the stipulated timeout, NSO will close its end of the control socket to the data provider. The NSO VMs will detect the socket close and exit.
For connection initiation requests between NSO and data providers, NSO uses connect-timeout to ensure the data provider send the initial message after connecting the socket to NSO within the given time. If the data provider fails to do so within the stipulated timeout, NSO will close its end of the control socket to the data provider. The NSO VMs will detect the socket close and exit.
For requests invoked by NSO, NSO uses new-session-timeout to ensure the data provider respond to the control socket request within the given time. If the data provider fails to do so within the stipulated timeout, NSO will close its end of the control socket to the data provider. The NSO VMs will detect the socket close and exit.
As your NSO application evolves, you will create newer versions of your application package, which will replace the existing one. If the application becomes sufficiently complex, you might even split it across multiple packages.
When you replace a package, NSO must redeploy the application code and potentially replace the package-provided part of the YANG schema. For the latter, NSO can perform the data migration for you, as long as the schema is backward compatible. This process is documented in Automatic Schema Upgrades and Downgrades and is automatic when you request a reload of the package with packages reload or a similar command.
If your schema changes are not backward compatible, you can implement a data migration procedure, which NSO invokes when upgrading the schema. Among other things, this allows you to reuse and migrate the data that is no longer present in the new schema. You can specify the migration procedure as part of the package-meta-data.xml file, using a component of the upgrade type. See The Upgrade Component (Python) and examples.ncs/getting-started/developing-with-ncs/14-upgrade-service example (Java) for details.
Note that changing the schema in any way requires you to recompile the .fxs files in the package, which is typically done by running make in the package's src folder.
However, if the schema does not change, you can request that only the application code and templates be redeployed by using the packages package`` ``my-pkg`` ``redeploy command.
Develop and deploy a nano service using a guided example.
This section shows how to develop and deploy a simple NSO nano service for managing the provisioning of SSH public keys for authentication. For more details on nano services, see Nano Services for Staged Provisioning in Development. The example showcasing development is available under $NCS_DIR/examples.ncs/development-guide/nano-services/netsim-sshkey. In addition, there is a reference from the README in the example's directory to the deployment version of the example.
After installing NSO with the Local Install option, development often begins with either retrieving an existing YANG model representing what the managed network element (a virtual or physical device, such as a router) can do or constructing a new YANG model that at least covers the configuration of interest to an NSO service. To enable NSO service development, the network element's YANG model can be used with NSO's netsim tool that uses ConfD (Configuration Daemon) to simulate the network elements and their management interfaces like NETCONF. Read more about netsim in Network Simulator.
The simple network element YANG model used for this example is available under packages/ne/src/yang/ssh-authkey.yang. The ssh-authkey.yang model implements a list of SSH public keys for identifying a user. The list of keys augments a list of users in the ConfD built-in tailf-aaa.yang module that ConfD uses to authenticate users.
On the network element, a Python application subscribes to ConfD to be notified of configuration changes to the user's public keys and updates the user's authorized_keys file accordingly. See packages/ne/netsim/ssh-authkey.py for details.
The first step is to create an NSO package from the network element YANG model. Since NSO will use NETCONF over SSH to communicate with the device, the package will be a NETCONF NED. The package can be created using the ncs-make-package command or the NETCONF NED builder tool. The ncs-make-package command is typically used when the YANG models used by the network element are available. Hence, the packages/ne package for this example was generated using the ncs-make-package command.
As the ssh-authkey.yang model augments the users list in the ConfD built-in tailf-aaa.yang model, NSO needs a representation of that YANG model too to build the NED. However, the service will only configure the user's public keys, so only a subset of the tailf-aaa.yang model that only includes the user list is sufficient. To compare, see the packages/ne/src/yang/tailf-aaa.yang in the example vs. the network element's version under $NCS_DIR/netsim/confd/src/confd/aaa/tailf-aaa.yang.
Now that the network element package is defined, next up is the service package, beginning with finding out what steps are required for NSO to authenticate with the network element using SSH public key authentication:
First, generate private and public keys using, for example, the ssh-keygen OpenSSH authentication key utility.
Distribute the public keys to the ConfD-enabled network element's list of authorized keys.
Configure NSO to use public key authentication with the network element.
Finally, test the public key authentication by connecting NSO with the network element.
The outline above indicates that the service will benefit from implementing several smaller (nano) steps:
The first step only generates private and public key files with no configuration. Thus, the first step should be implemented by an action before the second step runs, not as part of the second step transaction create() callback code configuring the network elements. The create() callback runs multiple times, for example, for service configuration changes, re-deploy, or commit dry-run. Therefore, generating keys should only happen when creating the service instance.
The third step cannot be executed before the second step is complete, as NSO cannot use the public key for authenticating with the network element before the network element has it in its list of authorized keys.
The fourth step uses the NSO built-in connect() action and should run after the third step finishes.
What configuration input do the above steps need?
The name of the network element that will authenticate a user with an SSH public key.
The name of the local NSO user that maps to the remote network element user the public key authenticates.
The name of the remote network element user.
A passphrase is used for encrypting the private key, guarding its privacy. The passphrase should be encrypted when storing it in the CDB, just like any other password.
The name of the NSO authentication group to configure for public-key authentication with the NSO-managed network element.
A service YANG model that implements the above configuration:
For details on the YANG statements used by the YANG model, such as leaf, container, list, leafref, mandatory, length, pattern, etc., see the IETF RFC 7950 that documents the YANG 1.1 Data Modeling Language. The tailf:xyz are YANG extension statements documented by tailf_yang_extensions(5) in Manual Pages.
The service configuration is implemented in YANG by a key-auth list where the network element and local user names are the list keys. In addition, the list has a distkey-servicepoint service point YANG extension statement to enable the list parameters used by the Python service callbacks that this example implements. Finally, the used service-data and nano-plan-data groupings add the common definitions for a service and the plan data needed when the service is a nano service.
For the nano service YANG part, an NSO YANG nano service behavior tree extension that references a plan outline extension implements the above steps for setting up SSH public key authentication with a network element:
The nano service-behavior-tree for the service point creates a nano service component for each list entry in the key-auth list. The last connection verification step of the nano service, the connected state, uses the NE-NAME variable. The NAME variable concatenates the ne-name and local-user keys from the key-auth list to create a unique nano service component name.
The only step that requires both a create and delete part is the generated state action that generates the SSH keys. If a user deletes a service instance and another network element does not currently use the generated keys, this deletes the keys too. NSO will revert the configuration automatically as part of the FASTMAP algorithm. Hence, the service list instances also need actions for generating and deleting keys.
The actions have no input statements, as the input is the configuration in the service instance list entry.
The generated state uses the ncs:sync statement to ensure that the keys exist before the distributed state runs. Similarly, the distributed state uses the force-commit statement to commit the configuration to the NSO CDB and the network elements before the configured state runs.
See the packages/distkey/src/yang/distkey.yang YANG model for the nano service behavior tree, plan outline, and service configuration implementation.
Next, handling the key generation, distributing keys to the network element, and configuring NSO to authenticate using the keys with the network element requires some code, here written in Python, implemented by the packages/distkey/python/distkey/distkey-app.py script application.
The Python script application defines a Python DistKeyApp class specified in the packages/distkey/package-meta-data.xml file that NSO starts in a Python thread. This Python class inherits ncs.application.Application and implements the setup() and teardown() methods. The setup() method registers the nano service create() callbacks and the action handlers for generating and deleting the key files. Using the nano service state to separate the two nano service create() callbacks for the distribution and NSO configuration of keys, only one Python class, the DistKeyServiceCallbacks class, is needed to implement them.
The action for generating keys calls the OpenSSH ssh-keygen command to generate the private and public key files. Calling ssh-keygen is kept out of the service create() callback to avoid the key generation running multiple times, for example, for service changes, re-deploy, or dry-run commits. Also, NSO encrypts the passphrase used when generating the keys for added security, see the YANG model, so the Python code decrypts it before using it with the ssh-keygen command.
The DeleteActionHandler action deletes the key files if no more network elements use the user's keys:
The Python class for the nano service create() callbacks handles both the distribution and NSO configuration of the keys. The dk:distributed state create() callback code adds the public key data to the network element's list of authorized keys. For the create() call for the dk:configured state, a template is used to configure NSO to use public key authentication with the network element. The template can be called directly from the nano service, but in this case, it needs to be called from the Python code to input the current working directory to the template:
The template to configure NSO to use public key authentication with the network element is available under packages/distkey/templates/distkey-configured.xml:
The example uses three scripts to showcase the nano service:
A shell script, showcase.sh, which uses the ncs_cli program to run CLI commands via the NSO IPC port.
A Python script, showcase-rc.sh, which uses the requests package for RESTCONF edit operations and receiving event notifications.
A Python script that uses NSO MAAPI, showcase-maapi.sh, via the NSO IPC port.
The ncs_cli program identifies itself with NSO as the admin user without authentication, and the RESTCONF client uses plain HTTP and basic user password authentication. All three scripts demonstrate the service by generating keys, distributing the public key, and configuring NSO for public key authentication with the network elements. To run the example, see the instructions in the README file of the example.
See the README in the netsim-sshkey example's directory for a reference to an NSO system installation in a container deployment variant.
The deployment variant differs from the development example by:
Installing NSO with a system installation for deployment instead of a local installation suitable for development
Addressing NSO security by running NSO as the admin user and authenticating using a public key and token.
Rotating NSO logs to avoid running out of disk space
Installing the distkey service package and ne NED package at startup
The NSO CLI showcase script uses SSH with public key authentication instead of the ncs_cli program over unsecured IPC
There is no Python MAAPI showcase script. Use RESTCONF over HTTPS with Python instead of Python MAAPI over unsecured IPC.
Having NSO and the network elements (simulated by the ConfD subscriber application) run in separate containers
NSO is either pre-installed in the NSO production container image or installed in a generic Linux container.
The deployment example sets up a minimal production installation where the NSO process runs as the admin OS user, relying on PAM authentication for the admin and oper NSO users. The admin user is authenticated over SSH using a public key for CLI and NETCONF access and over RESTCONF HTTPS using a token. The read-only oper user uses password authentication. The oper user can access the NSO WebUI over HTTPS port 443 from the container host.
A modified version of the NSO configuration file ncs.conf from the example running with a local install NSO is located in the $NCS_CONFIG_DIR (/etc/ncs) directory. The packages, ncs-cdb, state, and scripts directories are now under the $NCS_RUN_DIR (/var/opt/ncs) directory. The log directory is now the $NCS_LOG_DIR (/var/log/ncs) directory. Finally, the $NCS_DIR variable points to /opt/ncs/current.
Two scripts showcase the nano service:
A shell script that runs NSO CLI commands over SSH.
A Python script that uses the requests package to perform edit operations and receive event notifications.
As with the development version, both scripts will demo the service by generating keys, distributing the public key, and configuring NSO for public key authentication with the network elements.
To run the example and for more details, see the instructions in the README file of the deployment example.
| Type | Supported In | YANG Extension | Description |
|---|---|---|---|
Service
Python, Java, Erlang
ncs:servicepoint
Transforms a list or container into a model for service instances. When the configuration of a service instance changes, NSO invokes Service Manager and FASTMAP, which may call service create and similar callbacks. See Developing a Simple Service for an introduction.
Action
Python, Java, Erlang
tailf:actionpoint
Defines callbacks when an action or RPC is invoked. See Actions for an introduction.
Validation
Python, Java, Erlang
tailf:validate
Defines callbacks for additional validation of data when the provided YANG functionality, such as must and unique statements are insufficient. See the respective API documentation for examples; the section ValidationPoint Handler (Python), the section Validation Callbacks (Java), and Embedded Erlang applications (Erlang).
Data Provider
Java, Python (low-level API with experimental high-level API), Erlang
tailf:callpoint
Defines callbacks for transparently accessing external data (data not stored in the CDB) or callbacks for special processing of data nodes (transforms, set, and transaction hooks). Requires careful implementation and understanding of transaction intricacies. Rarely used in NSO.
