This section describes some recipes, tools, and other resources that you may find useful throughout development. The topics are tailored to novice users and focus on making development with NSO a more enjoyable experience.

Development NSO Instance

Many developers prefer their own, dedicated NSO instance to avoid their work clashing with other team members. You can use either a local or remote Linux machine (such as a VM), or a macOS computer for this purpose.

The advantage of running local Linux with a GUI or macOS is that it is easier to set up the Integrated Development Environment (IDE) and other tools when they run on the same system as NSO. However, many IDEs today also allow working remotely, such as through the SSH protocol, making the choice of local versus remote less of a concern.

For development, using the so-called Local Install of NSO has some distinct advantages:

• It does not require elevated privileges to install or run.

• It keeps all NSO files in the same place (user-defined).

• It allows you to quickly switch between projects and NSO versions.

If you work with multiple projects in parallel, local install also allows you to take advantage of Python virtual environments to separate Python packages per project; simply start the NSO instance in an environment you have activated.

The main downside of using a local install is that it differs slightly from a system (production) install, such as in the filesystem paths used and the out-of-the-box configuration.

See Local Install for installation instructions.

Examples and Showcases

There are a number of examples and showcases in this guide. We encourage you to follow them through. They are also a great reference if you are experimenting with a new feature and have trouble getting it to work; you can inspect and compare with the implementation in the example.

To run the examples, you will need access to an NSO instance. A development instance described in this chapter is the perfect option for running locally. See Running NSO Examples.

IDE

Modern IDEs offer many features on top of advanced file editing support, such as code highlighting, syntax checks, and integrated debugging. While the initial setup takes some effort, the benefits of using an IDE are immense.

Visual Studio Code (VS Code) is a freely available and extensible IDE. You can add support for Java, Python, and YANG languages, as well as remote access through SSH via VS Code extensions. Consider installing the following extensions:

• Python by Microsoft: Adds Python support.

• Language Support for Java(TM) by Red Hat: Adds Java support.

• NSO Developer Studio by Cisco: Adds NSO-specific features as described in NSO Developer Studio.

• Remote - SSH by Microsoft: Adds support for remote development.

The Remote - SSH extension is especially useful when you must work with a system through an SSH session. Once you connect to the remote host by clicking the >< button (typically found in the bottom-left corner of the VS Code window), you can open and edit remote files with ease. If you also want language support (syntax highlighting and alike), you may need to install VS Code extensions remotely. That is, install the extensions after you have connected to the remote host, otherwise the extension installation screen might not show the option for installation on the connected host.

You will also benefit greatly from setting up SSH certificate authentication if you are using an SSH session for your work.

Automating Instance Setup

Once you get familiar with NSO development and gain some experience, a single NSO instance is likely to be insufficient; either because you need instances for unit testing, because you need one-off (throwaway) instances for an experiment, or something else entirely.

NSO includes tooling to help you quickly set up new local instances when such a need arises.

The following recipe relies on the ncs-setup command, which is available in the local install variant and requires a correctly set up shell environment (e.g. running source ncsrc). See Local Install for details.

A new instance typically needs a few things to be useful:

• Packages

• Initial data

• Devices to manage

In its simplest form, the ncs-setup invocation requires only a destination directory. However, you can specify additional packages to use with the --package option. Use the option to add as many packages as you need.

Running ncs-setup creates the required filesystem structure for an NSO instance. If you wish to include initial configuration data, put the XML-encoded data in the ncs-cdb subdirectory and NSO will load it at the first start, as described in Initialization Files.

NSO also needs to know about the managed devices. In case you are using ncs-netsim simulated devices (described in Network Simulator), you can use the --netsim-dir option with ncs-setup to add them directly. Otherwise, you may need to create some initial XML files with the relevant device configuration data — much like how you would add a device to NSO manually.

Most of the time, you must also invoke a sync with the device so that it performs correctly with NSO. If you wish to push some initial configuration to the device, you may add the configuration in the form of initial XML data and perform a sync-to. Alternatively, you can simply do a sync-from. You can use the ncs_cmd command for this purpose.

Combining all of this together, consider the following example:

1. Start by creating a new directory to hold the files:

   Copy

   $ mkdir nso-throwaway
   $ cd nso-throwaway

2. Create and start a few simulated devices with ncs-netsim, using ./netsim as directory:

   Copy

   $ ncs-netsim ncs-netsim create-network $NCS_DIR/packages/neds/cisco-ios-cli-3.8 3 c
   DEVICE c0 CREATED
   DEVICE c1 CREATED
   DEVICE c2 CREATED
   $ ncs-netsim start

3. Next, create the running directory with the NED package for the simulated devices and one more package. Also, add configuration data to NSO on how to connect to these simulated devices.

   Copy

       $ ncs-setup --dest ncs-run --netsim-dir ./netsim \
           --package $NCS_DIR/packages/neds/cisco-ios-cli-3.8 \
           --package $NCS_DIR/packages/neds/cisco-iosxr-cli-3.0

4. Now you can add custom initial data as XML files to ncs-run/ncs-cdb/. Usually, you would use existing files but you can also create them on-the-fly.

   Copy

   $ cat >ncs-run/ncs-cdb/my_init.xml <<'EOF'
   <config xmlns="http://tail-f.com/ns/config/1.0">
     <session xmlns="http://tail-f.com/ns/aaa/1.1">
       <idle-timeout>0</idle-timeout>
     </session>
   </config>
   EOF

5. At this point, you are ready to start NSO:

   Copy

   $ cd ncs-run
   $ ncs

6. Finally, request an initial sync-from:

   Copy

   $ ncs_cmd -u admin -c 'maction /devices/sync-from'
   sync-result begin
     device c0
     result true
   sync-result end
   sync-result begin
     device c1
     result true
   sync-result end
   sync-result begin
     device c2
     result true
   sync-result end

7. The instance is now ready for work. Once you are finished, you can stop it with ncs --stop. Remember to also stop the simulated devices with ncs-netsim stop if you no longer need them. Then, delete the containing folder (nso-throwaway) to remove all the leftover files and data.

Many services in NSO rely on NEDs to perform network provisioning. These services map service-specific configuration to the device data models, provided by the NEDs. As the NED packages can be upgraded independently, they can introduce changes in the device YANG models that cause issues for the services using them.

NSO provides tools to migrate between backward incompatible NED versions. The tools are designed to give you a structured analysis of which paths will change between two NED versions and visibility into the scope of the potential impact that a change in the NED will drive in the service code.

The tools allow for a usage-based analysis of which parts of the NED data model (and instance tree) a particular service has written to. This will give you an (at least opportunistic) sense of which paths must change in the service code.

These features aim to lower the barrier of upgrading NEDs and significantly reduce the amount of uncertainty and side effects that NED upgrades were historically associated with.

The migrate Action

By using the /ncs:devices/device/migrate action, you can change the NED major/minor version of a device. The action migrates all configuration and service meta-data. The action can also be executed in parallel on a device group or on all devices matching a NED identity. The procedure for migrating devices is further described in .

Additionally, the example examples.ncs/getting-started/developing-with-ncs/26-ned-migration in the NSO examples collection illustrates how to migrate devices between different NED versions using the migrate action.

What makes it particularly useful to a service developer is that the action reports what paths have been modified and the service instances affected by those changes. This information can then be used to prepare the service code to handle the new NED version. If the verbose option is used, all service instances are reported instead of just the service points. If the dry-run option is used, the action simply reports what it would do. This gives you the chance to analyze before any actual change is performed.

As using Java for service development may be somewhat more involved than Python, this section provides further examples and additional tips for setting up the development environment for Java.

The two examples, a simple VLAN service and a Layer 3 MPLS VPN service are more elaborate but show the same techniques as .

Creating a Simple VLAN Service

In this example, you will create a simple VLAN service in Java. In order to illustrate the concepts, the device configuration is simplified from a networking perspective and only uses one single device type (Cisco IOS).

Overview of Steps

We will first look at the following preparatory steps:

1. Prepare a simulated environment of Cisco IOS devices: in this example, we start from scratch in order to illustrate the complete development process. We will not reuse any existing NSO examples.

2. Generate a template service skeleton package: use NSO tools to generate a Java-based service skeleton package.

3. Write and test the VLAN Service Model.

4. Analyze the VLAN service mapping to IOS configuration.

These steps are no different from defining services using templates. Next is to start playing with the Java Environment:

1. Configuring the start and stop of the Java VM.

2. First look at the Service Java Code: introduction to service mapping in Java.

3. Developing by tailing log files.

4. Developing using Eclipse.

Setting Up the Environment

We will start by setting up a run-time environment that includes simulated Cisco IOS devices and configuration data for NSO. Make sure you have sourced the ncsrc file.

1. Create a new directory that will contain the files for this example, such as:

1. Now, let's create a simulated environment with 3 IOS devices and an NSO that is ready to run with this simulated network:

1. Start the simulator and NSO:

1. Use the Cisco CLI towards one of the devices:

1. Use the NSO CLI to get the configuration:

1. Finally, set VLAN information manually on a device to prepare for the mapping later.

Creating a Service Package

1. In the run-time directory, you created:

Note the packages directory, cd to it:

Currently, there is only one package, the Cisco IOS NED.

1. We will now create a new package that will contain the VLAN service.

This creates a package with the following structure:

During the rest of this section, we will work with the vlan/src/yang/vlan.yang and vlan/src/java/src/com/example/vlan/vlanRFS.java files.

The Service Model

So, if a user wants to create a new VLAN in the network what should the parameters be? Edit the vlan/src/yang/vlan.yang according to below:

This simple VLAN service model says:

1. We give a VLAN a name, for example net-1.

2. The VLAN has an id from 1 to 4096.

3. The VLAN is attached to a list of devices and interfaces. In order to make this example as simple as possible the interface name is just a string. A more correct and useful example would specify this is a reference to an interface to the device, but for now it is better to keep the example simple.

The VLAN service list is augmented into the services tree in NSO. This specifies the path to reach VLANs in the CLI, REST, etc. There are no requirements on where the service shall be added into NCS, if you want VLANs to be at the top level, simply remove the augments statement.

Make sure you keep the lines generated by the ncs-make-package:

The two lines tell NSO that this is a service. The first line expands to a YANG structure that is shared amongst all services. The second line connects the service to the Java callback.

To build this service model, cd to packages/vlan/src and type make (assumes that you have the prerequisite make build system installed).

We can now test the service model by requesting NSO to reload all packages:

You can also stop and start NSO, but then you have to pass the option --with-package-reload when starting NSO. This is important, NSO does not by default take any changes in packages into account when restarting. When packages are reloaded the state/packages-in-use is updated.

Now, create a VLAN service, (nothing will happen since we have not defined any mapping).

Now, let us move on and connect that to some device configuration using Java mapping. Note well that Java mapping is not needed, templates are more straightforward and recommended but we use this as a "Hello World" introduction to Java service programming in NSO. Also at the end, we will show how to combine Java and templates. Templates are used to define a vendor-independent way of mapping service attributes to device configuration and Java is used as a thin layer before the templates to do logic, call-outs to external systems, etc.

Managing the NSO Java VM

The default configuration of the Java VM is:

By default, NCS will start the Java VM by invoking the command $NCS_DIR/bin/ncs-start-java-vm. That script will invoke

The class NcsJVMLauncher contains the main() method. The started Java VM will automatically retrieve and deploy all Java code for the packages defined in the load path of the ncs.conf file. No other specification than the package-meta-data.xml for each package is needed.

The verbosity of Java error messages can be controlled by:

A First Look at Java Development

The service model and the corresponding Java callback are bound by the servicepoint name. Look at the service model in packages/vlan/src/yang:

The corresponding generated Java skeleton, (one print 'Hello World!' statement added):

Modify the generated code to include the print "Hello World!" statement in the same way. Re-build the package:

Whenever a package has changed, we need to tell NSO to reload the package. There are three ways:

1. Just reload the implementation of a specific package, will not load any model changes: admin@ncs# packages package vlan redeploy.

2. Reload all packages including any model changes: admin@ncs# packages reload.

3. Restart NSO with reload option: $ncs --with-package-reload.

When that is done we can create a service (or modify an existing one) and the callback will be triggered:

Now, have a look at the logs/ncs-java-vm.log:

Tailing the ncs-java-vm.log is one way of developing. You can also start and stop the Java VM explicitly and see the trace in the shell. To do this, tell NSO not to start the VM by adding the following snippet to ncs.conf:

Then, after restarting NSO or reloading the configuration, from the shell prompt:

So modifying or creating a VLAN service will now have the "Hello World!" string show up in the shell. You can modify the package, then reload/redeploy, and see the output.

Using Eclipse

To use a GUI-based IDE Eclipse, first generate an environment for Eclipse:

This will generate two files, .classpath and .project. If we add this directory to Eclipse as a File -> New -> Java Project, uncheck the Use default location and enter the directory where the .classpath and .project have been generated.

We are immediately ready to run this code in Eclipse.

All we need to do is choose the main() routine in the NcsJVMLauncher class. The Eclipse debugger works now as usual, and we can, at will, start and stop the Java code.

Edit the file and enter the following XML entry just after the Webui entry:

Now, restart ncs, and from now on start it as:

You can verify that the Java VM is not running by checking the package status:

Create a new project and start the launcher main in Eclipse:

You can start and stop the Java VM from Eclipse. Note well that this is not needed since the change cycle is: modify the Java code, make in the src directory, and then reload the package. All while NSO and the JVM are running.

Change the VLAN service and see the console output in Eclipse:

Another option is to have Eclipse connect to the running VM. Start the VM manually with the -d option.

Then you can set up Eclipse to connect to the NSO Java VM:

In order for Eclipse to show the NSO code when debugging, add the NSO Source Jars (add external Jar in Eclipse):

Navigate to the service create for the VLAN service and add a breakpoint:

Commit a change of a VLAN service instance and Eclipse will stop at the breakpoint:

Writing the Service Code

Fetching the Service Attributes

So the problem at hand is that we have service parameters and a resulting device configuration. Previously, we showed how to do that with templates. The same principles apply in Java. The service model and the device models are YANG models in NSO irrespective of the underlying protocol. The Java mapping code transforms the service attributes to the corresponding configuration leafs in the device model.

The NAVU API lets the Java programmer navigate the service model and the device models as a DOM tree. Have a look at the create signature:

Two NAVU nodes are passed: the actual service serviceinstance and the NSO root ncsRoot.

We can have a first look at NAVU by analyzing the first try statement:

NAVU is a lazy evaluated DOM tree that represents the instantiated YANG model. So knowing the NSO model: devices/device, (container/list) corresponds to the list of capabilities for a device, this can be retrieved by ncsRoot.container("devices").list("device").

The service node can be used to fetch the values of the VLAN service instance:

• vlan/name

• vlan/vlan-id

• vlan/device-if/device and vlan/device-if/interface

The first snippet that iterates the service model and prints to the console looks like below:

The com.tailf.conf package contains Java Classes representing the YANG types like ConfUInt32.

Try it out in the following sequence:

1. Rebuild the Java Code: In packages/vlan/src type make.

2. Reload the Package: In the NSO Cisco CLI, do admin@ncs# packages package vlan redeploy.

3. Create or Modify a vlan Service: In NSO CLI, do admin@ncs(config)# services vlan net-0 vlan-id 844 device-if c0 interface 1/0, and commit.

Mapping Service Attributes to Device Configuration

Remember the service attribute is passed as a parameter to the create method. As a starting point, look at the first three lines:

1. To reach a specific leaf in the model use the NAVU leaf method with the name of the leaf as a parameter. This leaf then has various methods like getting the value as a string.

2. service.leaf("vlan-id") and service.leaf(vlan._vlan_id_) are two ways of referring to the VLAN-id leaf of the service. The latter alternative uses symbols generated by the compilation steps. If this alternative is used, you get the benefit of compilation time checking. From this leaf you can get the value according to the type in the YANG model ConfUInt32 in this case.

3. Line 3 shows an example of casting between types. In this case, we prepare the VLAN ID as a 16 unsigned int for later use.

The next step is to iterate over the devices and interfaces. The NAVU elements() returns the elements of a NAVU list.

In order to write the mapping code, make sure you have an understanding of the device model. One good way of doing that is to create a corresponding configuration on one device and then display that with the pipe target display xpath. Below is a CLI output that shows the model paths for FastEthernet 1/0:

Another useful tool is to render a tree view of the model:

This can then be opened in a Web browser and model paths are shown to the right:

Now, we replace the print statements with setting real configuration on the devices.

Let us walk through the above code line by line. The device-name is a leafref. The deref method returns the object that the leafref refers to. The getParent() might surprise the reader. Look at the path for a leafref: /device/name/config/ios:interface/name. The name leafref is the key that identifies a specific interface. The deref returns that key, while we want to have a reference to the interface, (/device/name/config/ios:interface), that is the reason for the getParent().

The next line sets the VLAN list on the device. Note well that this follows the paths displayed earlier using the NSO CLI. The sharedCreate() is important, it creates device configuration based on this service, and it says that other services might also create the same value, "shared". Shared create maintains reference counters for the created configuration in order for the service deletion to delete the configuration only when the last service is deleted. Finally, the interface name is used as a key to see if the interface exists, "containsNode()".

The last step is to update the VLAN list for each interface. The code below adds an element to the VLAN leaf-list.

Note that the code uses the sharedCreate() functions instead of create(), as the shared variants are preferred and a best practice.

The above create method is all that is needed for create, read, update, and delete. NSO will automatically handle any changes, like changing the VLAN ID, adding an interface to the VLAN service, and deleting the service. This is handled by the FASTMAP engine, it renders any change based on the single definition of the create method.

Simple VLAN Service with Templates

Overview

The mapping strategy using only Java is illustrated in the following figure.

This strategy has some drawbacks:

• Managing different device vendors. If we would introduce more vendors in the network this would need to be handled by the Java code. Of course, this can be factored into separate classes in order to keep the general logic clean and just pass the device details to specific vendor classes, but this gets complex and will always require Java programmers to introduce new device types.

• No clear separation of concerns, domain expertise. The general business logic for a service is one thing, detailed configuration knowledge of device types is something else. The latter requires network engineers and the first category is normally separated into a separate team that deals with OSS integration.

Java and templates can be combined:

In this model, the Java layer focuses on required logic, but it never touches concrete device models from various vendors. The vendor-specific details are abstracted away using feature templates. The templates take variables as input from the service logic, and the templates in turn transform these into concrete device configuration. The introduction of a new device type does not affect the Java mapping.

This approach has several benefits:

• The service logic can be developed independently of device types.

• New device types can be introduced at runtime without affecting service logic.

• Separation of concerns: network engineers are comfortable with templates, they look like a configuration snippet. They have expertise in how configuration is applied to real devices. People defining the service logic often are more programmers, they need to interface with other systems, etc, this suites a Java layer.

Note that the logic layer does not understand the device types, the templates will dynamically apply the correct leg of the template depending on which device is touched.

The VLAN Feature Template

From an abstraction point of view, we want a template that takes the following variables:

• VLAN ID

• Device and interface

So the mapping logic can just pass these variables to the feature template and it will apply it to a multi-vendor network.

Create a template as described before.

• Create a concrete configuration on a device, or several devices of different type

• Request NSO to display that as XML

• Replace values with variables

This results in a feature template like below:

This template only maps to Cisco IOS devices (the xmlns="urn:ios" namespace), but you can add "legs" for other device types at any point in time and reload the package.

The VLAN Java Logic

The Java mapping logic for applying the template is shown below:

Note that the Java code has no clue about the underlying device type, it just passes the feature variables to the template. At run-time, you can update the template with mapping to other device types. The Java code stays untouched, if you modify an existing VLAN service instance to refer to the new device type the commit will generate the corresponding configuration for that device.

The smart reader will complain, "Why do we have the Java layer at all?", this could have been done as a pure template solution. That is true, but now this simple Java layer gives room for arbitrary complex service logic before applying the template.

Steps to Build a Java and Template Solution

The steps to build the solution described in this section are:

1. Create a run-time directory: $ mkdir ~/service-template; cd ~/service-template.

2. Generate a netsim environment: $ ncs-netsim create-network $NCS_DIR/packages/neds/cisco-ios 3 c.

3. Generate the NSO runtime environment: $ ncs-setup --netsim-dir ./netsim --dest ./.

4. Create the VLAN package in the packages directory: $ cd packages; ncs-make-package --service-skeleton java vlan.

5. Create a template directory in the VLAN package: $ cd vlan; mkdir templates.

6. Save the above-described template in packages/vlan/templates.

7. Create the YANG service model according to the above: packages/vlan/src/yang/vlan.yang.

8. Update the Java code according to the above: packages/vlan/src/java/src/com/example/vlan/vlanRFS.java.

9. Build the package: in packages/vlan/src do make.

10. Start NSO.

Layer 3 MPLS VPN Service

This service shows a more elaborate service mapping. It is based on the examples.ncs/service-provider/mpls-vpn example.

MPLS VPNs are a type of Virtual Private Network (VPN) that achieves segmentation of network traffic using Multiprotocol Label Switching (MPLS), often found in Service Provider (SP) networks. The Layer 3 variant uses BGP to connect and distribute routes between sites of the VPN.

The figure below illustrates an example configuration for one leg of the VPN. Configuration items in bold are variables that are generated from the service inputs.

Auxiliary Service Data

Sometimes the input parameters are enough to generate the corresponding device configurations. But in many cases, this is not enough. The service mapping logic may need to reach out to other data in order to generate the device configuration. This is common in the following scenarios:

• Policies: it might make sense to define policies that can be shared between service instances. The policies, for example, QoS, have data models of their own (not service models) and the mapping code reads from that.

• Topology Information: the service mapping might need to know connected devices, like which PE the CE is connected to.

• Resources like VLAN IDs, and IP Addresses: these might not be given as input parameters. This can be modeled separately in NSO or fetched from an external system.

It is important to design the service model to consider the above examples: what is input? what is available from other sources? This example illustrates how to define QoS policies "on the side". A reference to an existing QoS policy is passed as input. This is a much better principle than giving all QoS parameters to every service instance. Note well that if you modify the QoS definitions that services are referring to, this will not change the existing services. In order to have the service to read the changed policies you need to perform a re-deploy on the service.

This example also uses a list that maps every CE to a PE. This list needs to be populated before any service is created. The service model only has the CE as input parameter, and the service mapping code performs a lookup in this list to get the PE. If the underlying topology changes a service re-deploy will adopt the service to the changed CE-PE links. See more on topology below.

NSO has a package to manage resources like VLAN and IP addresses as a pool within NSO. In this way the resources are managed within the transaction. The mapping code could also reach out externally to get resources. Nano services are recommended for this.

Topology

Using topology information in the instantiation of an NSO service is a common approach, but also an area with many misconceptions. Just like a service in NSO takes a black-box view of the configuration needed for that service in the network NSO treats topologies in the same way. It is of course common that you need to reference topology information in the service but it is highly desirable to have a decoupled and self-sufficient service that only uses the part of the topology that is interesting/needed for the specific service should be used.

Other parts of the topology could either be handled by other services or just let the network state sort it out - it does not necessarily relate to the configuration of the network. A routing protocol will for example handle the IP path through the network.

It is highly desirable to not introduce unneeded dependencies towards network topologies in your service.

To illustrate this, let's look at a Layer 3 MPLS VPN service. A logical overview of an MPLS VPN with three endpoints could look something like this. CE routers connecting to PE routers, that are connected to an MPLS core network. In the MPLS core network, there are a number of P routers.

In the service model, you only want to configure the CE devices to use as endpoints. In this case, topology information could be used to sort out what PE router each CE router is connected to. However, what type of topology do you need? Lets look at a more detailed picture of what the L1 and L2 topology could look like for one side of the picture above.

In pretty much all networks there is an access network between the CE and PE router. In the picture above the CE routers are connected to local Ethernet switches connected to a local Ethernet access network, connected through optical equipment. The local Ethernet access network is connected to a regional Ethernet access network, connected to the PE router. Most likely the physical connections between the devices in this picture have been simplified, in the real world redundant cabling would be used. The example above is of course only one example of how an access network could look like and it is very likely that a service provider have different access technologies. For example Ethernet, ATM, or a DSL-based access network.

Depending on how you design the L3VPN service, the physical cabling or the exact traffic path taken in the layer 2 Ethernet access network might not be that interesting, just like we don't make any assumptions or care about how traffic is transported over the MPLS core network. In both these cases we trust the underlying protocols handling state in the network, spanning tree in the Ethernet access network, and routing protocols like BGP in the MPLS cloud. Instead in this case, it could make more sense to have a separate NSO service for the access network, both so it can be reused for both for example L3VPNs and L2VPN but also to not tightly couple to the access network with the L3VPN service since it can be different (Ethernet or ATM etc.).

Looking at the topology again from the L3VPN service perspective, if services assume that the access network is already provisioned or taken care of by another service, it could look like this.

The information needed to sort out what PE router a CE router is connected to as well as configuring both CE and PE routers is:

• Interface on the CE router that is connected to the PE router, and IP address of that interface.

• Interface on the PE router that is connected to the CE router, and IP address to the interface.

Creating a Multi-Vendor Service

This section describes the creation of an MPLS L3VPN service in a multi-vendor environment by applying the concepts described above. The example discussed can be found in examples.ncs/service-provider/mpls-vpn. The example network consists of Cisco ASR 9k and Juniper core routers (P and PE) and Cisco IOS-based CE routers.

The goal of the NSO service is to set up an MPLS Layer3 VPN on a number of CE router endpoints using BGP as the CE-PE routing protocol. Connectivity between the CE and PE routers is done through a Layer2 Ethernet access network, which is out of the scope of this service. In a real-world scenario, the access network could for example be handled by another service.

In the example network, we can also assume that the MPLS core network already exists and is configured.

YANG Service Model Design

When designing service YANG models there are a number of things to take into consideration. The process usually involves the following steps:

1. Identify the resulting device configurations for a deployed service instance.

2. Identify what parameters from the device configurations are common and should be put in the service model.

3. Ensure that the scope of the service and the structure of the model work with the NSO architecture and service mapping concepts. For example, avoid unnecessary complexities in the code to work with the service parameters.

4. Ensure that the model is structured in a way so that integration with other systems north of NSO works well. For example, ensure that the parameters in the service model map to the needed parameters from an ordering system.

Steps 1 and 2: Device Configurations and Identifying Parameters:

Deploying an MPLS VPN in the network results in the following basic CE and PE configurations. The snippets below only include the Cisco IOS and Cisco IOS-XR configurations. In a real process, all applicable device vendor configurations should be analyzed.

The device configuration parameters that need to be uniquely configured for each VPN have been marked in bold.

Steps 3 and 4: Model Structure and Integration with other Systems:

When configuring a new MPLS l3vpn in the network we will have to configure all CE routers that should be interconnected by the VPN, as well as the PE routers they connect to.

However, when creating a new l3vpn service instance in NSO it would be ideal if only the endpoints (CE routers) are needed as parameters to avoid having knowledge about PE routers in a northbound order management system. This means a way to use topology information is needed to derive or compute what PE router a CE router is connected to. This makes the input parameters for a new service instance very simple. It also makes the entire service very flexible, since we can move CE and PE routers around, without modifying the service configuration.

Resulting YANG Service Model:

The snipped above contains the l3vpn service model. The structure of the model is very simple. Every VPN has a name, an as-number, and a list of all the endpoints in the VPN. Each endpoint has:

• A unique ID.

• A reference to a device (a CE router in our case).

• A pointer to the LAN local interface on the CE router. This is kept as a string since we want this to work in a multi-vendor environment.

• LAN private IP network.

• Bandwidth on the VPN connection.

To be able to derive the CE to PE connections we use a very simple topology model. Notice that this YANG snippet does not contain any service point, which means that this is not a service model but rather just a YANG schema letting us store information in CDB.

The model basically contains a list of connections, where each connection points out the device, interface, and IP address in each of the connections.

Defining the Mapping

Since we need to look up which PE routers to configure using the topology model in the mapping logic it is not possible to use a declarative configuration template-based mapping. Using Java and configuration templates together is the right approach.

The Java logic lets you set a list of parameters that can be consumed by the configuration templates. One huge benefit of this approach is that all the parameters set in the Java code are completely vendor-agnostic. When writing the code, there is no need for knowledge of what kind of devices or vendors exist in the network, thus creating an abstraction of vendor-specific configuration. This also means that in to create the configuration template there is no need to have knowledge of the service logic in the Java code. The configuration template can instead be created and maintained by subject matter experts, the network engineers.

With this service mapping approach, it makes sense to modularize the service mapping by creating configuration templates on a per-feature level, creating an abstraction for a feature in the network. In this example means, we will create the following templates:

• CE router

• PE router

This is both to make services easier to maintain and create but also to create components that are reusable from different services. This can of course be even more detailed with templates with for example BGP or interface configuration if needed.

Since the configuration templates are decoupled from the service logic it is also possible to create and add additional templates in a running NSO system. You can for example add a CE router from a new vendor to the layer3 VPN service by only creating a new configuration template, using the set of parameters from the service logic, to a running NSO system without changing anything in the other logical layers.

The Java Code

The Java code part for the service mapping is very simple and follows the following pseudo code steps:

This section will go through relevant parts of Java outlined by the pseudo-code above. The code starts with defining the configuration templates and reading the list of endpoints configured and the topology. The Navu API is used for navigating the data models.

The next step is iterating over the VPN endpoints configured in the service, finding out connected PE router using small helper methods navigating the configured topology.

The parameter dictionary is created from the TemplateVariables class and is populated with appropriate parameters.

The last step after all parameters have been set is applying the templates for the CE and PE routers for this VPN endpoint.

Configuration Templates

The configuration templates are XML templates based on the structure of device YANG models. There is a very easy way to create the configuration templates for the service mapping if NSO is connected to a device with the appropriate configuration on it, using the following steps.

1. Configure the device with the appropriate configuration.

2. Add the device to NSO

3. Sync the configuration to NSO.

4. Display the device configuration in XML format.

5. Save the XML output to a configuration template file and replace configured values with parameters

The commands in NSO give the following output. To make the example simpler, only the BGP part of the configuration is used:

The final configuration template with the replaced parameters marked in bold is shown below. If the parameter starts with a $-sign, it's taken from the Java parameter dictionary; otherwise, it is a direct xpath reference to the value from the service instance.

NSO Developer Studio provides an integrated framework for developing NSO services using Visual Studio (VS) Code extensions. The extensions come with a core feature set to help you create services and connect to running CDB instances from within the VS Code environment. The following extensions are available as part of the NSO Developer Studio:

• NSO Developer Studio - Developer: Used for creating NSO services. Also referred to as NSO Developer extension in this guide.

• NSO Developer Studio - Explorer: Used for connecting to and inspecting NSO instance. Also referred to as NSO Explorer extension in this guide.

NSO Developer Studio - Developer Extension

This section describes the installation and functionality of the NSO Developer extension.

The purpose of the NSO Developer extension is to provide a base framework for developers to create their own NSO services. The focus of this guide is to manifest the creation of a simple NSO service package using the NSO Developer extension. At this time, reactive FastMAP and Nano services are not supported with this extension.

In terms of an NSO package, the extension supports YANG, XML, and Python to bring together various elements required to create a simple service.

After the installation, you can use the extension to create services and perform additional functions described below.

System Requirements

To get started with development using the NSO Developer extension, ensure that the following prerequisites are met on your system. The prerequisites are not a requirement to install the NSO Developer extension, but for NSO development after the extension is installed.

• Visual Studio Code.

• Java JDK 11 or higher.

• Python 3.9 or higher (recommended).

Install the Extension

Installation of the NSO Developer extension is done via the VS Code marketplace.

To install the NSO Developer extension in your VS Code environment:

1. Open VS Code and click the Extensions icon on the Activity Bar.

2. Search for the extension using the keywords "nso developer studio" in the Search Extensions in Marketplace field.

3. In the search results, locate the extension (NSO Developer Studio - Developer) and click Install.

4. Wait while the installation completes. A notification at the bottom-right corner indicates that the installation has finished. After the installation, an NSO icon is added to the Activity Bar.

Make a New Service Package (Python only)

Use the Make Package command in VS Code to create a new Python package. The purpose of this command is to provide functionality similar to the ncs-make-package CLI command, that is, to create a basic structure for you to start developing a new Python service package. The ncs-make-package command, however, comes with several additional options to create a package.

To make a new Python service package:

1. In the VS Code menu, go to View, and choose Command Palette.

2. In the Command Palette, type or pick the command NSO: Make Package. This brings up the Make Package dialog where you can configure package details.

3. In the Make Package dialog, specify the following package details:

   • Package Name: Name of the package.

   • Package Location: Destination folder where the package is to be created.

   • Namespace: Namespace of the YANG module, e.g. http://www.cisco.com/myModule.

   • Prefix: The prefix to be given to the YANG module, e.g. msp.

   • Yang Version: The YANG version that this module follows.

4. Click Create Package. This creates the required package and opens up a new instance of VS Code with the newly created NSO package.

5. If the Workspace Trust dialog is shown, click Yes, I Trust the Authors.

Open an Existing Package

Use the Open Existing Package command to open an already existing package.

To open an existing package:

1. In the VS Code menu, go to View, then choose Command Palette.

2. In the Command Palette, type or pick the command NSO: Open Existing Package.

3. Browse for the package on your local disk and open it. This brings up a new instance of VS Code and opens the package in it.

Edit YANG files

Opening a YANG file for edit results in VS Code detecting syntax errors in the YANG file. The errors show up due to missing path to YANG files and can be resolved using the following procedure.

Add YANG models for Yangster

For YANG support, a third-party extension called Yangster is used. Yangster is able to resolve imports for core NSO models but requires additional configuration.

To add YANG models for Yangster:

1. Create a new file named yang.settings by right-clicking in the blank area of the Explorer view and choosing New File from the pop-up.

2. Locate the NSO source YANG files on your local disk and copy the path.

3. In the file yang.settings, enter the path in the JSON format: { "yangPath": "<path to Yang files >" }, for example, { "yangPath": /home/my-user-name/nso-6.0/src/ncs/yang}. On Microsoft Windows, make sure that the backslash (\) is escaped, e.g., "C:\\user\\folder\\src\\yang".

4. Save the file.

5. Wait while the Yangster extension indexes and parses the YANG file to resolve NSO imports. After the parsing is finished, errors in the YANG file will disappear.

View YANG Diagram

YANG diagram is a feature provided by the Yangster extension.

To view the YANG diagram:

1. Update the YANG file. (Pressing Ctrl+space brings up auto-completion where applicable.)

2. Right-click anywhere in the VS Code Editor area and select Open in Diagram in the pop-up.

Add a New YANG Module

To add a new YANG module:

1. In the Explorer view, navigate to the yang folder and select it.

2. Right-click on the yang folder and select NSO: Add Yang Module from the pop-up menu. This brings up the Create Yang Module dialog where you can configure module details.

3. In the Create Yang Module dialog, fill in the following details:

   • Module Name: Name of the module.

   • Namespace: Namespace of the module, e.g., http://www.cisco.com/myModule.

   • Prefix: Prefix for the YANG module.

   • Yang Version: Version of YANG for this module.

4. Click Finish. This creates and opens up the newly created module.

Add a Service Point

Often while working on a package, there is a requirement to create a new service. This usually involves adding a service point. Adding a service point also requires other parts of the files to be updated, for example, Python.

Service points are usually added to lists.

To add a service point:

1. Update your YANG model as required. The extension automatically detects the list elements and displays a CodeLens called Add Service Point. An example is shown below.
2. Click the Add Service Point CodeLens. This brings up the Add Service Point dialog.

3. Fill in the Service Point ID that is used to identify the service point, for example, mySimpleService.

4. Next, in the Python Details section, select using the Python Module field if you want to create a new Python module or use an existing one.

   • If you opt to create a new Python file, relevant sections are automatically updated in package-meta-data.xml.

   • If you select an existing Python module from the list, it is assumed that you are selecting the correct module and that, it has been created correctly, i.e., the package-meta-data.xml file is updated with the component definition.

5. Enter the Service CB Class, for example, SimpleServiceCB.

6. Finish creating the service by clicking Add Service Point.

Register an Action Point

All action points in a YANG model must be registered in NSO. Registering an action point also requires other parts of the files to be updated, for example, Python (register_action), and update package-meta-data if needed.

Action points are usually defined to lists or containers.

To register an action point:

1. Update your YANG model as required. The extension automatically detects the action point elements in YANG and displays a CodeLens called Add Action Point. An example is shown below.

   Note that it is mandatory to specify tailf:actionpoint <actionpointname> under tailf:action <actionname>. This is a known limitation.

   The action point CodeLens at this time only works for the tailf:action statement, and not for the YANG rpc or YANG 1.1 action statements.

2. Click the Add Action Point CodeLens. This brings up the Register Action Point dialog.

3. Next, in the Python Details section, select using the Python Module field if you want to create a new Python module or use an existing one.

   • If you opt to create a new Python file, relevant sections are automatically updated in package-meta-data.xml.

   • If you select an existing Python module from the list, it is assumed that you are selecting the correct module, and that it has been created correctly, i.e., the package-meta-data.xml file is updated with the component definition.

4. Enter the action class name in the Main Class name used as entry point field, for example, MyAction.

5. Finish by clicking Register Action Point.

Edit Python Files

Opening a Python file uses the Microsoft Pylance extension. This extension provides syntax highlighting and other features such as code completion.

Add a New Python Module

To add a new Python module:

1. In the Primary Sidebar, Explorer view, right-click on the python folder.

2. Select NSO: Add Python Module from the pop-up. This brings up the Create Python Module dialog.

3. In the Create Python Module dialog, fill in the following details:

   • Module Name: Name of the module, for example, MyServicePackage.service.

   • Component Name: Name of the component that will be used to identify this module, for example, service.

   • Class Name: Name of the class to be invoked, for example, Main.

4. Click Finish.

Use Python Code Completion Snippets

Pre-defined snippets in VS Code allow for NSO Python code completion.

To use a Python code completion snippet:

1. Open a Python file for editing.

2. Type in one of the following pre-defined texts to display snippet options:

   • maapi: to view options for creating a maapi write transaction.

   • ncs: to view options for snippet for ncs template and variables.

3. Select a snippet from the pop-up to insert its code. This also highlights config items that can be changed. Press the Tab key to cycle through each value.

Edit XML Template Files

The final part of a typical service development is creating and editing the XML configuration template.

Add a New XML Template

To add a new XML template:

1. In the Primary Sidebar, Explorer view, right-click on the templates folder.

2. Select NSO: Add XML Template from the pop-up. This brings up the Add XML Template dialog.

3. In the Add XML Template dialog, fill in the XML Template name, for example, mspSimpleService.

4. Click Finish.

Use XML Code Completion Snippets

Pre-defined snippets in VS Code allow for NSO XML code completion of processing instructions and variables.

To use an XML code completion snippet:

1. Open an XML file for editing.

2. Type in one of the following pre-defined texts to display snippet options:

   • For processing instructions: <? followed by a character, for example <?i to view snippets for an if statement. All supported processing instructions are available as snippets.

   • For variables: $ followed by a character(s) matching the variable name, for example, $VA to view the variable snippet. Variables defined in the XML template via the <?set processing instruction or defined in Python code are displayed.

     Note: Auto-completion can also be triggered by pressing the Ctrl+Space keys.

3. Select an option from the pop-up to insert the relevant XML processing instruction or variable. Items that require further configuration are highlighted. Press the Tab key to cycle through the items.

XML Code Validation

The NSO Developer extension also performs code validation wherever possible. The following warning and error messages are shown if the extension is unable to validate the code:

• A warning is shown if a user enters a variable in an XML template that is not detected by the NSO Developer extension.

• An error message is shown if the ending tags in a processing instruction do not match.

Limitations

The extension provides help on a best-effort basis by showing error messages and warnings wherever possible. Still, in certain situations, code validation is not possible. An example of such a limitation is when the extension is not able to detect a template variable that is defined elsewhere and passed indirectly (i.e., the variable is not directly called).

Consider the following code for example, where the extension will successfully detect that a template variable IP_ADDRESS has been set.

vars.add('IP_ADDRESS','192.168.0.1')

Now consider the following code. While it serves the same purpose, it will fail to be detected.

ip_add_var_name = 'IP_ADDRESS' vars.add(ip_add_var_name, '192.168.0.1')

NSO Developer Studio - Explorer Extension

This section describes the installation and functionality of the NSO Explorer extension.

The purpose of the NSO Explorer extension is to allow the user to connect to a running instance of NSO and navigate the CDB from within VS Code.

System Requirements

To get started with the NSO Explorer extension, ensure that the following prerequisites are met on your system. The prerequisites are not a requirement to install the NSO Explorer extension, but for NSO development after the extension is installed.

• Visual Studio Code.

• Java JDK 11 or higher.

• Python 3.9 or higher (recommended).

Install the Extension

Installation of the NSO Explorer extension is done via the VS Code marketplace.

To install the NSO Explorer extension in your VS Code environment:

1. Open VS Code and click the Extensions icon on the Activity Bar.

2. Search for the extension using the keywords "nso developer studio" in the Search Extensions in Marketplace field.

3. In the search results, locate the extension (NSO Developer Studio - Explorer) and click Install.

4. Wait while the installation completes. A notification at the bottom-right corner indicates that the installation has finished. After the installation, an NSO icon is added to the Activity Bar.

Connect to NSO Instance

The NSO Explorer extension allows you to connect to and inspect a live NSO instance from within the VS Code. This procedure assumes that you have not previously connected to an NSO instance.

To connect to an NSO instance:

1. In the Activity Bar, click the NSO icon to open NSO Explorer.

2. If no NSO instance is already configured, a welcome screen is displayed with an option to add a new NSO instance.

3. Click the Add NSO Instance button to open the Settings editor.

4. In the Settings editor, click the link Edit in settings.json. This opens the settings.jsonfile for editing.

5. Next, edit the settings.json file as shown below:
6. Save the file when done.

   If settings have been configured correctly, NSO Explorer will attempt to connect to the running NSO instance and display the NSO configuration.

Inspect the CDB Tree

Once the NSO Explorer extension is configured, the user can inspect the CDB tree.

To inspect the CDB tree, use the following functions:

• Get Element Info: Click the i (info) icon on the Explorer bar, or alternatively inline next to an element in the Explorer view.

• Copy KeyPath: Click the {KP} icon to copy the keypath for the selected node.

• Copy XPath: Click the {XP} icon to copy the XPath for the selected node.

• Get XML Config: Click the XML icon to retrieve the XML configuration for the selected node and copy it to the clipboard.

If data has changed in NSO, click the refresh button at the top of the Explorer pane to fetch it.

Kickers constitute a declarative notification mechanism for triggering actions on certain stimuli like a database change or a received notification. These different stimuli and their kickers are defined separately as data kicker and notification kicker respectively.

Common to all types of kickers is that they are declarative. Kickers are modeled in YANG and Kicker instances are stored as configuration data in CDB.

Immediately after a transaction, that defines a new kicker, is committed, the kicker will be active. The same holds for removal. This also implies that the amount of programming for a kicker is a matter of implementing the action to be invoked.

The data-kicker replicates much of the functionality otherwise attained by a CDB subscriber. Without the extra coding in registration and runtime daemon that comes with a CDB subscriber. The data-kicker works for all data providers.

The notification-kicker reacts to notifications received by NSO using a defined notification subscription under /ncs:devices/device/notifications/subscription. This simplifies the handling of southbound emitted notifications. Traditionally these were chosen to be stored in CDB as operational data and a separate CDB subscriber was used to act on the received notifications. With the use of the notification-kicker, the CDB subscriber can be removed and there is no longer any need to store the received notification in CDB.

Kicker Action Invocation

An action as defined by YANG contains an input parameter definition and an output parameter definition. However, a kicker that invokes an action treats the input parameters in a specific way.

The kicker mechanism first checks if the input parameters match those in the kicker:action-input-params YANG grouping defined in the tailf-kicker.yang file. If so, the action will be invoked with the input parameters:

• kicker-id: The id (name) of the invoking kicker.

• path: The path of the current monitor triggering the kicker.

• tid: The transaction ID to a synthetic transaction containing the changes that lead to the triggering of the kicker.

The "synthetic" transaction implies that this is a copy of the original transaction that led to the kicker triggering. It only contains the data tree under the monitor. The original transaction is already committed and this data might no longer reflect the "running" datastore. It's useful in that the action implementation can attach and diff-iterate over this transaction and retrieve the certain changes that lead to the kicker invocation.

If the kicker mechanism finds an action that does not match the above input parameters, it will invoke the action with an empty parameter list. This implies that a kicker action must either match the above kicker:action-input-params grouping precisely or accept an empty incoming parameter list. Otherwise, the action invocation will fail.

Data Kicker Concepts

For a data kicker, the following principles hold:

• Kickers are triggered by changes in the sub-tree indicated by the monitor parameter.

• Actions are invoked during the commit phase. Hence aborted transactions never trigger kickers.

• Kickers process both, configuration and operational data changes, but can be configured to react to a certain type of change only.

• No distinction is made between CRUD types, i.e., create, delete, update. All changes potentially trigger kickers.

• Kickers may have constraints that suppress invocations. Changes in the sub-tree indicated by monitor is a necessary but perhaps not a sufficient condition for the action to be invoked.

Generalized Monitors

For a data kicker, it is the monitor that specifies which subtree under which a change should invoke the kicker. The monitor leaf is of type node-instance-identifier which means that predicates for keys are optional, i.e., keys may be omitted and then represent all instances for that key.

The resulting evaluation of the monitor defines a node set. Each node in this node set will be the root context for any further xpath evaluations necessary before invoking the kicker action.

The following example shows the strengths of using xpath to define the kickers. Say that we have a situation described by the following YANG model snippet:

module example {
  namespace "http://tail-f.com/ns/test/example";
  prefix example;

  ...

  container sys {
    list ifc {
      key name;
      max-elements 64;
      leaf name {
        type interfaceName;
      }
      leaf description {
        type string;
      }
      leaf enabled {
        type boolean;
        default true;
      }
      container hw {
        leaf speed {
          type interfaceSpeed;
        }
        leaf duplex {
          type interfaceDuplex;
        }
        leaf mtu {
          type mtuSize;
        }
        leaf mac {
          type string;
        }
      }
      list ip {
        key address;
        max-elements 1024;
        leaf address {
          type inet:ipv4-address;
        }
        leaf prefix-length {
          type prefixLengthIPv4;
          mandatory true;
        }
        leaf broadcast {
          type inet:ipv4-address;
        }
      }

      tailf:action local_me {
        tailf:actionpoint kick-me-point;
        input {
        }
        output {
        }
      }
    }

    tailf:action kick_me {
      tailf:actionpoint kick-me-point;
      input {
      }
      output {
      }
    }

    tailf:action iter_me {
      tailf:actionpoint kick-me-point;
      input {
        uses kicker:action-input-params;
      }
      output {
      }
    }

  }
}

Then, we can define a kicker for monitoring a specific element in the list and call the correlated local_me action:

admin@ncs(config)# kickers data-kicker e1 \
> monitor /sys/ifc[name='port-0'] \
>kick-node /sys/ifc[name='port-0']\
> action-name local_me

admin(config-data-kicker-e1)# commit
Commit complete
admin(config-data-kicker-e1)# top
admin@ncs(config)#  show full-configuration kickers
kickers data-kicker e1
 monitor     /sys/ifc[name='port-0']
 kick-node   /sys/ifc[name='port-0']
 action-name local_me
!

On the other hand, we can define a kicker for monitoring all elements of the list and call the correlated local_me action for each element:

admin@ncs(config)# kickers data-kicker e2 \
> monitor /sys/ifc \
>kick-node . \
> action-name local_me

admin(config-data-kicker-e2)# commit
Commit complete
admin(config-data-kicker-e2)# top
admin@ncs(config)#  show full-configuration kickers
kickers data-kicker e2
 monitor     /sys/ifc
 kick-node   .
 action-name local_me
!

Here the . in the kick-node refers to the current node in the node set defined by the monitor.

Kicker Constraints/Filters

A data kicker may be constrained by adding conditions that suppress invocations. The leaf trigger-expression contains a boolean XPath expression that is evaluated twice, before and after the change-set of the commit has been applied to the database(s).

The XPath expression has to be evaluated twice to detect the change caused by the transaction.

The two boolean results together with the leaf trigger-type control if the kicker should be triggered or not:

• enter-and-leave: false -> true (i.e. positive flank) or true -> false (negative flank).

• enter: false -> true.

admin(config)# kickers data-kicker k1 monitor /sys/ifc \
> trigger-expr "hw/mtu > 800" \
> trigger-type enter \
> kick-node /sys \
> action-name kick_me
admin(config-data-kicker-k1)# commit
Commit complete
admin(config-data-kicker-k1)# top
admin@ncs%
admin@ncs% show kickers
kickers data-kicker k1
 monitor      /sys/ifc
 trigger-expr "hw/mtu > 800"
 trigger-type enter
 kick-node    /sys
 action-name  kick_me
!

Start by changing the MTU to 800:

admin(config)# sys ifc port-0 hw mtu 800
admin(config-ifc-port-0)# commit | debug kicker
 2017-02-15T16:35:36.039 kicker: k1 at /kicker_example:sys/kicker_example:ifc[kicker_example:name='port-0'] changed;
not invoking 'kick_me' trigger-expr false -> false
Commit complete.

Since the trigger-expression evaluates to false, the kicker is not triggered. Let's try again:

admin(config)# sys ifc port-0 hw mtu 801
admin(config-ifc-port-0)# commit | debug kicker
 2017-02-15T16:35:36.039 kicker: k1 at /kicker_example:sys/kicker_example:ifc[kicker_example:name='port-0'] changed;
invoking 'kick-me' trigger-expr false -> true
Commit complete.

The trigger-expression can in some cases be used to refine the monitor of kicker, to avoid unnecessary evaluations. Let's change something below the monitor that doesn't touch the nodes in the trigger-expression:

admin(config)# sys ifc port-0 speed ten
admin(config-ifc-port-0)# commit | debug kicker
Commit complete.

Notice there was no evaluation done.

Variable Bindings

A data kicker may be provided with a list of variables (named values). Each variable binding consists of a name and a XPath expression. The XPath expressions are evaluated on-demand, i.e. when used in either of monitor or trigger-expression nodes.

admin@ncs(config)# set kickers data-kicker k3 monitor $PATH/c
                         kick-node /x/y[id='n1']
                         action-name kick-me
                         variable PATH value "/a/b[k1=3][k2='3']"
admin@ncs(config)#

In the example above, PATH is defined and referred to by the monitor expression by using the expression $PATH.

A Simple Data Kicker Example

This example is part of the examples.ncs/web-server-farm/web-site-service example. It consists of an action and a README_KICKER file. For all kickers defined in this example, the same action is used. This action is defined in the web-site-service package.

The following is the YANG snippet for the action definition from the website.yang file:

module web-site {
  namespace "http://examples.com/web-site";
  prefix wse;

  ...

  augment /ncs:services {

    ...

    container actions {
      tailf:action diffcheck {
        tailf:actionpoint diffcheck;
        input {
          uses kicker:action-input-params;
        }
        output {
        }
      }
    }
  }

}

The implementation of the action can be found in the WebSiteServiceRFS.java class file. Since it takes the kicker:action-input-params as input, the Tid for the synthetic transaction is available. This transaction is attached and diff-iterated. The result of the diff-iteration is printed in the ncs-java-vm.log:

class WebSiteServiceRFS {

    ....

    @ActionCallback(callPoint="diffcheck", callType=ActionCBType.ACTION)
    public ConfXMLParam[] diffcheck(DpActionTrans trans, ConfTag name,
                                   ConfObject[] kp, ConfXMLParam[] params)
    throws DpCallbackException {
        try {

            System.out.println("-------------------");
            System.out.println(params[0]);
            System.out.println(params[1]);
            System.out.println(params[2]);

            ConfUInt32 val = (ConfUInt32) params[2].getValue();
            int tid = (int)val.longValue();

            Socket s3 = new Socket("127.0.0.1", Conf.NCS_PORT);
            Maapi maapi3 = new Maapi(s3);
            maapi3.attach(tid, -1);

            maapi3.diffIterate(tid, new MaapiDiffIterate() {
                // Override the Default iterate function in the TestCase class
                public DiffIterateResultFlag iterate(ConfObject[] kp,
                                                     DiffIterateOperFlag op,
                                                     ConfObject oldValue,
                                                     ConfObject newValue,
                                                     Object initstate) {
                    System.out.println("path = " + new ConfPath(kp));
                    System.out.println("op = " + op);
                    System.out.println("newValue = " + newValue);
                    return DiffIterateResultFlag.ITER_RECURSE;

                }

            });


            maapi3.detach(tid);
            s3.close();


        return new ConfXMLParam[]{};

        } catch (Exception e) {
            throw new DpCallbackException("diffcheck failed", e);
        }
    }
}

We are now ready to start the web-site-service example and define our data kicker. Do the following:

$ make all
$ ncs-netsim start
$ ncs
$ ncs_cli -C -u admin

admin@ncs# devices sync-from
sync-result {
    device lb0
    result true
}
sync-result {
    device www0
    result true
}
sync-result {
    device www1
    result true
}
sync-result {
    device www2
    result true
}

The kickers are defined under the hide-group debug. To be able to show and declare kickers, we need first to unhide this hide group:

admin@ncs# config
admin@ncs(config)# unhide debug

We now define a data-kicker for the profile list under the service augmented container /services/properties/wsp:web-site:

admin@ncs(config)# kickers data-kicker a1 \
> monitor /services/properties/wsp:web-site/profile \
> kick-node /services/wse:actions action-name diffcheck

admin@ncs(config-data-kicker-a1)# commit
admin@ncs(config-data-kicker-a1)# top
admin@ncs(config)# show full-configuration kickers data-kicker a1
kickers data-kicker a1
 monitor     /services/properties/wsp:web-site/profile
 kick-node   /services/wse:actions
 action-name diffcheck
!

We now commit a change in the profile list and we use the debug kicker pipe option to be able to follow the kicker invocation:

admin@ncs(config)# services properties web-site profile lean lb lb0
admin@ncs(config-profile-lean)# commit | debug kicker
 2017-02-15T16:35:36.039 kicker: a1 at /ncs:services/ncs:properties/wsp:web-site/wsp:profile[wsp:name='lean'] changed; invoking diffcheck
Commit complete.

admin@ncs(config-profile-lean)# top
admin@ncs(config)# exit

We can also check the result of the action by looking into the ncs-java-vm.log:

admin@ncs# file show logs/ncs-java-vm.log

In the end, we will find the following printout from the diffcheck action:

-------------------
{[669406386|id], a1}
{[669406386|monitor], /ncs:services/properties/web-site/profile{lean}}
{[669406386|tid], 168}
path = /ncs:services/properties/wsp:web-site/profile{lean}
op = MOP_CREATED
newValue = null
path = /ncs:services/properties/wsp:web-site/profile{lean}/name
op = MOP_VALUE_SET
newValue = lean
path = /ncs:services/properties/wsp:web-site/profile{lean}/lb
op = MOP_VALUE_SET
newValue = lb0
[ok][2017-02-15 17:11:59]

Notification Kicker Concepts

For a notification kicker, the following principles hold:

• Notification Kickers are triggered by the arrival of notifications from any device subscription. These subscriptions are defined under the /devices/device/notification/subscription path.

• Storing the received notifications in CDB is optional and not part of the notification kicker functionality.

• The ordering of kicker invocations is generally not guaranteed. That is, a kicker triggered at a later time might execute before a kicker that was triggered earlier, and kickers triggered for the same subscription may execute in any order. A priority and a serializer value can be used to modify this behavior.

Notification Selector Expression

The notification kicker is defined using a mandatory selector-expr which is an XPATH 1.0 expression. When the notification is received a synthetic transaction is started and the notification is written as if it would be stored under the path /devices/device/notification/received-notifications/data. Storing the notification in CDB is optional. The selector-expr is evaluated with the notification node as the current context and / as the root context. For example, if the device model defines a notification like this:

module device {
  ...
  notification mynotif {
    leaf message {
      type string;
    }
  }
  ...
}

The notification node mynotif will be the current context for the selector-expr There are four predefined variable bindings used when evaluating this expression:

• DEVICE: The name of the device emitting the current notification.

• SUBSCRIPTION_NAME: The name of the current subscription from which the notification was received. the kicker

• NOTIFICATION_NAME: The name of the current notification.

• NOTIFICATION_NS: The namespace of the current notification.

The selector-expr technique for defining the notification kickers is very flexible. For instance, a kicker can be defined to:

• Receive all notifications for a device.

• Receive all notifications of a certain type for any device.

• Receive a subset of notifications of a subset of devices by the use of specific subscriptions with the same name in several devices.

In addition to this usage of the predefined variable bindings, it is possible to further drill down into the specific notification to trigger on certain leafs in the notification.

Variable Bindings

In addition to the four variable bindings mentioned above, a notification kicker may also be provided with a list of variables (named values). Each variable binding consists of a name and an XPath expression. The XPath expression is evaluated when the selector-expr is run.

            admin@ncs(config)# set kickers notification-kicker k4
            selector-expr "$NOTIFICATION_NAME=linkUp and address[ip=$IP]"
            kick-node /x/y[id='n1']
            action-name kick-me
            variable IP value '192.168.128.55'
admin@ncs(config)#

In the example above, PATH is defined and referred to by the monitor expression by using the expression $PATH.

Serializer and Priority Values

These values are used to ensure the order of kicker execution. Priority orders kickers for the same notification event, while serializer orders kickers chronologically for different notification events. By default, when no serializer or priority value is given, kickers may be triggered in any order and in parallel. However, some situations may require stricter ordering, and setting serializer and priority in kicker configuration allows you to achieve it.

If priority for a set of kickers is specified, for each individual notification event, the kickers that match are executed in order, going from priority 0 to 255. For example, kicker K1 with priority 5 is executed before the kicker K2 with priority 8, which triggered for the same notification.

Parallel execution of kickers can also result in a situation where a kicker for a notification is executed after the kicker for a later notification. That is, even though the trigger for the first kicker came first, this kicker might have a priority set and must wait for other kickers to execute first, while the kicker for the next notification can execute right away. If there is a dependency between these two kickers, serializer value can ensure chronological ordering.

A serializer is a simple integer value between 0 and 255. Notification kickers configured with the same value will be executed in the order in which they were triggered, relative to each other. For example, suppose there are three kickers configured: T1 and T2 with serializer set to 10, and T3 with serializer of 20. NSO receives two notifications, the first triggering T1 and T3, and the second triggering T2. Because of the serializer, NSO guarantees T1 will be invoked before T2. But T2, even though it came in later, could potentially be invoked before T3 because they are not serialized (have different serializer value).

When using both, serializer and priority, only kickers with the same serializer value are priority ordered, that is, serializer value takes precedence. For example, the kicker Q1 with serializer 10 and priority 15 may execute before or after the kicker Q2 with serializer 20 and priority 4. The reason is Q1 may need to wait for other kickers with serializer 10 from previous events. The same is true for Q2 and previous kickers with serializer 20.

A Simple Notification Kicker Example

In this example, we use the same action and setup as in the data kicker example above. The procedure for starting is also the same.

The web-site-service example has devices that have notifications generated on the stream "interface". We start with defining the notification kicker for a certain SUBSCRIPTION_NAME = 'mysub'. This subscription does not exist for the moment and the kicker will therefore not be triggered:

admin@ncs# config

admin@ncs(config)# kickers notification-kicker n1 \
> selector-expr "$SUBSCRIPTION_NAME = 'mysub'" \
> kick-node /services/wse:actions \
> action-name diffcheck

admin@ncs(config-notification-kicker-n1)# commit
admin@ncs(config-notification-kicker-n1)# top

admin@ncs(config)# show full-configuration kickers notification-kicker n1
kickers notification-kicker n1
 selector-expr "$SUBSCRIPTION_NAME = 'mysub'"
 kick-node     /services/wse:actions
 action-name   diffcheck
!

Now we define the mysub subscription on a device www0 and refer to the notification stream interface. As soon as this definition is committed, the kicker will start triggering:

admin@ncs(config)# devices device www0 notifications subscription mysub \
> local-user admin stream interface
admin@ncs(config-subscription-mysub)# commit

admin@ncs(config-profile-lean)# top
admin@ncs(config)# exit

If we now inspect the ncs-java-vm.log, we will see a number of notifications that are received. We also see that the transaction that is diff-iterated contains the notification as data under the path /devices/device/notifications/received-notifications/notification/data. This is an operational data list. However, this transaction is synthetic and will not be committed. If the notification will be stored CDB is optional and not depending on the notification kicker functionality:

admin@ncs# file show logs/ncs-java-vm.log

-------------------
{[669406386|id], n1}
{[669406386|monitor], /ncs:devices/device{www0}/notifications.../data/linkUp}
{[669406386|tid], 758}
path = /ncs:devices/device{www0}
op = MOP_MODIFIED
newValue = null
path = /ncs:devices/device{www0}/notifications...
op = MOP_CREATED
newValue = null
path = /ncs:devices/device{www0}/notifications.../event-time
op = MOP_VALUE_SET
newValue = 2017-02-15T16:35:36.039204+00:00
path = /ncs:devices/device{www0}/notifications.../sequence-no
op = MOP_VALUE_SET
newValue = 0
path = /ncs:devices/device{www0}/notifications.../data/notif:linkUp
op = MOP_CREATED
newValue = null
path = /ncs:devices/device{www0}/notifications.../data/notif:linkUp/address{192.168.128.55}
op = MOP_CREATED
newValue = null
path = /ncs:devices/device{www0}/notifications.../data/notif:linkUp/address{192.168.128.55}/ip
op = MOP_VALUE_SET
newValue = 192.168.128.55
path = /ncs:devices/device{www0}/notifications.../data/notif:linkUp/address{192.168.128.55}/mask
op = MOP_VALUE_SET
newValue = 255.255.255.0
path = /ncs:devices/device{www0}/notifications.../data/notif:linkUp/ifName
op = MOP_VALUE_SET
newValue = eth2
path = /ncs:devices/device{www0}/notifications.../data/notif:linkUp/linkProperty{0}
op = MOP_CREATED
newValue = null
path = /ncs:devices/device{www0}/notifications.../data/notif:linkUp/linkProperty{0}/extensions{0}
op = MOP_CREATED
newValue = 4668
path = /ncs:devices/device{www0}/notifications.../data/notif:linkUp/linkProperty{0}/extensions{1}/name
op = MOP_VALUE_SET
newValue = 2
path = /ncs:devices/device{www0}/notifications.../data/notif:linkUp/linkProperty{0}/flags
op = MOP_VALUE_SET
newValue = 42
path = /ncs:devices/device{www0}/notifications.../data/notif:linkUp/linkProperty{0}/newlyAdded
op = MOP_CREATED
newValue = null

We end by removing the kicker and the subscription:

admin@ncs# config
admin@ncs(config)# no kickers notification-kicker
admin@ncs(config)# no devices device www0 notifications subscription
admin@ncs(config)# commit

Nano Services Reactive FastMap with Kicker

Nano services use kickers to trigger executing state callback code, run templates, and execute actions according to a plan when pre-conditions are met. For more information see Nano Services for Provisioning with Side Effects and Nano Services for Staged Provisioning.

Debugging Kickers

Kicker CLI Debug Target

To find out why a Kicker kicked when it shouldn't or more commonly and annoying, why it didn't kick when it should, use the CLI pipe debug kicker.

Evaluation of potential Kicker invocations are reported in the CLI together with XPath evaluation results:

admin@ncs(config)# set sys ifc port-0 hw mtu 8000
admin@ncs(config)# commit | debug kicker
 2017-02-15T16:35:36.039 kicker: k1 at /kicker_example:sys/kicker_example:ifc[kicker_example:name='port-0'] changed;
not invoking 'kick-me' trigger-expr false -> false
Commit complete.
admin@ncs(config)#

Unhide Kickers

The top-level container kickers is by default invisible due to a hidden attribute. To make kickers visible in the CLI, two steps are required.

1. First, the following XML snippet must be added to ncs.conf.\

   Copy

   <hide-group>
       <name>debug</name>
   </hide-group>

2. Next, the unhide command can be used in the CLI session.\

   Copy

   admin@ncs(config)# unhide debug
   admin@ncs(config)#

XPath Log

Detailed information from the XPath evaluator can be enabled and made available in the xpath log. Add the following snippet to ncs.conf.

<xpathTraceLog>
  <enabled>true</enabled>
  <filename>./xpath.trace</filename>
</xpathTraceLog>

Devel Log

Error information is written in the development log. The development log is meant to be used as support while developing the application. It is enabled in ncs.conf:

<developer-log>
  <enabled>true</enabled>
  <file>
    <name>./logs/devel.log</name>
     <enabled>true</enabled>
  </file>
</developer-log>
<developer-log-level>trace</developer-log-level>

With an increasing number of services and managed devices in NSO, performance becomes a more important aspect of the system. At the same time, other aspects, such as the way you organize code, also start playing an important role when using NSO on a bigger scale.

The following section examines these concerns and presents the available options for scaling your NSO automation solution.

Understanding Your Use Case

NSO allows you to tackle different automation challenges and every solution has its own specifics. Therefore, the best approach to scaling depends on the way the solution is implemented. What works in one case may be useless, or effectively degrade performance, for another. You must first analyze and understand how your particular use case behaves, which will then allow you to take the right approach to scaling.

When trying to improve the performance, a very good, possibly even the best starting point is to inspect the tracing data. Tracing is further described in Progress Trace. Yet a simple commit | details command already provides a lot of useful data.

admin@ncs(config-mysvc-test)# commit | details
 2022-09-16T09:17:48.977 applying transaction...
entering validate phase for running usid=54 tid=225 trace-id=3a4a3b7f-a09f-4f9d-b05e-1656310ea5b6
 2022-09-16T09:17:48.977 creating rollback checkpoint... ok (0.000 s)
 2022-09-16T09:17:48.978 creating rollback file... ok (0.004 s)
 2022-09-16T09:17:48.983 creating pre-transform checkpoint... ok (0.000 s)
 2022-09-16T09:17:48.983 run pre-transform validation... ok (0.000 s)
 2022-09-16T09:17:48.983 creating transform checkpoint... ok (0.000 s)
 2022-09-16T09:17:48.983 run transforms and transaction hooks...
 2022-09-16T09:17:48.985 taking service write lock... ok (0.000 s)
 2022-09-16T09:17:48.985 holding service write lock...
 2022-09-16T09:17:48.986 service /mysvc[name='test']: run service... ok (0.012 s)
 2022-09-16T09:17:48.999 run transforms and transaction hooks: ok (0.016 s)
 2022-09-16T09:17:48.999 creating validation checkpoint... ok (0.000 s)
 2022-09-16T09:17:49.000 mark inactive... ok (0.000 s)
 2022-09-16T09:17:49.001 pre validate... ok (0.000 s)
 2022-09-16T09:17:49.001 run validation over the changeset... ok (0.000 s)
 2022-09-16T09:17:49.002 run dependency-triggered validation... ok (0.000 s)
 2022-09-16T09:17:49.003 check configuration policies... ok (0.000 s)
 2022-09-16T09:17:49.003 check for read-write conflicts... ok (0.000 s)
 2022-09-16T09:17:49.004 taking transaction lock... ok (0.000 s)
 2022-09-16T09:17:49.004 holding transaction lock...
 2022-09-16T09:17:49.004 check for read-write conflicts... ok (0.000 s)
 2022-09-16T09:17:49.004 applying service meta-data... ok (0.000 s)
leaving validate phase for running usid=54 tid=225 trace-id=3a4a3b7f-a09f-4f9d-b05e-1656310ea5b6 (0.028 s)
entering write-start phase for running usid=54 tid=225 trace-id=3a4a3b7f-a09f-4f9d-b05e-1656310ea5b6
 2022-09-16T09:17:49.005 cdb: write-start
 2022-09-16T09:17:49.006 ncs-internal-service-mux: write-start
 2022-09-16T09:17:49.006 ncs-internal-device-mgr: write-start
 2022-09-16T09:17:49.007 cdb: match subscribers... ok (0.000 s)
 2022-09-16T09:17:49.007 cdb: create pre commit running... ok (0.000 s)
 2022-09-16T09:17:49.007 cdb: write changeset... ok (0.000 s)
 2022-09-16T09:17:49.008 check data kickers... ok (0.000 s)
leaving write-start phase for running usid=54 tid=225 trace-id=3a4a3b7f-a09f-4f9d-b05e-1656310ea5b6 (0.003 s)
entering prepare phase for running usid=54 tid=225 trace-id=3a4a3b7f-a09f-4f9d-b05e-1656310ea5b6
 2022-09-16T09:17:49.009 cdb: prepare
 2022-09-16T09:17:49.009 ncs-internal-device-mgr: prepare
 2022-09-16T09:17:49.022 device ex1: push configuration...
leaving prepare phase for running usid=54 tid=225 trace-id=3a4a3b7f-a09f-4f9d-b05e-1656310ea5b6 (0.121 s)
entering commit phase for running usid=54 tid=225 trace-id=3a4a3b7f-a09f-4f9d-b05e-1656310ea5b6
 2022-09-16T09:17:49.130 cdb: commit
 2022-09-16T09:17:49.130 cdb: switch to new running... ok (0.000 s)
 2022-09-16T09:17:49.132 ncs-internal-device-mgr: commit
 2022-09-16T09:17:49.149 device ex1: push configuration: ok (0.126 s)
 2022-09-16T09:17:49.151 holding service write lock: ok (0.166 s)
 2022-09-16T09:17:49.151 holding transaction lock: ok (0.147 s)
leaving commit phase for running usid=54 tid=225 trace-id=3a4a3b7f-a09f-4f9d-b05e-1656310ea5b6 (0.021 s)
 2022-09-16T09:17:49.151 applying transaction: ok (0.174 s)
Commit complete.
admin@ncs(config-mysvc-test)#

Pay attention to the time NSO spends doing specific tasks. For a simple service, these are mainly:

• Validate service data (pre-transform validation)

• Run service mapping logic

• Validate produced configuration (changeset)

• Push changes to affected devices

• Commit the new configuration

Tracing data can often quickly reveal a bottleneck, a hidden delay, or some other unexpected inefficiency in your code. The best strategy is to first address any such concerns if they show up since only well-performing code is a good candidate for further optimization. Otherwise, you might find yourself optimizing the wrong parameters and hitting a dead end. Visualizing the progress trace is often helpful in identifying bottlenecks. See Measuring Transaction Throughput.

Analyzing the service in isolation can yield useful insight. But it may also lead you in the wrong direction because some issues only manifest under load and the data from a live system can surprise you. That is why NSO supports different ways of exposing tracing information, including operational data and notification events. Remember to always verify that your observations and assumptions hold for a live, production system, too.

Where to Start?

The times for different parts of the transaction, as reported by the tracing data, are very useful in determining where to focus your efforts.

For example, if your service data model uses a very broad must or similar XPath statement, then NSO may potentially need to evaluate thousands of data entries. Such evaluation requires a considerable amount of additional processing and is, in turn, reflected in increased time spent in validation. The solution in this case is to limit the scope of the data referenced in the YANG constraint, which you can often achieve with a more specific XPath expression.

Similarly, if a significant amount of time is spent constructing a service mapping, perhaps there is some redundant work occurring that you could optimize? Sometimes, however, provisioning requires calls to other systems or some computationally expensive operation, which you cannot easily manage without. Then you might want to consider splitting the provisioning process into smaller pieces, using nano services, for example. See Simplify the Per-Device Concurrent Transaction Creation Using a Nano Service for an example use-case and references to the Nano service documentation.

In general, your own code for a single transaction with no additional load on NSO should execute quickly (sub-second, as a rule of thumb). The faster each service or action code is, the better the overall system performance. Using a service design pattern to both improve performance and scale and avoid conflicts is described in Design to Minimize Conflicts.

Divide the Work Correctly

Things such as reading external data or large computations should not be done inside the create code. Consider using an action to encapsulate these functions. An action does not run under the lock unless it triggers a transaction and can perform side effects as desired.

There are several ways to utilize an action:

• An action is allowed to perform side effects.

• An action can read operational data from devices or external systems.

• An action can write values to operational data in CDB, for later use from the service.

• An action can write configuration to CDB, potentially triggering a service.

Actions can be used together with nano services, see Simplify the Per-Device Concurrent Transaction Creation Using a Nano Service.

Optimizing Device Communication

With the default configuration, one of the first things you might notice standing out in the tracing data is that pushing device configuration takes a significant amount of time compared to other parts of service provisioning. Why is that?

All changes in NSO happen inside a transaction. Network devices participate in the transaction, which gives you the all-or-nothing behavior, to ensure correctness and consistency across the network. But network communication is not instantaneous and a transaction in NSO holds a lock while waiting for devices to process the change. This way, changes to network devices are serialized, even when there are multiple simultaneous transactions. However, a lock blocks other transactions from proceeding, ultimately limiting the overall NSO transaction rate.

So, in many cases, the NSO system is not really resource-constrained but merely experiencing lock contention. Therefore, making locks as short as possible is the best way to improve performance. In the example trace from the section Understanding Your Use Case, most of the time is spent in the prepare phase, where configuration changes are propagated to the network devices. Change propagation requires a management session with each participating device, as well as updating and validating the new configuration on the device side. Understandably, all of these tasks take time.

NSO allows you to influence this behavior. Take a look at Commit Queue on how to avoid long device locks with commit queues and the trade-offs they bring. Usually, enabling the commit queue feature is the first and the most effective step to significantly improving transaction times.

Improving Subscribers

The CDB subscriber mechanism is used to notify the application code about CDB changes and runs at the end of the transaction commit, inside a global lock. Due to this fact, the number and configuration of subscribers affect performance and should be investigated early in your performance optimization efforts.

A badly implemented subscriber prolongs the time the transaction holds the lock, preventing other transactions from completing, in addition to the original transaction taking more time to commit. There are mainly two reasons for suboptimal operation: either the subscriber is too broad and must process too many (irrelevant) changes, or it performs more work inside the lock as necessary. As a recommended practice, the subscriber should only note the changes and schedule the processing to be done later, in order to return and release the lock as quickly as possible.

Moreover, subscribers incur processing overhead regardless of their implementation because NSO needs to communicate with the custom subscriber code, typically written in Java or Python.

That is why modern, performant code in NSO should use the kicker mechanism instead of implementing custom subscribers. While it is still possible to create a badly performing kicker, you are less likely to do so inadvertently. In most situations, kickers are also easier to implement and troubleshoot. You can read more on kickers in Kicker.

Minimizing Concurrency Conflicts

The time it takes to complete a transaction is certainly an important performance metric. However, after a certain point, it gets increasingly hard or even impossible to get meaningful improvement from optimizing each individual transaction. As it turns out, on a busy system, there are usually multiple outstanding requests. So, instead of trying to process each as fast as possible one after another, the system might process them in parallel.

In practice and as the figure shows, some parts must still be processed sequentially to ensure transactional properties. However, there is a significant gain in the overall time it takes to process all transactions in a busy system, even though each might take a little longer individually due to the concurrency overhead.

Throughput then becomes a more relevant metric. It is the number of requests or transactions that the system can process in a given time unit. While throughput is still related to individual transaction times, other factors also come into play. An important one is the way in which NSO implements concurrency and the interaction between the transaction system and your, user, code. Designing for transaction throughput is covered in detail later in this section, and the NSO concurrency model is detailed in NSO Concurrency Model.

The section provides guidance on identifying transaction conflicts and what affects their occurrence, so you can make your code more resistant to producing them. Conflicts arise more frequently on busier systems and negatively affect throughput, which makes them a good candidate for optimization.

Fine-tuning the Concurrency Parameters

Depending on the specifics of the server running NSO, additional performance improvement might be possible by fine-tuning the transaction-limits set of configuration parameters in ncs.conf. Please see the ncs.conf(1) manpage for details.

Enabling Even More Parallelism

If you are experiencing high resource utilization, such as memory and CPU usage, while individual transactions are optimized to execute fast and the rate of conflicts is low, it's possible you are starting to see the level of demand that pushes the limits of this system.

First, you should try adding more resources, in a scale-up manner, if possible. At the same time, you might also have some services that are using an older, less performant user code execution model. For example, the way Python code is executed is controlled by the callpoint-model option, described in The application Component, which you should ensure is set to the most performant setting.

Regardless, a single system cannot scale indefinitely. After you have exhausted all other options, you will need to “scale out,” that is, split the workload across multiple NSO instances. You can achieve this by using the Layered Service Architecture (LSA) approach. But the approach has its trade-offs, so make sure it provides the right benefits in your case. The LSA is further documented in LSA Overview in Layered Service Architecture.

Limit sync-from

In a brownfield environment, where the configuration is not 100% automated and controlled by NSO alone but also written to by other systems or operators, NSO is bound to end up out-of-sync with the device. How to handle synchronization is a big topic, and it is vital to understand what it means to you when things are out of sync. This will help guide your strategy.

If NSO is frequently brought out of sync, it can be tempting to invoke sync-from from the create callback. While it does achieve a higher degree of reliability in the sense that service modifications won't return an out-of-sync error, the impact on performance is usually catastrophic. The typical sync-from operation takes orders of magnitudes longer than the typical service modification, and transactional throughput will suffer greatly.

But other alternatives are often better:

• You can synchronize the configuration from the device when it reports a change rather than when the service is modified by listening for configuration change events from the device, e.g., via RESTCONF or NETCONF notifications, SNMP traps, or Syslog, and invoking sync-from or partial-sync-from when another party (not NSO) has modified the device. See also the section called Partial Sync.

• Using the devices sync-from command does not hold the transaction lock and run across devices concurrently, which reduces the total amount of time spent time synchronizing. This is particularly useful for periodic synchronization to lower the risk of being out-of-sync when committing configuration changes.

• Using the no-overwrite commit flag, you can be more lax about being in sync and focus on not overwriting the modified configuration.

• If the configuration is 100% automated and controlled by NSO alone, using out-of-sync-behaviour accept, you can completely ignore if the device is in sync or not.

• Letting your modification fail with an out-of-sync error and handling that error at the calling side.

Designing for Maximal Transaction Throughput

Maximal transaction throughput refers to the maximum number of transactions a system can handle within a given period. Factors that can influence maximal transaction throughput include:

• Hardware capabilities (e.g., processing power, memory).

• Software efficiency.

• Network bandwidth.

• The complexity of the transactions themselves.

Besides making sure the system hardware capabilities and network bandwidth are not a bottleneck, there are four areas where the NSO user can significantly affect the transaction throughput performance for an NSO node:

• Run multiple transactions concurrently. For example, multiple concurrent RESTCONF or NETCONF edits, CLI commits, MAAPI apply(), nano service re-deploy, etc.

• Design to avoid conflicts and minimize the service create() and validation implementation. For example, in service templates and code mapping to devices or other service instances, YANG must statements with XPath expressions or validation code.

• Using commit queues to exclude the time to push configuration changes to devices from inside the transaction lock.

• Simplify using nano and stacked services. If the processor where NSO with a stacked service runs becomes a severe bottleneck, the added complexity of migrating the stacked service to an LSA setup can be motivated. LSA helps expose only a single service instance when scaling up the number of devices by increasing the number of available CPU cores beyond a single processor.

Measuring Transaction Throughput

Measuring transaction performance includes measuring the total wall-clock time for the service deployment transaction(s) and using the detailed NSO progress trace of the transactions to find bottlenecks. The developer log helps debug the NSO internals, and the XPath trace log helps find misbehaving XPath expressions used in, for example, YANG must statements.

The picture below shows a visualization of the NSO progress trace when running a single transaction for two service instances configuring a device each:

The total RESTCONF edit took ~5 seconds, and the service mapping (“creating service” event) and validation (“run validation ...” event) were done sequentially for the service instances and took 2 seconds each. The configuration push to the devices was done concurrently in 1 second.

For progress trace documentation, see Progress Trace.

Running the perf-trans Example Using a Single Transaction

The perf-trans example from the NSO example set explores the opportunities to improve the wall-clock time performance and utilization, as well as opportunities to avoid common pitfalls.

The example uses simulated CPU loads for service creation and validation work. Device work is simulated with sleep() as it will not run on the same processor in a production system.

The example shows how NSO can benefit from running many transactions concurrently if the service and validation code allow concurrency. It uses the NSO progress trace feature to get detailed timing information for the transactions in the system.

The provided code sets up an NSO instance that exports tracing data to a .csv file, provisions one or more service instances, which each map to a device, and shows different (average) transaction times and a graph to visualize the sequences plus concurrency.

Play with the perf-trans example by tweaking the measure.py script parameters:

plain patch

See the README in the perf-trans example for details.

To run the perf-trans example from the NSO example set and recreate the variant shown in the progress trace above:

cd $NCS_DIR/examples.ncs/development-guide/concurrency-model/perf-trans
make NDEVS=2 python
python3 measure.py --ntrans 1 --nwork 2 --ndtrans 2 --cqparam bypass --ddelay 1
python3 ../common/simple_progress_trace_viewer.py $(ls logs/*.csv)

The following is a sequence diagram and the progress trace of the example, describing the transaction t1. The transaction deploys service configuration to the devices using a single RESTCONF patch request to NSO and then NSO configures the netsim devices using NETCONF:

RESTCONF   service   validate   push config
patch      create    config     ndtrans=2        netsim
ntrans=1   nwork=2   nwork=2    cqparam=bypass   device    ddelay=1
  t1 ------> 2s -----> 2s -----------------------> ex0 -----> 1s
                                    \------------> ex1 -----> 1s
  wall-clock 2s        2s                                     1s = 5s

The only part running concurrently in the example above was configuring the devices. It is the most straightforward option if transaction throughput performance is not a concern or the service creation and validation work are insignificant. A single transaction service deployment will not need to use commit queues as it is the only transaction holding the transaction lock configuring the devices inside the critical section. See the “holding transaction lock” event in the progress trace above.

Stop NSO and the netsim devices:

make stop

Concurrent Transactions

Everything from smartphones and tablets to laptops, desktops, and servers now contain multi-core processors. For maximal throughput, the powerful multi-core systems need to be fully utilized. This way, the wall clock time is minimized when deploying service configuration changes to the network, which is usually equated with performance. Therefore, enabling NSO to spread as much work as possible across all available cores becomes important. The goal is to have service deployments maximize their utilization of the total available CPU time to deploy services faster to the users who ordered them.

Close to full utilization of every CPU core when running under maximal load, for example, ten transactions to ten devices, is ideal, as some process viewer tools such as htop visualize with meters:

    0[|||||||||||||||||||||||||||||||||||||||||||||||||100.0%]
    1[|||||||||||||||||||||||||||||||||||||||||||||||||100.0%]
    2[||||||||||||||||||||||||||||||||||||||||||||||||||99.3%]
    3[||||||||||||||||||||||||||||||||||||||||||||||||||99.3%]
    4[||||||||||||||||||||||||||||||||||||||||||||||||||99.3%]
    5[||||||||||||||||||||||||||||||||||||||||||||||||||99.3%]
    6[||||||||||||||||||||||||||||||||||||||||||||||||||98.7%]
    7[||||||||||||||||||||||||||||||||||||||||||||||||||98.7%]
    8[||||||||||||||||||||||||||||||||||||||||||||||||||98.7%]
    9[||||||||||||||||||||||||||||||||||||||||||||||||||98.7%]
    ...

One transaction per RFS instance and device will allow each NSO transaction to run on a separate core concurrently. Multiple concurrent RESTCONF or NETCONF edits, CLI commits, MAAPI apply(), nano service re-deploy, etc. Keep the number of running concurrent transactions equal to or below the number of cores available in the multi-core processor to avoid performance degradation due to increased contention on system internals and resources. NSO helps by limiting the number of transactions applying changes in parallel to, by default, the number of logical processors (e.g., CPU cores). See ncs.conf(5) in Manual Pages under /ncs-config/transaction-limits/max-transactions for details.

Design to Minimize Conflicts

Conflicts between transactions and how to avoid them are described in Minimizing Concurrency Conflicts and in detail by the NSO Concurrency Model. While NSO can handle transaction conflicts gracefully with retries, retries affect transaction throughput performance. A simple but effective design pattern to avoid conflicts is to update one device with one Resource Facing Service (RFS) instance where service instances do not read each other's configuration changes.

Design to Minimize Service and Validation Processing Time

An overly complex service or validation implementation using templates, code, and XPath expressions increases the processing required and, even if transactions are processed concurrently, will affect the wall-clock time spent processing and, thus, transaction throughput.

When data processing performance is of interest, the best practice rule of thumb is to ensure that must and when statement XPath expressions in YANG models and service templates are only used as necessary and kept as simple as possible.

If a service creates a significant amount of configuration data for devices, it is often significantly faster using a single MAAPI shared_set_values() call instead of using multiple create() and set() calls or a service template.

Running the perf-setvals Example Using a Single Call to MAAPI shared_set_values()

The perf-setvals example writes configuration to an access control list and a route list of a Cisco Adaptive Security Appliance (ASA) device. It uses either MAAPI Python create() and set() calls, Python shared_set_values(), or Java sharedSetValues() to write the configuration in XML format.

To run the perf-setvals example using MAAPI Python create() and set() calls to create 3000 rules and 3000 routes on one device:

cd $NCS_DIR/examples.ncs/development-guide/concurrency-model/perf-setvals
./measure.sh -r 3000 -t py_create -n true

The commit uses the no-networking parameter to skip pushing the configuration to the simulated and un-proportionally slow Cisco ASA netsim device. The resulting NSO progress trace:

Next, run the perf-setvals example using a single MAAPI Python shared_set_values() call to create 3000 rules and 3000 routes on one device:

./measure.sh -r 3000 -t py_setvals_xml -n true

The resulting NSO progress trace:

Using the MAAPI shared_set_values() function, the service create callback is, for this example, ~5x faster than using the MAAPI create() and set() functions. The total wall-clock time for the transaction is more than 2x faster, and the difference will increase for larger transactions.

Stop NSO and the netsim devices:

make stop

Use a Data Kicker Instead of a CDB Subscriber

A kicker triggering on a CDB change, a data-kicker, should be used instead of a CDB subscriber when the action taken does not have to run inside the transaction lock, i.e., the critical section of the transaction. A CDB subscriber will be invoked inside the critical section and, thus, will have a negative impact on the transaction throughput. See Improving Subscribers for more details.

Shorten the Time Used for Writing Configuration to Devices

Writing to devices and other network elements that are slow to configure will stall transaction throughput if you do not enable commit queues, as transactions waiting for the transaction lock to be released cannot start configuring devices before the transaction ahead of them is done writing. For example, if one device is configured using CLI transported with IP over Avian Carriers, the transactions, including such a device, will significantly stall transactions behind it going to devices supporting RESTCONF or NETCONF over a fast optical transport. Where transaction throughput performance is a concern, choosing devices that can be configured efficiently to implement their part of the service configuration is wise.

Running the perf-trans Example Using One Transaction per Device

Dividing the service creation and validation work into two separate transactions, one per device, allows the work to be spread across two CPU cores in a multi-core processor. To run the perf-trans example with the work divided into one transaction per device:

cd $NCS_DIR/examples.ncs/development-guide/concurrency-model/perf-trans
make stop clean NDEVS=2 python
python3 measure.py --ntrans 2 --nwork 1 --ndtrans 1 --cqparam bypass --ddelay 1
python3 ../common/simple_progress_trace_viewer.py $(ls logs/*.csv)

The resulting NSO progress trace:

A sequence diagram with transactions t1 and t2 deploying service configuration to two devices using RESTCONF patch requests to NSO with NSO configuring the netsim devices using NETCONF:

RESTCONF   service   validate   push config
patch      create    config     ndtrans=1       netsim            netsim
ntrans=2   nwork=1   nwork=1    cqparam=bypass  device  ddelay=1  device  ddelay=1
  t1 ------> 1s -----> 1s ---------------------> ex0 ---> 1s
  t2 ------> 1s -----> 1s ---------------------------------------> ex1 ---> 1s
  wall-clock 1s        1s                                 1s                1s = 4s

Note how the service creation and validation work now is divided into 1s per transaction and runs concurrently on one CPU core each. However, the two transactions cannot push the configuration concurrently to a device each as the config push is done inside the critical section, making one of the transactions wait for the other to release the transaction lock. See the two “holding the transaction lock” events in the above progress trace visualization.

To enable transactions to push configuration to devices concurrently, we must enable commit queues.

Using Commit Queues

The concept of a network-wide transaction requires NSO to wait for the managed devices to process the configuration change before exiting the critical section, i.e., before NSO can release the transaction lock. In the meantime, other transactions have to wait their turn to write to CDB and the devices. The commit queue feature avoids waiting for configuration to be written to the device and increases the throughput. For most use cases, commit queues improve transaction throughput significantly.

Writing to a commit queue instead of the device moves the device configuration push outside of the critical region, and the transaction lock can instead be released when the change has been written to the commit queue.

For commit queue documentation, see Commit Queue.

Enabling Commit Queues for the perf-trans Example

Enabling commit queues allows the two transactions to spread the create, validation, and configuration push to devices work across CPU cores in a multi-core processor. Only the CDB write and commit queue write now remain inside the critical section, and the transaction lock is released as soon as the device configuration changes have been written to the commit queues instead of waiting for the config push to the devices to complete. To run the perf-trans example with the work divided into one transaction per device and commit queues enabled:

make stop clean NDEVS=2 python
python3 measure.py --ntrans 2 --nwork 1 --ndtrans 1 --cqparam sync --ddelay 1
python3 ../common/simple_progress_trace_viewer.py $(ls logs/*.csv)

The resulting NSO progress trace:

A sequence diagram with transactions t1 and t2 deploying service configuration to two devices using RESTCONF patch requests to NSO with NSO configuring the netsim devices using NETCONF:

RESTCONF   service   validate   push config
patch      create    config     ndtrans=1        netsim
ntrans=2   nwork=1   nwork=1    cqparam=sync     device    ddelay=1
  t1 ------> 1s -----> 1s --------------[----]---> ex0 -----> 1s
  t2 ------> 1s -----> 1s --------------[----]---> ex1 -----> 1s
  wall-clock 1s        1s                                     1s = 3s

Note how the two transactions now push the configuration concurrently to a device each as the config push is done outside of the critical section. See the two push configuration events in the above progress trace visualization.

Stop NSO and the netsim devices:

make stop

Running the perf-setvals example with two devices and commit queues enabled will produce a similar result.

Simplify the Per-Device Concurrent Transaction Creation Using a Nano Service

The perf-trans example service uses one transaction per service instance where each service instance configures one device. This enables transactions to run concurrently on separate CPU cores in a multi-core processor. The example sends RESTCONF patch requests concurrently to start transactions that run concurrently with the NSO transaction manager. However, dividing the work into multiple processes may not be practical for some applications using the NSO northbound interfaces, e.g., CLI or RESTCONF. Also, it makes a future migration to LSA more complex.

To simplify the NSO manager application, a resource-facing nano service (RFS) can start a process per service instance. The NSO manager application or user can then use a single transaction, e.g., CLI or RESTCONF, to configure multiple service instances where the NSO nano service divides the service instances into transactions running concurrently in separate processes.

The nano service can be straightforward, for example, using a single t3:configured state to invoke a service template or a create() callback. If validation code is required, it can run in a nano service post-action, t3:validated state, instead of a validation point callback to keep the validation code in the process created by the nano service.

See Nano Services for Staged Provisioning and Develop and Deploy a Nano Service for Nano service documentation.

Simplify Using a CFS

A Customer Facing Service (CFS) that is stacked with the RFS and maps to one RFS instance per device can simplify the service that is exposed to the NSO northbound interfaces so that a single NSO northbound interface transaction spawns multiple transactions, for example, one transaction per RFS instance when using the converge-on-re-deploy YANG extension with the nano service behavior tree.

Running the CFS and Nano Service enabled perf-stack Example

The perf-stack example showcases how a CFS on top of a simple resource-facing nano service can be implemented with the perf-trans example by modifying the existing t3 RFS and adding a CFS. Instead of multiple RESTCONF transactions, the example uses a single CLI CFS service commit that updates the desired number of service instances. The commit configures multiple service instances in a single transaction where the nano service runs each service instance in a separate process to allow multiple cores to be used concurrently.

Run as below to start two transactions with a 1-second CPU time workload per transaction in both the service and validation callbacks, each transaction pushing the device configuration to one device, each using a synchronous commit queue, where each device simulates taking 1 second to make the configuration changes to the device:

cd $NCS_DIR/examples.ncs/development-guide/concurrency-model/perf-stack
./showcase.sh -d 2 -t 2 -w 1 -r 1 -q 'True' -y 1

The above progress trace visualization is truncated to fit, but notice how the t3:validated state action callbacks, t3:configured state service creation callbacks and configuration push from the commit queues are running concurrently (on separate CPU cores) when initiating the service deployment with a single transaction started by the CLI commit.

A sequence diagram describing the transaction t1 deploying service configuration to the devices using the NSO CLI:

                                                              config
        CFS             validate  service  push config        change
CLI     create    Nano  config    create   ndtrans=1   netsim subscriber
commit  trans=2   RFS   nwork=1   nwork=1  cq=True     device ddelay=1
                  t1 --> 1s -----> 1s -------[----]---> ex0 ---> 1s
  t -----> t --->
                  t2 --> 1s -----> 1s -------[----]---> ex1 ---> 1s
              wall-clock 1s        1s                            1s=3s

The two transactions run concurrently, deploying the service in ~3 seconds (plus some overhead) of wall-clock time. Like the perf-trans example, you can play around with the perf-stack example by tweaking the parameters.

-d  NDEVS
    The number of netsim (ConfD) devices (network elements) started.
    Default 4

-t  NTRANS
    The number of transactions updating the same service in parallel.
    Default: $NDEVS

-w  NWORK
    Work per transaction in the service creation and validation phases. One
    second of CPU time per work item.
    Default: 3 seconds of CPU time.

-r  NDTRANS
    Number of devices the service will configure per service transaction.
    Default: 1

-c  USECQ
    Use device commit queues.
    Default: True

-y  DEV_DELAY
    Transaction delay (simulated by sleeping) on the netsim devices (seconds).
    Default: 1 second

See the README in the perf-stack example for details. For even more details, see the steps in the showcase script.

Stop NSO and the netsim devices:

make stop

Migrating to and Scale Up Using an LSA Setup

If the processor where NSO runs becomes a severe bottleneck, the CFS can migrate to a layered service architecture (LSA) setup. The perf-stack example implements stacked services, a CFS abstracting the RFS. It allows for easy migration to an LSA setup to scale with the number of devices or network elements participating in the service deployment. While adding complexity, LSA allows exposing a single CFS instance for all processors instead of one per processor.

Running the LSA-enabled perf-lsa Example

The perf-lsa example builds on the perf-stack example and showcases an LSA setup using two RFS NSO instances, lower-nso-1 and lower-nso-2, with a CFS NSO instance, upper-nso.

You can imagine adding more RFS NSO instances, lower-nso-3, lower-nso-4, etc., to the existing two as the number of devices increases. One NSO instance per multi-core processor and at least one CPU core per device (network element) is likely the most performant setup for this simulated work example. See LSA Overview in Layered Service Architecture for more.

As an example, a variant that starts four RFS transactions with a 1-second CPU time workload per transaction in both the service and validation callbacks, each RFS transaction pushing the device configuration to 1 device using synchronous commit queues, where each device simulates taking 1 second to make the configuration changes to the device:

cd $NCS_DIR/examples.ncs/development-guide/concurrency-model/perf-lsa
./showcase.sh -d 2 -t 2 -w 1 -r 1 -q 'True' -y 1

The three NSO progress trace visualizations show NSO on the CFS and the two RFS nodes. Notice how the CLI commit starts a transaction on the CFS node and configures four service instances with two transactions on each RFS node to push the resulting configuration to four devices.

A sequence diagram describing the transactions on RFS 1 t1 t2 and RFS 2 t1 t2. The transactions deploy service configuration to the devices using the NSO CLI:

                                                             config
       CFS             validate  service  push config        change
CLI    create    Nano  config    create   ndtrans=1   netsim subscriber
commit ntrans=2  RFS 1 nwork=1   nwork=1  cq=True     device ddelay=1
  t -----> t ---> t1 --> 1s -----> 1s -------[----]---> ex0 ---> 1s
            \     t2 --> 1s -----> 1s -------[----]---> ex1 ---> 1s
             \   RFS 2
              --> t1 --> 1s -----> 1s -------[----]---> ex2 ---> 1s
                  t2 --> 1s -----> 1s -------[----]---> ex3 ---> 1s
              wall-clock 1s        1s                            1s=3s

The four transactions run concurrently, two per RFS node, performing the work and configuring the four devices in ~3 seconds (plus some overhead) of wall-clock time.

You can play with the perf-lsa example by tweaking the parameters.

-d  LDEVS
    Number of netsim (ConfD) devices (network elements) started per RFS
    NSO instance.
    Default 2 (4 total)

-t  NTRANS
    Number of transactions updating the same service in parallel per RFS
    NSO instance. Here, one per device.
    Default: $LDEVS ($LDEVS * 2 total)

-w  NWORK
    Work per transaction in the service creation and validation phases. One
    second of CPU time per work item.
    Default: 3 seconds of CPU time.

-r  NDTRANS
    Number of devices the service will configure per service transaction.
    Default: 1

-q  USECQ
    Use device commit queues.
    Default: True

-y  DEV_DELAY
    Transaction delay (simulated by sleeping) on the netsim devices (seconds).
    Default: 1 second

See the README in the perf-lsa example for details. For even more details, see the steps in the showcase script.

Stop NSO and the netsim devices:

make stop

Scaling RAM and Disk

NSO contains an internal database called CDB, which stores both configuration and operational state data. Understanding the resource consumption of NSO at a steady state is mostly about understanding CDB, as it typically stands for the vast majority of resource usage.

CDB

Optimized for fast access, CDB is an in-memory database that holds all data in RAM. It also keeps the data on disk for persistence. The in-memory data structure is optimized for navigating tree data but is still a compact and efficient memory structure. The on-disk format uses a log structure, making it fast to write and very compact.

The in-memory structure usually consumes 2 - 3x more than the size of the on-disk format. The on-disk log will grow as more changes are performed in the system. A periodic compaction process compacts the write log and reduces its size. Upon startup of NSO, the on-disk version of CDB will be read, and the in-memory structure will be recreated based on the log. A recently compacted CDB will thus start up faster.

By default, NSO automatically determines when to compact CDB. It is visible in the devel.log when CDB compaction takes place. Compaction may require significant time, during which write transactions cannot be performed. In certain use cases, it may be preferable to disable automatic compaction by CDB and instead trigger compaction manually according to the specific needs. See Compaction for more details.

Services and Devices in CDB

CDB is a YANG-modeled database. By writing a YANG model, it is possible to store any kind of data in NSO and access it via one of the northbound interfaces of NSO. From this perspective, a service or a device's configuration is like most other YANG-modeled data. The number of service instances in NSO in the steady state affects how much space the data consumes in RAM and on disk.

But keep in mind that services tend to be modified from time to time, and with a higher total number of service instances, changes to those services are more likely. A higher number of service instances means more transactions to deploy changes, which means an increased need for optimizing transactional throughput, available CPU processing, RAM, and disk. See Designing for Maximal Transaction Throughput for details.

CDB Stores the YANG Model Schema

In addition to storing instance data, CDB also stores the schema (the YANG models), on disk and reads it into memory on startup. Having a large schema (many or large YANG models) loaded means both disk and RAM will be used, even when starting up an "empty" NSO, i.e., no instance data is stored in CDB.

In particular, device YANG models can be of considerable size. For example, the YANG models in recent versions of Cisco IOS XR have over 750,000 lines. Loading one such NED will consume about 1GB GB of RAM and slightly less disk space. In a mixed vendor network, you would load NEDs for all or some of these device types. With CDM, you can have multiple XR NEDs loaded to support communicating with different versions of XR and similarly for other devices, further consuming resources.

In comparison, most CLI NEDs only model a subset of a device and, are as a result, much smaller, most often under 100,000 lines of YANG.

For small NSO systems, the schema will usually consume more resources than the instance data, and NEDs, in particular, are the most significant contributors to resource consumption. As the system grows and more service and device configurations are added, the percentage of the total resource usage used for NED YANG models will decrease.

Note that the schema is memory mapped into shared memory, so even though multiple Python VMs might be started, memory usage will not increase as it shares memory between different clients. The Java VM uses its own copy of the schema, which is also why we can see that the JVM memory consumption follows the size of the loaded YANG schema.

The Size of CDB

Accurately predicting the size of CDB means accurately modeling its internal data structure. Since the result will depend on the YANG models and what actual values are stored in the database, the easiest way to understand of how the size grows is to start NSO with the schema and data in question and then measure the resource usage.

Performing accurate measurements can be a tedious process or sometimes impossible. When impossible, an estimate can be reached by extrapolating from known data, which is usually much more manageable and accurate enough.

We can look at the disk and RAM used for the running datastore, which stores configuration. On a freshly started NSO, it doesn't occupy much space at all:

# show ncs-state internal cdb datastore running | select ram-size | select disk-size
         DISK
NAME     SIZE      RAM SIZE
------------------------------
running  3.83 KiB  26.27 KiB

Devices, Small and Large

Adding a device with a small configuration, in this case, a Cisco NXOS switch with about 700 lines of CLI configuration, there is a clear increase:

# show ncs-state internal cdb datastore running | select ram-size | select disk-size
NAME     DISK SIZE  RAM SIZE
--------------------------------
running  28.51 KiB  240.99 KiB

Compared to the size of CDB before we added the device, we can deduce that the device with its configuration takes up ~214 kB in RAM and 25 kB on disk. Adding 1000 such devices, we see how CDB resource consumption increases linearly with more devices. This graph shows the RAM and memory usage of the running datastore in CDB over time. We perform a sequential sync-from operation on the 1000 devices, and while it is executing, we see how resource consumption increases. At the end, resource consumption has reached about 150 MB of RAM and 25 MB of disk, equating to ~150 KiB of RAM and ~25 KiB of disk per device.

# request devices device * sync-from

A device with a larger configuration will consume more space. With a single Juniper MX device that has a configuration with close to half a million lines of configuration, there's a substantial increase:

# show ncs-state internal cdb datastore running | select ram-size | select disk-size
NAME     DISK SIZE  RAM SIZE
--------------------------------
running  4.59 MiB  33.97 MiB

Similarly, adding more such devices allows monitoring of how it scales linearly. In the end, with 100 devices, CDB consumes 3.35 GB of RAM and 450 MB of disk, or ~33.5 MiB of RAM and ~4.5 MiB disk space per device.

Thus, you must do more than dimension your NSO installation based on the number of devices. You must also understand roughly how much resources each device will consume.

Unless a device uses NETCONF, NSO will not store the configuration as retrieved from the device. When configuration is retrieved, it is parsed by the NED into a structured format.

For example, here is a basic BGP stanza from a Cisco IOS device:

router bgp 64512
address-family ipv4 vrf TEST
no synchronization
redistribute connected metric 123 route-map IPV4-REDISTRIBUTE-CONNECTED-TO-BGP
!

After being parsed by the IOS CLI NED, the equivalent configuration looks like this in NSO:

<router xmlns="urn:ios">
    <bgp>
    <as-no>64512</as-no>
    <address-family>
        <with-vrf>
        <ipv4>
            <af>unicast</af>
            <vrf>
            <name>TEST</name>
            <redistribute>
                <connected>
                    <metric>123</metric>
                    <route-map>IPV4-REDISTRIBUTE-CONNECTED-TO-BGP</route-map>
                </connected>
                <static/>
            </redistribute>
            </vrf>
        </ipv4>
        </with-vrf>
    </address-family>
    </bgp>
</router>

A single line, such as redistribute connected metric 123 route-map IPV4-REDISTRIBUTE-CONNECTED-TO-BGP , is parsed into a structure of multiple nodes / YANG leaves. There is no exact correlation between the number of lines of configuration with the space it consumes in NSO. The easiest way to determine the resource consumption of a device's configuration is thus to load it into NSO and check the size of CDB before and after.

Planning Resource Consumption

Forming a rough estimate of CDB resource consumption for planning can be helpful.

Divide your devices into categories. Get a rough measurement for an exemplar in each category, add a safety margin, e.g., double the resource consumption, and multiply by the number of devices in that category. Example:

The Size of a Service

A YANG model describes the input to services, and just like any other data in CDB, it consumes resources. Compared to the typical device configuration, where even small devices often have a few hundred lines of configuration, a small service might only have a handful of configurable inputs. Even extensive services rarely have more than 50 inputs.

When services write configuration, a reverse diff set is generated and saved as part of the service's private data. The more configuration a service writes, the larger its reverse diff set will be and, thus, the more resources it will consume. What appears as a small service with just a handful of inputs could consume considerable resources if it writes a lot of configuration. Similarly, we save a forward diff set by default, contributing to the size. Service metadata attributes, the back pointer list, and the recount are also added to the written configuration, which consumes some resources. For example, if 50 services all (share)create a node, there will be 50 backpointers in the database, which consumes some space.

Implications of a Large CDB

As shown above, CDB scales linearly. Modern servers commonly support multiple terabytes of RAM, making it possible to support 50,000 - 100,000 such large router devices in NSO, well beyond the size of any currently existing network. However, beyond consuming RAM and disk space, the size of the CDB also affects the startup time of NSO and certain other operations like upgrades. In the previous example, 100 devices were used, which resulted in a CDB size of 461 MB on disk. Starting that on a standard laptop takes about 100 seconds. With 50,000 devices, CDB on-disk would be over 230 GB, which would take around 6 hours to load on the same laptop, if it had enough RAM. The typical server is considerably faster than the average laptop here, but loading a large CDB will take considerable time.

This also affects the sync/resync time in high availability setups, where the database size increases the data transfer needed.

A working system needs more than just storing the data. It must also be possible to use the devices and services and apply the necessary operations to these for the environment in which they operate. For example, it is common in brownfield environments to frequently run the sync-from action. Most device-related operations, including sync-from, can run concurrently across multiple devices in NSO. Syncing an extensive device configuration will take a few minutes or so. With 50,000 such large devices, we are looking at a total time of tens of hours or even days. Many environments require higher throughput, which could be handled using an LSA setup and spreading the devices over many NSO RFS nodes. sync-from is an example of an action that is easy to scale up and runs concurrently. For example, spreading the 50,000 devices over 5 NSO RFS nodes, each with 10,000 devices, would lead to a speedup close to 5x.

Using LSA, multiple Resource Facing Service (RFS) nodes can be employed to spread the devices across multiple NSO instances. This allows increasing the parallelism in sync-from and other operations, as described in Designing for Maximal Transaction Throughput, making it possible to scale to an almost arbitrary number of devices. Similarly, the services associated with each device are also spread across the RFS nodes, making it possible to operate on them in parallel. Finally, a top CFS node communicates with all RFS nodes, making it possible to administrate the entire setup as one extensive system.

Checklists

For smooth operation of NSO instances consider all of the following:

• Ensure there is enough RAM for NSO to run, with ample headroom.

• create() should normally run in a few hundred milliseconds, perhaps a few seconds for extensive services.

  • Consider splitting into smaller services.

  • Stacked services allow the composition of many smaller services into a larger service. A common best-practice design pattern is to have one Resource Facing Service (RFS) instance map to one device or network element.

    • Avoid conflicts between service instances.

    • Improves performance compared to a single large service for typical modifications.

    • Only services with changed input will have their create() called.

    • A small change to the Customer Facing Service (CFS) that results in changes to a subset of the lower services avoids running create() for all lower services.

• No external calls or sync-from in create() code.

  • Use nano-services to do external calls asynchronously.

  • Never run sync-from from create() code.

• Carefully consider the complexity of XPath constraints, in particular around lists.

  • Avoid XPath expressions with linear scaling or worse.

    • For example, avoid checking something for every element in a list, as performance will drop radically as the list grows.

    • XPath expressions involving nested lists or comparisons between lists can lead to quadratic scaling.

• Make sure you have an efficient transaction ID method for NEDs.

  • In the worst case, the NED will compute the transaction ID based on a config hash, which means it will fetch the entire config to compute the transaction ID.

• Enable commit queues and ensure transactions utilize as many CPU cores in a multi-core system as possible to increase transactional throughput.

• Ensure there are enough file descriptors available.

  • In many Linux systems, the default limit is 1024.

  • If we, for example, assume that there are 4 northbound interface ports, CLI, RESTCONF, SNMP, JSON-RPC, or similar, plus a few hundred IPC ports, x 1024 == 5120. But one might as well use the next power of two, 8192, to be on the safe side.

• See Disable Memory Overcommit.