This section describes some recipes, tools, and other resources that you may find useful throughout development. The topics are tailored to novice users and focus on making development with NSO a more enjoyable experience.

Development NSO Instance

Many developers prefer their own, dedicated NSO instance to avoid their work clashing with other team members. You can use either a local or remote Linux machine (such as a VM), or a macOS computer for this purpose.

The advantage of running local Linux with a GUI or macOS is that it is easier to set up the Integrated Development Environment (IDE) and other tools when they run on the same system as NSO. However, many IDEs today also allow working remotely, such as through the SSH protocol, making the choice of local versus remote less of a concern.

For development, using the so-called Local Install of NSO has some distinct advantages:

• It does not require elevated privileges to install or run.

• It keeps all NSO files in the same place (user-defined).

• It allows you to quickly switch between projects and NSO versions.

If you work with multiple projects in parallel, local install also allows you to take advantage of Python virtual environments to separate Python packages per project; simply start the NSO instance in an environment you have activated.

The main downside of using a local install is that it differs slightly from a system (production) install, such as in the filesystem paths used and the out-of-the-box configuration.

See Local Install for installation instructions.

Examples and Showcases

There are a number of examples and showcases in this guide. We encourage you to follow them through. They are also a great reference if you are experimenting with a new feature and have trouble getting it to work; you can inspect and compare with the implementation in the example.

To run the examples, you will need access to an NSO instance. A development instance described in this chapter is the perfect option for running locally. See Running NSO Examples.

IDE

Modern IDEs offer many features on top of advanced file editing support, such as code highlighting, syntax checks, and integrated debugging. While the initial setup takes some effort, the benefits of using an IDE are immense.

Visual Studio Code (VS Code) is a freely available and extensible IDE. You can add support for Java, Python, and YANG languages, as well as remote access through SSH via VS Code extensions. Consider installing the following extensions:

• Python by Microsoft: Adds Python support.

• Language Support for Java(TM) by Red Hat: Adds Java support.

• NSO Developer Studio by Cisco: Adds NSO-specific features as described in NSO Developer Studio.

• Remote - SSH by Microsoft: Adds support for remote development.

The Remote - SSH extension is especially useful when you must work with a system through an SSH session. Once you connect to the remote host by clicking the >< button (typically found in the bottom-left corner of the VS Code window), you can open and edit remote files with ease. If you also want language support (syntax highlighting and alike), you may need to install VS Code extensions remotely. That is, install the extensions after you have connected to the remote host, otherwise the extension installation screen might not show the option for installation on the connected host.

You will also benefit greatly from setting up SSH certificate authentication if you are using an SSH session for your work.

Automating Instance Setup

Once you get familiar with NSO development and gain some experience, a single NSO instance is likely to be insufficient; either because you need instances for unit testing, because you need one-off (throwaway) instances for an experiment, or something else entirely.

NSO includes tooling to help you quickly set up new local instances when such a need arises.

The following recipe relies on the ncs-setup command, which is available in the local install variant and requires a correctly set up shell environment (e.g. running source ncsrc). See Local Install for details.

A new instance typically needs a few things to be useful:

• Packages

• Initial data

• Devices to manage

In its simplest form, the ncs-setup invocation requires only a destination directory. However, you can specify additional packages to use with the --package option. Use the option to add as many packages as you need.

Running ncs-setup creates the required filesystem structure for an NSO instance. If you wish to include initial configuration data, put the XML-encoded data in the ncs-cdb subdirectory and NSO will load it at the first start, as described in Initialization Files.

NSO also needs to know about the managed devices. In case you are using ncs-netsim simulated devices (described in Network Simulator), you can use the --netsim-dir option with ncs-setup to add them directly. Otherwise, you may need to create some initial XML files with the relevant device configuration data — much like how you would add a device to NSO manually.

Most of the time, you must also invoke a sync with the device so that it performs correctly with NSO. If you wish to push some initial configuration to the device, you may add the configuration in the form of initial XML data and perform a sync-to. Alternatively, you can simply do a sync-from. You can use the ncs_cmd command for this purpose.

Combining all of this together, consider the following example:

1. Start by creating a new directory to hold the files:

   Copy

   $ mkdir nso-throwaway
   $ cd nso-throwaway

2. Create and start a few simulated devices with ncs-netsim, using ./netsim as directory:

   Copy

   $ ncs-netsim ncs-netsim create-network $NCS_DIR/packages/neds/cisco-ios-cli-3.8 3 c
   DEVICE c0 CREATED
   DEVICE c1 CREATED
   DEVICE c2 CREATED
   $ ncs-netsim start

3. Next, create the running directory with the NED package for the simulated devices and one more package. Also, add configuration data to NSO on how to connect to these simulated devices.

   Copy

       $ ncs-setup --dest ncs-run --netsim-dir ./netsim \
           --package $NCS_DIR/packages/neds/cisco-ios-cli-3.8 \
           --package $NCS_DIR/packages/neds/cisco-iosxr-cli-3.0

4. Now you can add custom initial data as XML files to ncs-run/ncs-cdb/. Usually, you would use existing files but you can also create them on-the-fly.

   Copy

   $ cat >ncs-run/ncs-cdb/my_init.xml <<'EOF'
   <config xmlns="http://tail-f.com/ns/config/1.0">
     <session xmlns="http://tail-f.com/ns/aaa/1.1">
       <idle-timeout>0</idle-timeout>
     </session>
   </config>
   EOF

5. At this point, you are ready to start NSO:

   Copy

   $ cd ncs-run
   $ ncs

6. Finally, request an initial sync-from:

   Copy

   $ ncs_cmd -u admin -c 'maction /devices/sync-from'
   sync-result begin
     device c0
     result true
   sync-result end
   sync-result begin
     device c1
     result true
   sync-result end
   sync-result begin
     device c2
     result true
   sync-result end

7. The instance is now ready for work. Once you are finished, you can stop it with ncs --stop. Remember to also stop the simulated devices with ncs-netsim stop if you no longer need them. Then, delete the containing folder (nso-throwaway) to remove all the leftover files and data.

NSO Developer Studio provides an integrated framework for developing NSO services using Visual Studio (VS) Code extensions. The extensions come with a core feature set to help you create services and connect to running CDB instances from within the VS Code environment. The following extensions are available as part of the NSO Developer Studio:

• NSO Developer Studio - Developer: Used for creating NSO services. Also referred to as NSO Developer extension in this guide.

• NSO Developer Studio - Explorer: Used for connecting to and inspecting NSO instance. Also referred to as NSO Explorer extension in this guide.

NSO Developer Studio - Developer Extension

This section describes the installation and functionality of the NSO Developer extension.

The purpose of the NSO Developer extension is to provide a base framework for developers to create their own NSO services. The focus of this guide is to manifest the creation of a simple NSO service package using the NSO Developer extension. At this time, reactive FastMAP and Nano services are not supported with this extension.

In terms of an NSO package, the extension supports YANG, XML, and Python to bring together various elements required to create a simple service.

After the installation, you can use the extension to create services and perform additional functions described below.

System Requirements

To get started with development using the NSO Developer extension, ensure that the following prerequisites are met on your system. The prerequisites are not a requirement to install the NSO Developer extension, but for NSO development after the extension is installed.

• Visual Studio Code.

• Java JDK 11 or higher.

• Python 3.9 or higher (recommended).

Install the Extension

Installation of the NSO Developer extension is done via the VS Code marketplace.

To install the NSO Developer extension in your VS Code environment:

1. Open VS Code and click the Extensions icon on the Activity Bar.

2. Search for the extension using the keywords "nso developer studio" in the Search Extensions in Marketplace field.

3. In the search results, locate the extension (NSO Developer Studio - Developer) and click Install.

4. Wait while the installation completes. A notification at the bottom-right corner indicates that the installation has finished. After the installation, an NSO icon is added to the Activity Bar.

Make a New Service Package (Python only)

Use the Make Package command in VS Code to create a new Python package. The purpose of this command is to provide functionality similar to the ncs-make-package CLI command, that is, to create a basic structure for you to start developing a new Python service package. The ncs-make-package command, however, comes with several additional options to create a package.

To make a new Python service package:

1. In the VS Code menu, go to View, and choose Command Palette.

2. In the Command Palette, type or pick the command NSO: Make Package. This brings up the Make Package dialog where you can configure package details.

3. In the Make Package dialog, specify the following package details:

   • Package Name: Name of the package.

   • Package Location: Destination folder where the package is to be created.

   • Namespace: Namespace of the YANG module, e.g. http://www.cisco.com/myModule.

   • Prefix: The prefix to be given to the YANG module, e.g. msp.

   • Yang Version: The YANG version that this module follows.

4. Click Create Package. This creates the required package and opens up a new instance of VS Code with the newly created NSO package.

5. If the Workspace Trust dialog is shown, click Yes, I Trust the Authors.

Open an Existing Package

Use the Open Existing Package command to open an already existing package.

To open an existing package:

1. In the VS Code menu, go to View, then choose Command Palette.

2. In the Command Palette, type or pick the command NSO: Open Existing Package.

3. Browse for the package on your local disk and open it. This brings up a new instance of VS Code and opens the package in it.

Edit YANG files

Opening a YANG file for edit results in VS Code detecting syntax errors in the YANG file. The errors show up due to missing path to YANG files and can be resolved using the following procedure.

Add YANG models for Yangster

For YANG support, a third-party extension called Yangster is used. Yangster is able to resolve imports for core NSO models but requires additional configuration.

To add YANG models for Yangster:

1. Create a new file named yang.settings by right-clicking in the blank area of the Explorer view and choosing New File from the pop-up.

2. Locate the NSO source YANG files on your local disk and copy the path.

3. In the file yang.settings, enter the path in the JSON format: { "yangPath": "<path to Yang files >" }, for example, { "yangPath": /home/my-user-name/nso-6.0/src/ncs/yang}. On Microsoft Windows, make sure that the backslash (\) is escaped, e.g., "C:\\user\\folder\\src\\yang".

4. Save the file.

5. Wait while the Yangster extension indexes and parses the YANG file to resolve NSO imports. After the parsing is finished, errors in the YANG file will disappear.

View YANG Diagram

YANG diagram is a feature provided by the Yangster extension.

To view the YANG diagram:

1. Update the YANG file. (Pressing Ctrl+space brings up auto-completion where applicable.)

2. Right-click anywhere in the VS Code Editor area and select Open in Diagram in the pop-up.

Add a New YANG Module

To add a new YANG module:

1. In the Explorer view, navigate to the yang folder and select it.

2. Right-click on the yang folder and select NSO: Add Yang Module from the pop-up menu. This brings up the Create Yang Module dialog where you can configure module details.

3. In the Create Yang Module dialog, fill in the following details:

   • Module Name: Name of the module.

   • Namespace: Namespace of the module, e.g., http://www.cisco.com/myModule.

   • Prefix: Prefix for the YANG module.

   • Yang Version: Version of YANG for this module.

4. Click Finish. This creates and opens up the newly created module.

Add a Service Point

Often while working on a package, there is a requirement to create a new service. This usually involves adding a service point. Adding a service point also requires other parts of the files to be updated, for example, Python.

Service points are usually added to lists.

To add a service point:

1. Update your YANG model as required. The extension automatically detects the list elements and displays a CodeLens called Add Service Point. An example is shown below.

   Copy

     container users {
       list user {
         key "name";
         description
           "This is a list of users in the system.";
       leaf name {
         type string;
         }
       leaf type {
         type string;
         }
       leaf full-name {
         type string;
         }
       }
     }

2. Click the Add Service Point CodeLens. This brings up the Add Service Point dialog.

3. Fill in the Service Point ID that is used to identify the service point, for example, mySimpleService.

4. Next, in the Python Details section, select using the Python Module field if you want to create a new Python module or use an existing one.

   • If you opt to create a new Python file, relevant sections are automatically updated in package-meta-data.xml.

   • If you select an existing Python module from the list, it is assumed that you are selecting the correct module and that, it has been created correctly, i.e., the package-meta-data.xml file is updated with the component definition.

5. Enter the Service CB Class, for example, SimpleServiceCB.

6. Finish creating the service by clicking Add Service Point.

Register an Action Point

All action points in a YANG model must be registered in NSO. Registering an action point also requires other parts of the files to be updated, for example, Python (register_action), and update package-meta-data if needed.

Action points are usually defined to lists or containers.

To register an action point:

1. Update your YANG model as required. The extension automatically detects the action point elements in YANG and displays a CodeLens called Add Action Point. An example is shown below.

   Copy

     ...
     container server {
         tailf:action ping {
         tailf:actionpoint pingaction;
           input {
             leaf destination {
               type inet:ip-address;
             }
           }
           output {
             leaf packet-loss {
               type uint8;
             }
           }
         }
       }

   Note that it is mandatory to specify tailf:actionpoint <actionpointname> under tailf:action <actionname>. This is a known limitation.

   The action point CodeLens at this time only works for the tailf:action statement, and not for the YANG rpc or YANG 1.1 action statements.

2. Click the Add Action Point CodeLens. This brings up the Register Action Point dialog.

3. Next, in the Python Details section, select using the Python Module field if you want to create a new Python module or use an existing one.

   • If you opt to create a new Python file, relevant sections are automatically updated in package-meta-data.xml.

   • If you select an existing Python module from the list, it is assumed that you are selecting the correct module, and that it has been created correctly, i.e., the package-meta-data.xml file is updated with the component definition.

4. Enter the action class name in the Main Class name used as entry point field, for example, MyAction.

5. Finish by clicking Register Action Point.

Edit Python Files

Opening a Python file uses the Microsoft Pylance extension. This extension provides syntax highlighting and other features such as code completion.

Add a New Python Module

To add a new Python module:

1. In the Primary Sidebar, Explorer view, right-click on the python folder.

2. Select NSO: Add Python Module from the pop-up. This brings up the Create Python Module dialog.

3. In the Create Python Module dialog, fill in the following details:

   • Module Name: Name of the module, for example, MyServicePackage.service.

   • Component Name: Name of the component that will be used to identify this module, for example, service.

   • Class Name: Name of the class to be invoked, for example, Main.

4. Click Finish.

Use Python Code Completion Snippets

Pre-defined snippets in VS Code allow for NSO Python code completion.

To use a Python code completion snippet:

1. Open a Python file for editing.

2. Type in one of the following pre-defined texts to display snippet options:

   • maapi: to view options for creating a maapi write transaction.

   • ncs: to view options for snippet for ncs template and variables.

3. Select a snippet from the pop-up to insert its code. This also highlights config items that can be changed. Press the Tab key to cycle through each value.

Edit XML Template Files

The final part of a typical service development is creating and editing the XML configuration template.

Add a New XML Template

To add a new XML template:

1. In the Primary Sidebar, Explorer view, right-click on the templates folder.

2. Select NSO: Add XML Template from the pop-up. This brings up the Add XML Template dialog.

3. In the Add XML Template dialog, fill in the XML Template name, for example, mspSimpleService.

4. Click Finish.

Use XML Code Completion Snippets

Pre-defined snippets in VS Code allow for NSO XML code completion of processing instructions and variables.

To use an XML code completion snippet:

1. Open an XML file for editing.

2. Type in one of the following pre-defined texts to display snippet options:

   • For processing instructions: <? followed by a character, for example <?i to view snippets for an if statement. All supported processing instructions are available as snippets.

   • For variables: $ followed by a character(s) matching the variable name, for example, $VA to view the variable snippet. Variables defined in the XML template via the <?set processing instruction or defined in Python code are displayed.

     Note: Auto-completion can also be triggered by pressing the Ctrl+Space keys.

3. Select an option from the pop-up to insert the relevant XML processing instruction or variable. Items that require further configuration are highlighted. Press the Tab key to cycle through the items.

XML Code Validation

The NSO Developer extension also performs code validation wherever possible. The following warning and error messages are shown if the extension is unable to validate the code:

• A warning is shown if a user enters a variable in an XML template that is not detected by the NSO Developer extension.

• An error message is shown if the ending tags in a processing instruction do not match.

Limitations

The extension provides help on a best-effort basis by showing error messages and warnings wherever possible. Still, in certain situations, code validation is not possible. An example of such a limitation is when the extension is not able to detect a template variable that is defined elsewhere and passed indirectly (i.e., the variable is not directly called).

Consider the following code for example, where the extension will successfully detect that a template variable IP_ADDRESS has been set.

vars.add('IP_ADDRESS','192.168.0.1')

Now consider the following code. While it serves the same purpose, it will fail to be detected.

ip_add_var_name = 'IP_ADDRESS' vars.add(ip_add_var_name, '192.168.0.1')

NSO Developer Studio - Explorer Extension

This section describes the installation and functionality of the NSO Explorer extension.

The purpose of the NSO Explorer extension is to allow the user to connect to a running instance of NSO and navigate the CDB from within VS Code.

System Requirements

To get started with the NSO Explorer extension, ensure that the following prerequisites are met on your system. The prerequisites are not a requirement to install the NSO Explorer extension, but for NSO development after the extension is installed.

• Visual Studio Code.

• Java JDK 11 or higher.

• Python 3.9 or higher (recommended).

Install the Extension

Installation of the NSO Explorer extension is done via the VS Code marketplace.

To install the NSO Explorer extension in your VS Code environment:

1. Open VS Code and click the Extensions icon on the Activity Bar.

2. Search for the extension using the keywords "nso developer studio" in the Search Extensions in Marketplace field.

3. In the search results, locate the extension (NSO Developer Studio - Explorer) and click Install.

4. Wait while the installation completes. A notification at the bottom-right corner indicates that the installation has finished. After the installation, an NSO icon is added to the Activity Bar.

Connect to NSO Instance

The NSO Explorer extension allows you to connect to and inspect a live NSO instance from within the VS Code. This procedure assumes that you have not previously connected to an NSO instance.

To connect to an NSO instance:

1. In the Activity Bar, click the NSO icon to open NSO Explorer.

2. If no NSO instance is already configured, a welcome screen is displayed with an option to add a new NSO instance.

3. Click the Add NSO Instance button to open the Settings editor.

4. In the Settings editor, click the link Edit in settings.json. This opens the settings.jsonfile for editing.

5. Next, edit the settings.json file as shown below:

   Copy

     "NSO.Instance": [
       {
         "host": "<hostname/ip>",
         "port": "<port>",
         "scheme": "http|https",
         "username": "<username>",
         "password": "<password>"
       }
     ]

6. Save the file when done.

   If settings have been configured correctly, NSO Explorer will attempt to connect to the running NSO instance and display the NSO configuration.

Inspect the CDB Tree

Once the NSO Explorer extension is configured, the user can inspect the CDB tree.

To inspect the CDB tree, use the following functions:

• Get Element Info: Click the i (info) icon on the Explorer bar, or alternatively inline next to an element in the Explorer view.

• Copy KeyPath: Click the {KP} icon to copy the keypath for the selected node.

• Copy XPath: Click the {XP} icon to copy the XPath for the selected node.

• Get XML Config: Click the XML icon to retrieve the XML configuration for the selected node and copy it to the clipboard.

If data has changed in NSO, click the refresh button at the top of the Explorer pane to fetch it.

Many services in NSO rely on NEDs to perform network provisioning. These services map service-specific configuration to the device data models, provided by the NEDs. As the NED packages can be upgraded independently, they can introduce changes in the device YANG models that cause issues for the services using them.

NSO provides tools to migrate between backward incompatible NED versions. The tools are designed to give you a structured analysis of which paths will change between two NED versions and visibility into the scope of the potential impact that a change in the NED will drive in the service code.

The tools allow for a usage-based analysis of which parts of the NED data model (and instance tree) a particular service has written to. This will give you an (at least opportunistic) sense of which paths must change in the service code.

These features aim to lower the barrier of upgrading NEDs and significantly reduce the amount of uncertainty and side effects that NED upgrades were historically associated with.

The migrate Action

By using the /ncs:devices/device/migrate action, you can change the NED major/minor version of a device. The action migrates all configuration and service meta-data. The action can also be executed in parallel on a device group or on all devices matching a NED identity. The procedure for migrating devices is further described in NED Migration.

Additionally, the example examples.ncs/getting-started/developing-with-ncs/26-ned-migration in the NSO examples collection illustrates how to migrate devices between different NED versions using the migrate action.

What makes it particularly useful to a service developer is that the action reports what paths have been modified and the service instances affected by those changes. This information can then be used to prepare the service code to handle the new NED version. If the verbose option is used, all service instances are reported instead of just the service points. If the dry-run option is used, the action simply reports what it would do. This gives you the chance to analyze before any actual change is performed.

Common Service Model

Each service type in NSO extends a part of the data model (a list or a container) with the ncs:servicepoint statement and the ncs:service-data grouping. This is what defines an NSO service.

The service point instructs NSO to involve the service machinery (Service Manager) for management of that part of the data tree and the ncs:service-data grouping contains definitions common to all services in NSO. Defined in tailf-ncs-services.yang, ncs:service-data includes parts that are required for the proper operation of FASTMAP and the Service Manager. Every service must therefore use this grouping as part of its data model.

In addition, ncs:service-data provides a common service interface to the users, consisting of:

While not part of ncs:service-data as such, you may consider the service-commit-queue-event notification part of the core service interface. The notification provides information about the state of the service when the service uses the commit queue. As an example, an event-driven application uses this notification to find out when a service instance has been deployed to the devices. See the showcase_rc.py script in examples.ncs/development-guide/concurrency-model/perf-stack/ for sample Python code, leveraging the notification. See tailf-ncs-services.yang for the full definition of the notification.

NSO Service Manager is responsible for providing the functionality of the common service interface, requiring no additional user code. This interface is the same for classic and nano services, whereas nano services further extend the model.

Services and Transactions

NSO calls into Service Manager when accessing actions and operational data under the common service interface, or when the service instance configuration data (the data under the service point) changes. NSO being a transactional system, configuration data changes happen in a transaction.

When applied, a transaction goes through multiple stages, as shown by the progress trace (e.g. using commit | details in the CLI). The detailed output breaks up the transaction into four distinct phases:

1. validation

2. write-start

3. prepare

4. commit

These phases deal with how the network-wide transactions work:

The validation phase prepares and validates the new configuration (including NSO copy of device configurations), then the CDB processes the changes and prepares them for local storage in the write-start phase.

The prepare stage sends out the changes to the network through the Device Manager and the HA system. The changes are staged (e.g. in the candidate data store) and validated if the device supports it, otherwise, the changes are activated immediately.

If all systems took the new configuration successfully, enter the commit phase, marking the new NSO configuration as active and activating or committing the staged configuration on remote devices. Otherwise, enter the abort phase, discarding changes, and ask NEDs to revert activated changes on devices that do not support transactions (e.g. without candidate data store).

There are also two types of locks involved with the transaction that are of interest to the service developer; the service write lock and the transaction lock. The latter is a global lock, required to serialize transactions, while the former is a per-service-type lock for serializing services that cannot be run in parallel. See Scaling and Performance Optimization for more details and their impact on performance.

The first phase, historically called validation, does more than just validate data and is the phase a service deals with the most. The other three support the NSO service framework but a service developer rarely interacts with directly.

We can further break down the first phase into the following stages:

1. rollback creation

2. pre-transform validation

3. transforms

4. full data validation

5. conflict check and transaction lock

When the transaction starts applying, NSO captures the initial intent and creates a rollback file, which allows one to reverse or roll back the intent. For example, the rollback file might contain the information that you changed a service instance parameter but it would not contain the service-produced device changes.

Then the first, partial validation takes place. It ensures the service input parameters are valid according to the service YANG model, so the service code can safely use provided parameter values.

Next, NSO runs transaction hooks and performs the necessary transforms, which alter the data before it is saved, for example encrypting passwords. This is also where the Service Manager invokes FASTMAP and service mapping callbacks, recording the resulting changes. NSO takes service write locks in this stage, too.

After transforms, there are no more changes to the configuration data, and the full validation starts, including YANG model constraints over the complete configuration, custom validation through validation points, and configuration policies (see Policies in Operation and Usage).

Throughout the phase, the transaction engine makes checkpoints, so it can restart the transaction faster in case of concurrency conflicts. The check for conflicts happens at the end of this first phase when NSO also takes the global transaction lock. Concurrency is further discussed in NSO Concurrency Model.

Service Callbacks

The main callback associated with a service point is the create callback, designed to produce the required (new) configuration, while FASTMAP takes care of the other operations, such as update and delete.

NSO implements two additional, optional callbacks for scenarios where create is insufficient. These are pre- and post-modification callbacks that NSO invokes before (pre) or after (post) create. These callbacks work outside of the scope tracked by FASTMAP. That is, changes done in pre- and post-modification do not automatically get removed during the update or delete of the service instance.

For example, you can use the pre-modification callback to check the service prerequisites (pre-check) or make changes that you want persisted even after the service is removed, such as enabling some global device feature. The latter may be required when NSO is not the only system managing the device and removing the feature configuration would break non-NSO managed services.

Similarly, you might use post-modification to reset the configuration to some default after the service is removed. Say the service configures an interface on a router for customer VPN. However, when the service is deprovisioned (removed), you don't want to simply erase the interface configuration. Instead, you want to put it in shutdown and configure it for a special, unused VLAN. The post-modification callback allows you to achieve this goal.

The main difference from create callback is that pre- and post-modification are called on update and delete, as well as service create. Since the service data node may no longer exist in case of delete, the API for these callbacks does not supply the service object. Instead, the callback receives the operation and key path to the service instance. See the following API signatures for details.

    @Service.pre_modification
    def cb_pre_modification(self, tctx, op, kp, root, proplist): ...

    @Service.create
    def cb_create(self, tctx, root, service, proplist): ...

    @Service.post_modification
    def cb_post_modification(self, tctx, op, kp, root, proplist): ...

The Python callbacks use the following function arguments:

• tctx: A TransCtxRef object containing transaction data, such as user session and transaction handle information.

• op: Integer representing operation: create (ncs.dp.NCS_SERVICE_CREATE), update (ncs.dp.NCS_SERVICE_UPDATE), or delete (ncs.dp.NCS_SERVICE_DELETE) of the service instance.

• kp: A HKeypathRef object with a key path of the affected service instance, such as /svc:my-service{instance1}.

• root: A Maagic node for the root of the data model.

• service: A Maagic node for the service instance.

• proplist: Opaque service properties, see Persistent Opaque Data.

    @ServiceCallback(servicePoint = "...",
                     callType = ServiceCBType.PRE_MODIFICATION)
    public Properties preModification(ServiceContext context,
                                      ServiceOperationType operation,
                                      ConfPath path,
                                      Properties opaque)
                                      throws DpCallbackException;

    @ServiceCallback(servicePoint="...",
                     callType=ServiceCBType.CREATE)
    public Properties create(ServiceContext context,
                             NavuNode service,
                             NavuNode ncsRoot,
                             Properties opaque)
                             throws DpCallbackException;

    @ServiceCallback(servicePoint = "...",
                     callType = ServiceCBType.POST_MODIFICATION)
    public Properties postModification(ServiceContext context,
                                       ServiceOperationType operation,
                                       ConfPath path,
                                       Properties opaque)
                                       throws DpCallbackException;

The Java callbacks use the following function arguments:

• context: A ServiceContext object for accessing root and service instance NavuNode in the current transaction.

• operation: ServiceOperationType enum representing operation: CREATE, UPDATE, DELETE of the service instance.

• path: A ConfPath object with a key path of the affected service instance, such as /svc:my-service{instance1}.

• ncsRoot: A NavuNode for the root of the ncs data model.

• service: A NavuNode for the service instance.

• opaque: Opaque service properties, see Persistent Opaque Data.

See examples.ncs/development-guide/services/post-modification-py and examples.ncs/development-guide/services/post-modification-java examples for a sample implementation of the post-modification callback.

Additionally, you may implement these callbacks with templates. Refer to Service Callpoints and Templates for details.

Persistent Opaque Data

FASTMAP greatly simplifies service code, so it usually only needs to deal with the initial mapping. NSO achieves this by first discarding all the configuration performed during the create callback of the previous run. In other words, the service create code always starts anew, with a blank slate.

If you need to keep some private service data across runs of the create callback, or pass data between callbacks, such as pre- and post-modification, you can use opaque properties.

The opaque object is available in the service callbacks as an argument, typically named proplist (Python) or opaque (Java). It contains a set of named properties with their corresponding values.

If you wish to use the opaque properties, it is crucial that your code returns the properties object from the create call, otherwise, the service machinery will not save the new version.

Compared to pre- and post-modification callbacks, which also persist data outside of FASTMAP, NSO deletes the opaque data when the service instance is deleted, unlike with the pre- and post-modification data.

    @Service.create
    def cb_create(self, tctx, root, service, proplist):
        intf = None
        # proplist is of type list[tuple[str, str]]
        for pname, pvalue in proplist:
            if pname == 'INTERFACE':
                intf = pvalue

        if intf is None:
            intf = '...'
            proplist.append('INTERFACE', intf)

        return proplist

    public Properties create(ServiceContext context,
                             NavuNode service,
                             NavuNode ncsRoot,
                             Properties opaque)
                             throws DpCallbackException {
        // In Java API, opaque is null when service instance is first created.
        if (opaque == null) {
            opaque = new Properties();
        }
        String intf = opaque.getProperty("INTERFACE");
        if (intf == null) {
            intf = "...";
            opaque.setProperty("INTERFACE", intf);
        }

        return opaque;
    }

The examples.ncs/development-guide/services/post-modification-py and examples.ncs/development-guide/services/post-modification-java examples showcase the use of opaque properties.

Defining Static Service Conflicts

NSO by default enables concurrent scheduling and execution of services to maximize throughput. However, concurrent execution can be problematic for non-thread-safe services or services that are known to always conflict with themselves or other services, such as when they read and write the same shared data. See NSO Concurrency Model for details.

To prevent NSO from scheduling a service instance together with an instance of another service, declare a static conflict in the service model, using the ncs:conflicts-with extension. The following example shows a service with two declared static conflicts, one with itself and one with another service, named other-service.

        list example-service {
          key name;
          leaf name {
            type string;
          }
          uses ncs:service-data;
          ncs:servicepoint example-service {
            ncs:conflicts-with example-service;
            ncs:conflicts-with other-service;
          }
        }

This means each service instance will wait for other service instances that have started sooner than this one (and are of example-service or other-service type) to finish before proceeding.

Reference Counting Overlapping Configuration

FASTMAP knows that a particular piece of configuration belongs to a service instance, allowing NSO to revert the change as needed. But what happens when several service instances share a resource that may or may not exist before the first service instance is created? If the service implementation naively checks for existence and creates the resource when it is missing, then the resource will be tracked with the first service instance only. If, later on, this first instance is removed, then the shared resource is also removed, affecting all other instances.

A well-known solution to this kind of problem is reference counting. NSO uses reference counting by default with the XML templates and Python Maagic API, while in Java Maapi and Navu APIs, the sharedCreate(), sharedSet(), and sharedSetValues() functions need to be used.

When enabled, the reference counter allows FASTMAP algorithm to keep track of the usage and only delete data when the last service instance referring to this data is removed.

Furthermore, containers and list items created using the sharedCreate() and sharedSetValues() functions also get an additional attribute called backpointer. (But this functionality is currently not available for individual leafs.)

backpointer points back to the service instance that created the entity in the first place. This makes it possible to look at part of the configuration, say under /devices tree, and answer the question: which parts of the device configuration were created by which service?

To see reference counting in action, start the examples.ncs/implement-a-service/iface-v3 example with make demo and configure a service instance.

admin@ncs(config)# iface instance1 device c1 interface 0/1 ip-address 10.1.2.3 cidr-netmask 28
admin@ncs(config)# commit

Then configure another service instance with the same parameters and use the display service-meta-data pipe to show the reference counts and backpointers:

admin@ncs(config)# iface instance2 device c1 interface 0/1 ip-address 10.1.2.3 cidr-netmask 28
admin@ncs(config)# commit dry-run
cli {
    local-node {
        data +iface instance2 {
             +    device c1;
             +    interface 0/1;
             +    ip-address 10.1.2.3;
             +    cidr-netmask 28;
             +}
    }
}
admin@ncs(config)# commit and-quit
admin@ncs# show running-config devices device c1 config interface\
 GigabitEthernet 0/1 | display service-meta-data
devices device c1
 config
  ! Refcount: 2
  ! Backpointer: [ /iface:iface[iface:name='instance1'] /iface:iface[iface:name='instance2'] ]
  interface GigabitEthernet0/1
   ! Refcount: 2
   ip address 10.1.2.3 255.255.255.240
   ! Refcount: 2
   ! Backpointer: [ /iface:iface[iface:name='instance1'] /iface:iface[iface:name='instance2'] ]
   ip dhcp snooping trust
  exit
 !
!

Notice how commit dry-run produces no new device configuration but the system still tracks the changes. If you wish, remove the first instance and verify the GigabitEthernet 0/1 configuration is still there, but is gone when you also remove the second one.

But what happens if the two services produce different configurations for the same node? Say, one sets the IP address to 10.1.2.3 and the other to 10.1.2.4. Conceptually, these two services are incompatible, and instantiating both at the same time produces a broken configuration (instantiating the second service instance breaks the configuration for the first). What is worse is that the current configuration depends on the order the services were deployed or re-deployed. For example, re-deploying the first service will change the configuration from 10.1.2.4 back to 10.1.2.3 and vice versa. Such inconsistencies break the declarative configuration model and really should be avoided.

In practice, however, NSO does not prevent services from producing such configuration. But note that we strongly recommend against it and that there are associated limitations, such as service un-deploy not reverting configuration to that produced by the other instance (but when all services are removed, the original configuration is still restored).

The commit | debug service pipe command warns about any such conflict that it finds but may miss conflicts on individual leafs. The best practice is to use integration tests in the service development life cycle to ensure there are no conflicts, especially when multiple teams develop their own set of services that are to be deployed on the same NSO instance.

Stacked Services

Much like a service in NSO can provision device configurations, it can also provision other, non-device data, as well as other services. We call the approach of services provisioning other services 'service stacking' and the services that are involved — 'stacked'.

Service stacking concepts usually come into play for bigger, more complex services. There are a number of reasons why you might prefer stacked services to a single monolithic one:

• Smaller, more manageable services with simpler logic.

• Separation of concerns and responsibility.

• Clearer ownership across teams for (parts of) overall service.

• Smaller services reusable as components across the solution.

• Avoiding overlapping configuration between service instances causing conflicts, such as using one service instance per device (see examples in Designing for Maximal Transaction Throughput).

Stacked services are also the basis for LSA, which takes this concept even further. See Layered Service Architecture for details.

The standard naming convention with stacked services distinguishes between a Resource-Facing Service (RFS), that directly configures one or more devices, and a Customer-Facing Service (CFS), that is the top-level service, configuring only other services, not devices. There can be more than two layers of services in the stack, too.

While NSO does not prevent a single service from configuring devices as well as services, in the majority of cases this results in a less clean design and is best avoided.

Overall, creating stacked services is very similar to the non-stacked approach. First, you can design the RFS services as usual. Actually, you might take existing services and reuse those. These then become your lower-level services, since they are lower in the stack.

Then you create a higher-level service, say a CFS, that configures another service, or a few, instead of a device. You can even use a template-only service to do that, such as:

<config-template xmlns="http://tail-f.com/ns/config/1.0"
                 servicepoint="top-level-service">
  <iface xmlns="http://com/example/iface">
    <name>instance1</name>
    <device>c1</device>
    <interface>0/1</interface>
    <ip-address>10.1.2.3</ip-address>
    <cidr-netmask>28</cidr-netmask>
  </iface>
</config>

The preceding example references an existing iface service, such as the one in the examples.ncs/implement-a-service/iface-v3 example. The output shows hard-coded values but you can change those as you would for any other service.

In practice, you might find it beneficial to modularize your data model and potentially reuse parts in both, the lower- and higher-level service. This avoids duplication while still allowing you to directly expose some of the lower-level service functionality through the higher-level model.

The most important principle to keep in mind is that the data created by any service is owned by that service, regardless of how the mapping is done (through code or templates). If the user deletes a service instance, FASTMAP will automatically delete whatever the service created, including any other services. Likewise, if the operator directly manipulates service data that is created by another service, the higher-level service becomes out of sync. The check-sync service action checks this for services as well as devices.

In stacked service design, the lower-level service data is under the control of the higher-level service and must not be directly manipulated. Only the higher-level service may manipulate that data. However, two higher-level services may manipulate the same structures, since NSO performs reference counting (see Reference Counting Overlapping Configuration).

Caveats and Best Practices

This section lists some specific advice for implementing services, as well as any known limitations you might run into.

You may also obtain some useful information by using the debug service commit pipe command, such as commit dry-run | debug service. The command display the net effect of the service create code, as well as issue warnings about potentially problematic usage of overlapping shared data.

• Service callbacks must be deterministic: NSO invokes service callbacks in a number of situations, such as for dry-run, check sync, and actual provisioning. If a service does not create the same configuration from the same inputs, NSO sees it as being out of sync, resulting in a lot of configuration churn and making it incompatible with many NSO features. If you need to introduce some randomness or rely on some other nondeterministic source of data, make sure to cache the values across callback invocations, such as by using opaque properties (see Persistent Opaque Data) or persistent operational data (see Operational Data) populated in a pre-modification callback.

• Never overwrite service inputs: Service input parameters capture client intent and a service should never change its own configuration. Such behavior not only muddles the intent but is also temporary when done in the create callback, as the changes are reverted on the next invocation.

  If you need to keep some additional data that cannot be easily computed each time, consider using opaque properties (see Persistent Opaque Data) or persistent operational data (see Operational Data) populated in a pre-modification callback.

• No service ordering in a transaction: NSO is a transactional system and as such does not have the concept of order inside a single transaction. That means NSO does not guarantee any specific order in which the service mapping code executes if the same transaction touches multiple service instances. Likewise, your code should not make any assumptions about running before or after other service code.

• Return value of create callback: The create callback is not the exclusive user of the opaque object; the object can be chained in several different callbacks, such as pre- and post-modification. Therefore, returning None/null from create callback is not a good practice. Instead, always return the opaque object even if the create callback does not use it.

• Avoid delete in service create: Unlike creation, deleting configuration does not support reference counting, as there is no data left to reference count. This means the deleted elements are tied to the service instance that deleted them.

  Additionally, FASTMAP must store the entire deleted tree and restore it on every service change or re-deploy, only to be deleted again. Depending on the amount of deleted data, this is potentially an expensive operation.

  So, a general rule of thumb is to never use delete in service create code. If an explicit delete is used, debug service may display the following warning:\

  Copy

  *** WARNING ***: delete in service create code is unsafe if data is
                   shared by other services

  However, the service may also delete data implicitly, through when and choice statements in the YANG data model. If a when statement evaluates to false, the configuration tree below that node is deleted. Likewise, if a case is set in a choice statement, the previously set case is deleted. This has the same limitations as an explicit delete.

  To avoid these issues, create a separate service, that only handles deletion, and use it in the main service through the stacked service design (see Stacked Services). This approach allows you to reference count the deletion operation and contains the effect of restoring deleted data through a small, rarely-changing helper service. See examples.ncs/development-guide/services/shared-delete for an example.

  Alternatively, you might consider pre- and post-modification callbacks for some specific cases.

• Prefer shared*() functions: Non-shared create and set operations in the Java and Python low-level API do not add reference counts or backpointer information to changed elements. In case there is overlap with another service, unwanted removal can occur. See Reference Counting Overlapping Configuration for details.

  In general, you should prefer sharedCreate(), sharedSet(), and sharedSetValues(). If non-shared variants are used in a shared context, service debug displays a warning, such as:\

  Copy

  *** WARNING ***: set in service create code is unsafe if data is
                   shared by other services

  Likewise, do not use MAAPI load_config variants from the service code. Use the sharedSetValues() function to load XML data from a file or a string.

• Reordering ordered-by-user lists: If the service code rearranges an ordered-by-user list with items that were created by another service, that other service becomes out of sync. In some cases, you might be able to avoid out-of-sync scenarios by leveraging special XML template syntax (see Operations on ordered lists and leaf-lists) or using service stacking with a helper service.

  In general, however, you should reconsider your design and try to avoid such scenarios.

• Automatic upgrade of keys for existing services is unsupported: Service backpointers, described in Reference Counting Overlapping Configuration, rely on the keys that the service model defines to identify individual service instances. If you update the model by adding, removing, or changing the type of leafs used in the service list key, while there are deployed service instances, the backpointers will not be automatically updated. Therefore, it is best to not change the service list key.

  A workaround, if the service key absolutely must change, is to first perform a no-networking undeploy of the affected service instances, then upgrade the model, and finally no-networking re-deploy the previously un-deployed services.

• Avoid conflicting intents: Consider that a service is executed as part of a transaction. If, in the same transaction, the service gets conflicting intents, for example, it gets modified and deleted, the transaction is aborted. You must decide which intent has higher priority and design your services to avoid such situations.

Service Discovery and Import

A very common situation, when NSO is deployed in an existing network, is that the network already has services implemented. These services may have been deployed manually or through an older provisioning system. To take full advantage of the new system, you should consider importing the existing services into NSO. The goal is to use NSO to manage existing service instances, along with adding new ones in the future.

The process of identifying services and importing them into NSO is called Service Discovery and can be broken down into the following high-level parts:

• Implementing the service to match existing device configuration.

• Enumerating service instances and their parameters.

• Amend the service metadata references with reconciliation.

Ultimately, the problem that service discovery addresses is one of referencing or linking configuration to services. Since the network already contains target configuration, a new service instance in NSO produces no changes in the network. This means the new service in NSO by default does not own the network configuration. One side effect is that removing a service will not remove the corresponding device configuration, which is likely to interfere with service modification as well.

Some of the steps in the process can be automated, while others are mostly manual. The amount of work differs a lot depending on how structured and consistent the original deployment is.

Matching Configuration

A prerequisite (or possibly the product in an iterative approach) is an NSO service that supports all the different variants of the configuration for the service that are used in the network. This usually means there will be a few additional parameters in the service model that allow selecting the variant of device configuration produced, as well as some covering other non-standard configurations (if such configuration is present).

In the simplest case, there is only one variant and that is the one that the service needs to produce. Let's take the examples.ncs/implement-a-service/iface-v2-py example and consider what happens when a device already has an existing interface configuration.

admin@ncs# show running-config devices device c1 config\
 interface GigabitEthernet 0/1
devices device c1
 config
  interface GigabitEthernet0/1
   ip address 10.1.2.3 255.255.255.240
  exit
 !
!

Configuring a new service instance does not produce any new device configuration (notice that device c1 has no changes).

admin@ncs(config)# commit dry-run
cli {
    local-node {
        data +iface instance1 {
             +    device c1;
             +    interface 0/1;
             +    ip-address 10.1.2.3;
             +    cidr-netmask 28;
             +}
    }
}

However, when committed, NSO records the changes, just like in the case of overlapping configuration (see Reference Counting Overlapping Configuration). The main difference is that there is only a single backpointer, to a newly configured service, but the refcount is 2. The other item, that contributes to the refcount, is the original device configuration. Which is why the configuration is not deleted when the service instance is.

admin@ncs# show running-config devices device c1 config interface\
 GigabitEthernet 0/1 | display service-meta-data
devices device c1
 config
  ! Refcount: 2
  ! Backpointer: [ /iface:iface[iface:name='instance1'] ]
  interface GigabitEthernet0/1
   ! Refcount: 2
   ! Originalvalue: 10.1.2.3
   ip address 10.1.2.3 255.255.255.240
  exit
 !
!

Enumerating Instances

A prerequisite for service discovery to work is that it is possible to construct a list of the already existing services. Such a list may exist in an inventory system, an external database, or perhaps just an Excel spreadsheet.

You can import the list of services in a number of ways. If you are reading it in from a spreadsheet, a Python script using NSO API directly (Basic Automation with Python) and a module to read Excel files is likely a good choice.

import ncs
from openpyxl import load_workbook

def main()
    wb = load_workbook('services.xslx')
    sheet = wb[wb.sheetnames[0]]

    with ncs.maapi.single_write_trans('admin', 'python') as t:
        root = ncs.maagic.get_root(t)
        for sr in sheet.rows:
            # Suppose columns in spreadsheet are:
            # instance (A), device (B), interface (C), IP (D), mask (E)
            name = sr[0].value
            service = root.iface.create(name)
            service.device = sr[1].value
            service.interface = sr[2].value
            service.ip_address = sr[3].value
            service.cidr_netmask = sr[4].value

        t.apply()

main()

Or, you might generate an XML data file to import using the ncs_load command; use display xml filter to help you create a template:

admin@ncs# show running-config iface | display xml
<config xmlns="http://tail-f.com/ns/config/1.0">
  <iface xmlns="http://com/example/iface">
    <name>instance1</name>
    <device>c1</device>
    <interface>0/1</interface>
    <ip-address>10.1.2.3</ip-address>
    <cidr-netmask>28</cidr-netmask>
  </iface>
</config>

Regardless of the way you implement the data import, you can run into two kinds of problems.

On one hand, the service list data may be incomplete. Suppose that the earliest service instances deployed did not take the network mask as a parameter. Moreover, for some specific reasons, a number of interfaces had to deviate from the default of 28 and that information was never populated back in the inventory for old services after the netmask parameter was added.

Now the only place where that information is still kept may be the actual device configuration. Fortunately, you can access it through NSO, which may allow you to extract the missing data automatically, for example:

devconfig = root.devices.device[service.device].config
intf = devconfig.interface.GigabitEthernet[service.interface]
netmask = intf.ip.address.primary.mask
cidr = IPv4Network(f'0.0.0.0/{netmask}').prefixlen

On the other hand, some parameters may be NSO specific, such as those controlling which variant of configuration to produce. Again, you might be able to use a script to find this information, or it could turn out that the configuration is too complex to make such a script feasible.

In general, this can be the most tricky part of the service discovery process, making it very hard to automate. It all comes down to how good the existing data is. Keep in mind that this exercise is typically also a cleanup exercise, and every network will be different.

Reconciliation

The last step is updating the metadata, telling NSO that a given service controls (owns) the device configuration that was already present when the NSO service was configured. This is called reconciliation and you achieve it using a special re-deploy reconcile action for the service.

Let's examine the effects of this action on the following data:

admin@ncs# show running-config devices device c1 config\
 interface GigabitEthernet 0/1 | display service-meta-data
devices device c1
 config
  ! Refcount: 2
  ! Backpointer: [ /iface:iface[iface:name='instance1'] ]
  interface GigabitEthernet0/1
   ! Refcount: 2
   ! Originalvalue: 10.1.2.3
   ip address 10.1.2.3 255.255.255.240
  exit
 !
!

Having run the action, NSO has updated the refcount to remove the reference to the original device configuration:

admin@ncs# iface instance1 re-deploy reconcile
admin@ncs# show running-config devices device c1 config\
 interface GigabitEthernet 0/1 | display service-meta-data
devices device c1
 config
  ! Refcount: 1
  ! Backpointer: [ /iface:iface[iface:name='instance1'] ]
  interface GigabitEthernet0/1
   ! Refcount: 1
   ip address 10.1.2.3 255.255.255.240
  exit
 !
!

What is more, the reconcile algorithm works even if multiple service instances share configuration. What if you had two instances of the iface service, instead of one?

Before reconciliation, the device configuration would show a refcount of three.

admin@ncs# show running-config devices device c1 config\
 interface GigabitEthernet 0/1 | display service-meta-data
devices device c1
 config
  ! Refcount: 3
  ! Backpointer: [ /iface:iface[iface:name='instance1'] /iface:iface[iface:name='instance2'] ]
  interface GigabitEthernet0/1
   ! Refcount: 3
   ! Originalvalue: 10.1.2.3
   ip address 10.1.2.3 255.255.255.240
  exit
 !
!

Invoking re-deploy reconcile on either one or both of the instances makes the services sole owners of the configuration.

admin@ncs# show running-config devices device c1 config\
 interface GigabitEthernet 0/1 | display service-meta-data
devices device c1
 config
  ! Refcount: 2
  ! Backpointer: [ /iface:iface[iface:name='instance1'] /iface:iface[iface:name='instance2'] ]
  interface GigabitEthernet0/1
   ! Refcount: 2
   ip address 10.1.2.3 255.255.255.240
  exit
 !
!

This means the device configuration is removed only when you remove both service instances.

admin@ncs(config)# no iface instance1
admin@ncs(config)# commit dry-run outformat native
native {
}
admin@ncs(config)# no iface instance2
admin@ncs(config)# commit dry-run outformat native
native {
    device {
        name c1
        data no interface GigabitEthernet0/1
    }
}

The reconcile operation only removes the references to the original configuration (without the service backpointer), so you can execute it as many times as you wish. Just note that it is part of a service re-deploy, with all the implications that brings, such as potentially deploying new configuration to devices when you change the service template.

As an alternative to the re-deploy reconcile, you can initially add the service configuration with a commit reconcile variant, performing reconciliation right away.

Iterative Approach

It is hard to design a service in one go when you wish to cover existing configurations that are exceedingly complex or have a lot of variance. In such cases, many prefer an iterative approach, where you tackle the problem piece-by-piece.

Suppose there are two variants of the service configured in the network; iface-v2-py and the newer iface-v3, which produces a slightly different configuration. This is a typical scenario when a different (non-NSO) automation system is used and the service gradually evolves over time. Or, when a Method of Procedure (MOP) is updated if manual provisioning is used.

We will tackle this scenario to show how you might perform service discovery in an iterative fashion. We shall start with the iface-v2-py as the first iteration of the iface service, which represents what configuration the service should produce to the best of our current knowledge.

There are configurations for two service instances in the network already: for interfaces 0/1 and 0/2 on the c1 device. So, configure the two corresponding iface instances.

admin@ncs(config)# commit dry-run
cli {
    local-node {
        data +iface instance1 {
             +    device c1;
             +    interface 0/1;
             +    ip-address 10.1.2.3;
             +    cidr-netmask 28;
             +}
             +iface instance2 {
             +    device c1;
             +    interface 0/2;
             +    ip-address 10.2.2.3;
             +    cidr-netmask 28;
             +}
    }
}
admin@ncs(config)# commit

You can also use the commit no-deploy variant to add service parameters when a normal commit would produce device changes, which you do not want.

Then use the re-deploy reconcile { discard-non-service-config } dry-run command to observe the difference between the service-produced configuration and the one present in the network.

admin@ncs# iface instance1 re-deploy reconcile\
 { discard-non-service-config } dry-run
cli {
}

For instance1, the config is the same, so you can safely reconcile it already.

admin@ncs# iface instance1 re-deploy reconcile

But interface 0/2 (instance2), which you suspect was initially provisioned with the newer version of the service, produces the following:

admin@ncs# iface instance2 re-deploy reconcile\
 { discard-non-service-config } dry-run
cli {
    local-node {
        data  devices {
                   device c1 {
                       config {
                           interface {
                               GigabitEthernet 0/2 {
                                   ip {
                                       dhcp {
                                           snooping {
              -                                trust;
                                           }
                                       }
                                   }
                               }
                           }
                       }
                   }
               }

    }
}

The output tells you that the service is missing the ip dhcp snooping trust part of the interface configuration. Since the service does not generate this part of the configuration yet, running re-deploy reconcile { discard-non-service-config } (without dry-run) would remove the DHCP trust setting. This is not what we want.

One option, and this is the default reconcile mode, would be to use keep-non-service-config instead of discard-non-service-config. But that would result in the service taking ownership of only part of the interface configuration (the IP address).

Instead, the right approach is to add the missing part to the service template. There is, however, a little problem. Adding the DHCP snooping trust configuration unconditionally to the template can interfere with the other service instance, instance1.

In some cases, upgrading the old configuration to the new variant is viable, but in most situations, you likely want to avoid all device configuration changes. For the latter case, you need to add another parameter to the service model that selects the configuration variant. You must update the template too, producing the second iteration of the service.

iface instance2
 device       c1
 interface    0/2
 ip-address   10.2.2.3
 cidr-netmask 28
 variant      v3
!

With the updated configuration, you can now safely reconcile the service2 service instance:

admin@ncs# iface instance2 re-deploy reconcile\
 { discard-non-service-config } dry-run
cli {
}
admin@ncs# iface instance2 re-deploy reconcile

Nevertheless, keep in mind that the discard-non-service-config reconcile operation only considers parts of the device configuration under nodes that are created with the service mapping. Even if all data there is covered in the mapping, there could still be other parts that belong to the service but reside in an entirely different section of the device configuration (say DNS configuration under ip name-server, which is outside the interface GigabitEthernet part) or even a different device. That kind of configuration the discard-non-service-config option cannot find on its own and you must add manually.

You can find the complete iface service as part of the examples.ncs/development-guide/services/discovery example.

Since there were only two service instances to reconcile, the process is now complete. In practice, you are likely to encounter multiple variants and many more service instances, requiring you to make additional iterations. But you can follow the iterative process shown here.

Partial Sync

In some cases a service may need to rely on the actual device configurations to compute the changeset. It is often a requirement to pull the current device configurations from the network before executing such service. Doing a full sync-from on a number of devices is an expensive task, especially if it needs to be performed often. The alternative way in this case is using partial-sync-from.

In cases where a multitude of service instances touch a device that is not entirely orchestrated using NSO, i.e. relying on the partial-sync-from feature described above, and the device needs to be replaced then all services need to be re-deployed. This can be expensive depending on the number of service instances. Partial-sync-to enables the replacement of devices in a more efficient fashion.

Partial-sync-from and partial-sync-to actions allow to specify certain portions of the device's configuration to be pulled or pushed from or to the network, respectively, rather than the full config. These are more efficient operations on NETCONF devices and NEDs that support the partial-show feature. NEDs that do not support the partial-show feature will fall back to pulling or pushing the whole configuration.

Even though partial-sync-from and partial-sync-to allows to pull or push only a part of the device's configuration, the actions are not allowed to break the consistency of configuration in CDB or on the device as defined by the YANG model. Hence, extra consideration needs to be given to dependencies inside the device model. If some configuration item A depends on configuration item B in the device's configuration, pulling only A may fail due to unsatisfied dependency on B. In this case, both A and B need to be pulled, even if the service is only interested in the value of A.

It is important to note that partial-sync-from and partial-sync-to clear the transaction ID of the device in NSO unless the whole configuration has been selected (e.g. /ncs:devices/ncs:device[ncs:name='ex0']/ncs:config). This ensures NSO does not miss any changes to other parts of the device configuration but it does make the device out of sync.

Partial sync-from

Pulling the configuration from the network needs to be initiated outside the service code. At the same time, the list of configuration subtrees required by a certain service should be maintained by the service developer. Hence it is a good practice for such a service to implement a wrapper action that invokes the generic /devices/partial-sync-from action with the correct list of paths. The user or application that manages the service would only need to invoke the wrapper action without needing to know which parts of the configuration the service is interested in.

The snippet in the example below (Example of running partial-sync-from action via Java API) gives an example of running partial-sync-from action via Java, using router device from examples.ncs/getting-started/developing-with-ncs/0-router-network.

        ConfXMLParam[] params = new ConfXMLParam[] {
        new ConfXMLParamValue("ncs", "path", new ConfList(new ConfValue[] {
        new ConfBuf("/ncs:devices/ncs:device[ncs:name='ex0']/"
        + "ncs:config/r:sys/r:interfaces/r:interface[r:name='eth0']"),
        new ConfBuf("/ncs:devices/ncs:device[ncs:name='ex1']/"
        + "ncs:config/r:sys/r:dns/r:server")
        })),
        new ConfXMLParamLeaf("ncs", "suppress-positive-result")};
        ConfXMLParam[] result =
        maapi.requestAction(params, "/ncs:devices/ncs:partial-sync-from");

As using Java for service development may be somewhat more involved than Python, this section provides further examples and additional tips for setting up the development environment for Java.

The two examples, a simple VLAN service and a Layer 3 MPLS VPN service are more elaborate but show the same techniques as Implementing Services.

Creating a Simple VLAN Service

In this example, you will create a simple VLAN service in Java. In order to illustrate the concepts, the device configuration is simplified from a networking perspective and only uses one single device type (Cisco IOS).

Overview of Steps

We will first look at the following preparatory steps:

1. Prepare a simulated environment of Cisco IOS devices: in this example, we start from scratch in order to illustrate the complete development process. We will not reuse any existing NSO examples.

2. Generate a template service skeleton package: use NSO tools to generate a Java-based service skeleton package.

3. Write and test the VLAN Service Model.

4. Analyze the VLAN service mapping to IOS configuration.

These steps are no different from defining services using templates. Next is to start playing with the Java Environment:

1. Configuring the start and stop of the Java VM.

2. First look at the Service Java Code: introduction to service mapping in Java.

3. Developing by tailing log files.

4. Developing using Eclipse.

Setting Up the Environment

We will start by setting up a run-time environment that includes simulated Cisco IOS devices and configuration data for NSO. Make sure you have sourced the ncsrc file.

1. Create a new directory that will contain the files for this example, such as:

$ mkdir ~/vlan-service
$ cd ~/vlan-service

1. Now, let's create a simulated environment with 3 IOS devices and an NSO that is ready to run with this simulated network:

$ ncs-netsim create-network $NCS_DIR/packages/neds/cisco-ios 3 c
$ ncs-setup --netsim-dir ./netsim/ --dest ./

1. Start the simulator and NSO:

$ ncs-netsim start
DEVICE c0 OK STARTED
DEVICE c1 OK STARTED
DEVICE c2 OK STARTED
$ ncs

1. Use the Cisco CLI towards one of the devices:

$ ncs-netsim cli-i c0
admin connected from 127.0.0.1 using console on ncs
c0> enable
c0# configure
Enter configuration commands, one per line. End with CNTL/Z.
c0(config)# show full-configuration
no service pad
no ip domain-lookup
no ip http server
no ip http secure-server
ip routing
ip source-route
ip vrf my-forward
bgp next-hop Loopback 1
!
...

1. Use the NSO CLI to get the configuration:

$ ncs_cli -C -u admin

admin connected from 127.0.0.1 using console on ncs
admin@ncs# devices sync-from
sync-result {
    device c0
    result true
}
sync-result {
    device c1
    result true
}
sync-result {
    device c2
    result true
}
admin@ncs# config
Entering configuration mode terminal

admin@ncs(config)# show full-configuration devices device c0 config
devices device c0
 config
  no ios:service pad
  ios:ip vrf my-forward
   bgp next-hop Loopback 1
  !
  ios:ip community-list 1 permit
  ios:ip community-list 2 deny
  ios:ip community-list standard s permit
  no ios:ip domain-lookup
  no ios:ip http server
  no ios:ip http secure-server
  ios:ip routing
...

1. Finally, set VLAN information manually on a device to prepare for the mapping later.

admin@ncs(config)# devices device c0 config ios:vlan 1234 
admin@ncs(config)# devices device c0 config ios:interface
                   FastEthernet 1/0 switchport mode trunk 
admin@ncs(config-if)# switchport trunk allowed vlan 1234 
admin@ncs(config-if)# top 

admin@ncs(config)# show configuration 
devices device c0
 config
  ios:vlan 1234
  !
  ios:interface FastEthernet1/0
   switchport mode trunk
   switchport trunk allowed vlan 1234
  exit
 !
!

admin@ncs(config)# commit 

Creating a Service Package

1. In the run-time directory, you created:

$ ls -F1
README.ncs
README.netsim
logs/
ncs-cdb/
ncs.conf
netsim/
packages/
scripts/
state/

Note the packages directory, cd to it:

$ cd packages
$ ls -l
total 8
cisco-ios -> .../packages/neds/cisco-ios

Currently, there is only one package, the Cisco IOS NED.

1. We will now create a new package that will contain the VLAN service.

$ ncs-make-package --service-skeleton java vlan
$ ls
cisco-ios vlan

This creates a package with the following structure:

During the rest of this section, we will work with the vlan/src/yang/vlan.yang and vlan/src/java/src/com/example/vlan/vlanRFS.java files.

The Service Model

So, if a user wants to create a new VLAN in the network what should the parameters be? Edit the vlan/src/yang/vlan.yang according to below:

  augment /ncs:services {
    list vlan {
      key name;

      uses ncs:service-data;
      ncs:servicepoint "vlan-servicepoint";
      leaf name {
        type string;
      }

      leaf vlan-id {
        type uint32 {
          range "1..4096";
        }
      }

      list device-if {
        key "device-name";
          leaf device-name {
            type leafref {
              path "/ncs:devices/ncs:device/ncs:name";
            }
          }
          leaf interface {
            type string;
          }
      }
    }
  }

This simple VLAN service model says:

1. We give a VLAN a name, for example net-1.

2. The VLAN has an id from 1 to 4096.

3. The VLAN is attached to a list of devices and interfaces. In order to make this example as simple as possible the interface name is just a string. A more correct and useful example would specify this is a reference to an interface to the device, but for now it is better to keep the example simple.

The VLAN service list is augmented into the services tree in NSO. This specifies the path to reach VLANs in the CLI, REST, etc. There are no requirements on where the service shall be added into NCS, if you want VLANs to be at the top level, simply remove the augments statement.

Make sure you keep the lines generated by the ncs-make-package:

uses ncs:service-data;
ncs:servicepoint "vlan-servicepoint";

The two lines tell NSO that this is a service. The first line expands to a YANG structure that is shared amongst all services. The second line connects the service to the Java callback.

To build this service model, cd to packages/vlan/src and type make (assumes that you have the prerequisite make build system installed).

$ cd packages/vlan/src/
$ make

We can now test the service model by requesting NSO to reload all packages:

$ ncs_cli -C -U admin
admin@ncs# packages reload
>>> System upgrade is starting.
>>> Sessions in configure mode must exit to operational mode.
>>> No configuration changes can be performed until upgrade has completed.
>>> System upgrade has completed successfully.
result Done

You can also stop and start NSO, but then you have to pass the option --with-package-reload when starting NSO. This is important, NSO does not by default take any changes in packages into account when restarting. When packages are reloaded the state/packages-in-use is updated.

Now, create a VLAN service, (nothing will happen since we have not defined any mapping).

admin@ncs(config)# services vlan net-0 vlan-id 1234 device-if c0 interface 1/0
admin@ncs(config-device-if-c0)# top
admin@ncs(config)# commit

Now, let us move on and connect that to some device configuration using Java mapping. Note well that Java mapping is not needed, templates are more straightforward and recommended but we use this as a "Hello World" introduction to Java service programming in NSO. Also at the end, we will show how to combine Java and templates. Templates are used to define a vendor-independent way of mapping service attributes to device configuration and Java is used as a thin layer before the templates to do logic, call-outs to external systems, etc.

Managing the NSO Java VM

The default configuration of the Java VM is:

admin@ncs(config)# show full-configuration java-vm | details
java-vm stdout-capture enabled
java-vm stdout-capture file ./logs/ncs-java-vm.log
java-vm connect-time           60
java-vm initialization-time    60
java-vm synchronization-timeout-action log-stop
java-vm jmx jndi-address 127.0.0.1
java-vm jmx jndi-port 9902
java-vm jmx jmx-address 127.0.0.1
java-vm jmx jmx-port 9901

By default, NCS will start the Java VM by invoking the command $NCS_DIR/bin/ncs-start-java-vm. That script will invoke

$ java com.tailf.ncs.NcsJVMLauncher

The class NcsJVMLauncher contains the main() method. The started Java VM will automatically retrieve and deploy all Java code for the packages defined in the load path of the ncs.conf file. No other specification than the package-meta-data.xml for each package is needed.

The verbosity of Java error messages can be controlled by:

admin@ncs(config)# java-vm exception-error-message verbosity
Possible completions:
  standard  trace  verbose

For more details on the Java VM settings, see NSO Java VM.

A First Look at Java Development

The service model and the corresponding Java callback are bound by the servicepoint name. Look at the service model in packages/vlan/src/yang:

The corresponding generated Java skeleton, (one print 'Hello World!' statement added):

Modify the generated code to include the print "Hello World!" statement in the same way. Re-build the package:

$ cd packages/vlan/src/
$ make

Whenever a package has changed, we need to tell NSO to reload the package. There are three ways:

1. Just reload the implementation of a specific package, will not load any model changes: admin@ncs# packages package vlan redeploy.

2. Reload all packages including any model changes: admin@ncs# packages reload.

3. Restart NSO with reload option: $ncs --with-package-reload.

When that is done we can create a service (or modify an existing one) and the callback will be triggered:

admin@ncs(config)# vlan net-0 vlan-id 888
admin@ncs(config-vlan-net-0)# commit

Now, have a look at the logs/ncs-java-vm.log:

$ tail ncs-java-vm.log
...
<INFO> 03-Mar-2014::16:55:23.705 NcsMain JVM-Launcher: \
       - REDEPLOY PACKAGE COLLECTION  --> OK
<INFO> 03-Mar-2014::16:55:23.705 NcsMain JVM-Launcher: \
       - REDEPLOY ["vlan"] --> DONE
<INFO> 03-Mar-2014::16:55:23.706 NcsMain JVM-Launcher: \
       - DONE COMMAND --> REDEPLOY_PACKAGE
<INFO> 03-Mar-2014::16:55:23.706 NcsMain JVM-Launcher: \
       - READ SOCKET =>
Hello World!

Tailing the ncs-java-vm.log is one way of developing. You can also start and stop the Java VM explicitly and see the trace in the shell. To do this, tell NSO not to start the VM by adding the following snippet to ncs.conf:

<java-vm>
    <auto-start>false</auto-start>
</java-vm>

Then, after restarting NSO or reloading the configuration, from the shell prompt:

$ ncs-start-java-vm
.....
.. all stdout from JVM

So modifying or creating a VLAN service will now have the "Hello World!" string show up in the shell. You can modify the package, then reload/redeploy, and see the output.

Using Eclipse

To use a GUI-based IDE Eclipse, first generate an environment for Eclipse:

$ ncs-setup --eclipse-setup

This will generate two files, .classpath and .project. If we add this directory to Eclipse as a File -> New -> Java Project, uncheck the Use default location and enter the directory where the .classpath and .project have been generated.

We are immediately ready to run this code in Eclipse.

All we need to do is choose the main() routine in the NcsJVMLauncher class. The Eclipse debugger works now as usual, and we can, at will, start and stop the Java code.

$ cp $NCS_DIR/etc/ncs/ncs.conf .

Edit the file and enter the following XML entry just after the Webui entry:

<japi>
    <new-session-timeout>PT1000S</new-session-timeout>
    <query-timeout>PT1000S</query-timeout>
    <connect-timeout>PT1000S</connect-timeout>
</japi>

Now, restart ncs, and from now on start it as:

$ ncs -c ./ncs.conf

You can verify that the Java VM is not running by checking the package status:

admin@ncs# show packages package vlan
packages package vlan
 package-version 1.0
 description     "Skeleton for a resource facing service - RFS"
 ncs-min-version 3.0
 directory       ./state/packages-in-use/1/vlan
 component RFSSkeleton
  callback java-class-name [ com.example.vlan.vlanRFS ]
 oper-status java-uninitialized

Create a new project and start the launcher main in Eclipse:

You can start and stop the Java VM from Eclipse. Note well that this is not needed since the change cycle is: modify the Java code, make in the src directory, and then reload the package. All while NSO and the JVM are running.

Change the VLAN service and see the console output in Eclipse:

Another option is to have Eclipse connect to the running VM. Start the VM manually with the -d option.

$ ncs-start-java-vm -d
Listening for transport dt_socket at address: 9000
NCS JVM STARTING
...

Then you can set up Eclipse to connect to the NSO Java VM:

In order for Eclipse to show the NSO code when debugging, add the NSO Source Jars (add external Jar in Eclipse):

Navigate to the service create for the VLAN service and add a breakpoint:

Commit a change of a VLAN service instance and Eclipse will stop at the breakpoint:

Writing the Service Code

Fetching the Service Attributes

So the problem at hand is that we have service parameters and a resulting device configuration. Previously, we showed how to do that with templates. The same principles apply in Java. The service model and the device models are YANG models in NSO irrespective of the underlying protocol. The Java mapping code transforms the service attributes to the corresponding configuration leafs in the device model.

The NAVU API lets the Java programmer navigate the service model and the device models as a DOM tree. Have a look at the create signature:

 @ServiceCallback(servicePoint="vlan-servicepoint",
        callType=ServiceCBType.CREATE)
    public Properties create(ServiceContext context,
                             NavuNode service,
                             NavuNode ncsRoot,
                             Properties opaque)
                             throws DpCallbackException {

Two NAVU nodes are passed: the actual service serviceinstance and the NSO root ncsRoot.

We can have a first look at NAVU by analyzing the first try statement:

try {
            // check if it is reasonable to assume that devices
            // initially has been sync-from:ed
            NavuList managedDevices =
            ncsRoot.container("devices").list("device");
            for (NavuContainer device : managedDevices) {
                if (device.list("capability").isEmpty()) {
                    String mess = "Device %1$s has no known capabilities, " +
                                   "has sync-from been performed?";
                    String key = device.getKey().elementAt(0).toString();
                    throw new DpCallbackException(String.format(mess, key));
                }
            }

NAVU is a lazy evaluated DOM tree that represents the instantiated YANG model. So knowing the NSO model: devices/device, (container/list) corresponds to the list of capabilities for a device, this can be retrieved by ncsRoot.container("devices").list("device").

The service node can be used to fetch the values of the VLAN service instance:

• vlan/name

• vlan/vlan-id

• vlan/device-if/device and vlan/device-if/interface

The first snippet that iterates the service model and prints to the console looks like below:

The com.tailf.conf package contains Java Classes representing the YANG types like ConfUInt32.

Try it out in the following sequence:

1. Rebuild the Java Code: In packages/vlan/src type make.

2. Reload the Package: In the NSO Cisco CLI, do admin@ncs# packages package vlan redeploy.

3. Create or Modify a vlan Service: In NSO CLI, do admin@ncs(config)# services vlan net-0 vlan-id 844 device-if c0 interface 1/0, and commit.

Mapping Service Attributes to Device Configuration

Remember the service attribute is passed as a parameter to the create method. As a starting point, look at the first three lines:

1. To reach a specific leaf in the model use the NAVU leaf method with the name of the leaf as a parameter. This leaf then has various methods like getting the value as a string.

2. service.leaf("vlan-id") and service.leaf(vlan._vlan_id_) are two ways of referring to the VLAN-id leaf of the service. The latter alternative uses symbols generated by the compilation steps. If this alternative is used, you get the benefit of compilation time checking. From this leaf you can get the value according to the type in the YANG model ConfUInt32 in this case.

3. Line 3 shows an example of casting between types. In this case, we prepare the VLAN ID as a 16 unsigned int for later use.

The next step is to iterate over the devices and interfaces. The NAVU elements() returns the elements of a NAVU list.

In order to write the mapping code, make sure you have an understanding of the device model. One good way of doing that is to create a corresponding configuration on one device and then display that with the pipe target display xpath. Below is a CLI output that shows the model paths for FastEthernet 1/0:

admin@ncs% show devices device c0 config ios:interface
           FastEthernet 1/0 | display xpath 

/devices/device[name='c0']/config/ios:interface/
         FastEthernet[name='1/0']/switchport/mode/trunk

/devices/device[name='c0']/config/ios:interface/
         FastEthernet[name='1/0']/switchport/trunk/allowed/vlan/vlans [ 111 ]

Another useful tool is to render a tree view of the model:

$ pyang -f jstree tailf-ned-cisco-ios.yang -o ios.html

This can then be opened in a Web browser and model paths are shown to the right:

Now, we replace the print statements with setting real configuration on the devices.

Let us walk through the above code line by line. The device-name is a leafref. The deref method returns the object that the leafref refers to. The getParent() might surprise the reader. Look at the path for a leafref: /device/name/config/ios:interface/name. The name leafref is the key that identifies a specific interface. The deref returns that key, while we want to have a reference to the interface, (/device/name/config/ios:interface), that is the reason for the getParent().

The next line sets the VLAN list on the device. Note well that this follows the paths displayed earlier using the NSO CLI. The sharedCreate() is important, it creates device configuration based on this service, and it says that other services might also create the same value, "shared". Shared create maintains reference counters for the created configuration in order for the service deletion to delete the configuration only when the last service is deleted. Finally, the interface name is used as a key to see if the interface exists, "containsNode()".

The last step is to update the VLAN list for each interface. The code below adds an element to the VLAN leaf-list.

// The interface
NavuNode theIf = feIntfList.elem(feIntfName);
theIf.container("switchport").
      sharedCreate().
      container("mode").
      container("trunk").
      sharedCreate();
// Create the VLAN leaf-list element
theIf.container("switchport").
      container("trunk").
      container("allowed").
      container("vlan").
      leafList("vlans").
      sharedCreate(vlanID16);

Note that the code uses the sharedCreate() functions instead of create(), as the shared variants are preferred and a best practice.

The above create method is all that is needed for create, read, update, and delete. NSO will automatically handle any changes, like changing the VLAN ID, adding an interface to the VLAN service, and deleting the service. This is handled by the FASTMAP engine, it renders any change based on the single definition of the create method.

Simple VLAN Service with Templates

Overview

The mapping strategy using only Java is illustrated in the following figure.

This strategy has some drawbacks:

• Managing different device vendors. If we would introduce more vendors in the network this would need to be handled by the Java code. Of course, this can be factored into separate classes in order to keep the general logic clean and just pass the device details to specific vendor classes, but this gets complex and will always require Java programmers to introduce new device types.

• No clear separation of concerns, domain expertise. The general business logic for a service is one thing, detailed configuration knowledge of device types is something else. The latter requires network engineers and the first category is normally separated into a separate team that deals with OSS integration.

Java and templates can be combined:

In this model, the Java layer focuses on required logic, but it never touches concrete device models from various vendors. The vendor-specific details are abstracted away using feature templates. The templates take variables as input from the service logic, and the templates in turn transform these into concrete device configuration. The introduction of a new device type does not affect the Java mapping.

This approach has several benefits:

• The service logic can be developed independently of device types.

• New device types can be introduced at runtime without affecting service logic.

• Separation of concerns: network engineers are comfortable with templates, they look like a configuration snippet. They have expertise in how configuration is applied to real devices. People defining the service logic often are more programmers, they need to interface with other systems, etc, this suites a Java layer.

Note that the logic layer does not understand the device types, the templates will dynamically apply the correct leg of the template depending on which device is touched.

The VLAN Feature Template

From an abstraction point of view, we want a template that takes the following variables:

• VLAN ID

• Device and interface

So the mapping logic can just pass these variables to the feature template and it will apply it to a multi-vendor network.

Create a template as described before.

• Create a concrete configuration on a device, or several devices of different type

• Request NSO to display that as XML

• Replace values with variables

This results in a feature template like below:

<!-- Feature Parameters -->
<!-- $DEVICE -->
<!-- $VLAN_ID -->
<!-- $INTF_NAME -->

<config-template xmlns="http://tail-f.com/ns/config/1.0"
                 servicepoint="vlan">
  <devices xmlns="http://tail-f.com/ns/ncs">
    <device>
      <name>{$DEVICE}</name>
      <config>
        <vlan xmlns="urn:ios" tags="merge">
          <vlan-list>
            <id>{$VLAN_ID}</id>
          </vlan-list>
        </vlan>
        <interface xmlns="urn:ios" tags="merge">
          <FastEthernet tags="nocreate">
            <name>{$INTF_NAME}</name>
            <switchport>
              <trunk>
                <allowed>
                  <vlan tags="merge">
                    <vlans>{$VLAN_ID}</vlans>
                  </vlan>
                </allowed>
              </trunk>
            </switchport>
          </FastEthernet>
        </interface>
      </config>
    </device>
  </devices>
</config-template>

This template only maps to Cisco IOS devices (the xmlns="urn:ios" namespace), but you can add "legs" for other device types at any point in time and reload the package.

The VLAN Java Logic

The Java mapping logic for applying the template is shown below:

Note that the Java code has no clue about the underlying device type, it just passes the feature variables to the template. At run-time, you can update the template with mapping to other device types. The Java code stays untouched, if you modify an existing VLAN service instance to refer to the new device type the commit will generate the corresponding configuration for that device.

The smart reader will complain, "Why do we have the Java layer at all?", this could have been done as a pure template solution. That is true, but now this simple Java layer gives room for arbitrary complex service logic before applying the template.

Steps to Build a Java and Template Solution

The steps to build the solution described in this section are:

1. Create a run-time directory: $ mkdir ~/service-template; cd ~/service-template.

2. Generate a netsim environment: $ ncs-netsim create-network $NCS_DIR/packages/neds/cisco-ios 3 c.

3. Generate the NSO runtime environment: $ ncs-setup --netsim-dir ./netsim --dest ./.

4. Create the VLAN package in the packages directory: $ cd packages; ncs-make-package --service-skeleton java vlan.

5. Create a template directory in the VLAN package: $ cd vlan; mkdir templates.

6. Save the above-described template in packages/vlan/templates.

7. Create the YANG service model according to the above: packages/vlan/src/yang/vlan.yang.

8. Update the Java code according to the above: packages/vlan/src/java/src/com/example/vlan/vlanRFS.java.

9. Build the package: in packages/vlan/src do make.

10. Start NSO.

Layer 3 MPLS VPN Service

This service shows a more elaborate service mapping. It is based on the examples.ncs/service-provider/mpls-vpn example.

MPLS VPNs are a type of Virtual Private Network (VPN) that achieves segmentation of network traffic using Multiprotocol Label Switching (MPLS), often found in Service Provider (SP) networks. The Layer 3 variant uses BGP to connect and distribute routes between sites of the VPN.

The figure below illustrates an example configuration for one leg of the VPN. Configuration items in bold are variables that are generated from the service inputs.

Auxiliary Service Data

Sometimes the input parameters are enough to generate the corresponding device configurations. But in many cases, this is not enough. The service mapping logic may need to reach out to other data in order to generate the device configuration. This is common in the following scenarios:

• Policies: it might make sense to define policies that can be shared between service instances. The policies, for example, QoS, have data models of their own (not service models) and the mapping code reads from that.

• Topology Information: the service mapping might need to know connected devices, like which PE the CE is connected to.

• Resources like VLAN IDs, and IP Addresses: these might not be given as input parameters. This can be modeled separately in NSO or fetched from an external system.

It is important to design the service model to consider the above examples: what is input? what is available from other sources? This example illustrates how to define QoS policies "on the side". A reference to an existing QoS policy is passed as input. This is a much better principle than giving all QoS parameters to every service instance. Note well that if you modify the QoS definitions that services are referring to, this will not change the existing services. In order to have the service to read the changed policies you need to perform a re-deploy on the service.

This example also uses a list that maps every CE to a PE. This list needs to be populated before any service is created. The service model only has the CE as input parameter, and the service mapping code performs a lookup in this list to get the PE. If the underlying topology changes a service re-deploy will adopt the service to the changed CE-PE links. See more on topology below.

NSO has a package to manage resources like VLAN and IP addresses as a pool within NSO. In this way the resources are managed within the transaction. The mapping code could also reach out externally to get resources. Nano services are recommended for this.

Topology

Using topology information in the instantiation of an NSO service is a common approach, but also an area with many misconceptions. Just like a service in NSO takes a black-box view of the configuration needed for that service in the network NSO treats topologies in the same way. It is of course common that you need to reference topology information in the service but it is highly desirable to have a decoupled and self-sufficient service that only uses the part of the topology that is interesting/needed for the specific service should be used.

Other parts of the topology could either be handled by other services or just let the network state sort it out - it does not necessarily relate to the configuration of the network. A routing protocol will for example handle the IP path through the network.

It is highly desirable to not introduce unneeded dependencies towards network topologies in your service.

To illustrate this, let's look at a Layer 3 MPLS VPN service. A logical overview of an MPLS VPN with three endpoints could look something like this. CE routers connecting to PE routers, that are connected to an MPLS core network. In the MPLS core network, there are a number of P routers.

In the service model, you only want to configure the CE devices to use as endpoints. In this case, topology information could be used to sort out what PE router each CE router is connected to. However, what type of topology do you need? Lets look at a more detailed picture of what the L1 and L2 topology could look like for one side of the picture above.

In pretty much all networks there is an access network between the CE and PE router. In the picture above the CE routers are connected to local Ethernet switches connected to a local Ethernet access network, connected through optical equipment. The local Ethernet access network is connected to a regional Ethernet access network, connected to the PE router. Most likely the physical connections between the devices in this picture have been simplified, in the real world redundant cabling would be used. The example above is of course only one example of how an access network could look like and it is very likely that a service provider have different access technologies. For example Ethernet, ATM, or a DSL-based access network.

Depending on how you design the L3VPN service, the physical cabling or the exact traffic path taken in the layer 2 Ethernet access network might not be that interesting, just like we don't make any assumptions or care about how traffic is transported over the MPLS core network. In both these cases we trust the underlying protocols handling state in the network, spanning tree in the Ethernet access network, and routing protocols like BGP in the MPLS cloud. Instead in this case, it could make more sense to have a separate NSO service for the access network, both so it can be reused for both for example L3VPNs and L2VPN but also to not tightly couple to the access network with the L3VPN service since it can be different (Ethernet or ATM etc.).

Looking at the topology again from the L3VPN service perspective, if services assume that the access network is already provisioned or taken care of by another service, it could look like this.

The information needed to sort out what PE router a CE router is connected to as well as configuring both CE and PE routers is:

• Interface on the CE router that is connected to the PE router, and IP address of that interface.

• Interface on the PE router that is connected to the CE router, and IP address to the interface.

Creating a Multi-Vendor Service

This section describes the creation of an MPLS L3VPN service in a multi-vendor environment by applying the concepts described above. The example discussed can be found in examples.ncs/service-provider/mpls-vpn. The example network consists of Cisco ASR 9k and Juniper core routers (P and PE) and Cisco IOS-based CE routers.

The goal of the NSO service is to set up an MPLS Layer3 VPN on a number of CE router endpoints using BGP as the CE-PE routing protocol. Connectivity between the CE and PE routers is done through a Layer2 Ethernet access network, which is out of the scope of this service. In a real-world scenario, the access network could for example be handled by another service.

In the example network, we can also assume that the MPLS core network already exists and is configured.

YANG Service Model Design

When designing service YANG models there are a number of things to take into consideration. The process usually involves the following steps:

1. Identify the resulting device configurations for a deployed service instance.

2. Identify what parameters from the device configurations are common and should be put in the service model.

3. Ensure that the scope of the service and the structure of the model work with the NSO architecture and service mapping concepts. For example, avoid unnecessary complexities in the code to work with the service parameters.

4. Ensure that the model is structured in a way so that integration with other systems north of NSO works well. For example, ensure that the parameters in the service model map to the needed parameters from an ordering system.

Steps 1 and 2: Device Configurations and Identifying Parameters:

Deploying an MPLS VPN in the network results in the following basic CE and PE configurations. The snippets below only include the Cisco IOS and Cisco IOS-XR configurations. In a real process, all applicable device vendor configurations should be analyzed.

  interface GigabitEthernet0/1.77
   description Link to PE / pe0 - GigabitEthernet0/0/0/3
   encapsulation dot1Q 77
   ip address 192.168.1.5 255.255.255.252
   service-policy output volvo
  !
  policy-map volvo
   class class-default
    shape average 6000000
   !
  !
 interface GigabitEthernet0/11
   description volvo local network
   ip address 10.7.7.1 255.255.255.0
  exit
  router bgp 65101
   neighbor 192.168.1.6 remote-as 100
   neighbor 192.168.1.6 activate
   network 10.7.7.0
  !

  vrf volvo
   address-family ipv4 unicast
    import route-target
     65101:1
    exit
    export route-target
     65101:1
    exit
   exit
  exit
  policy-map volvo-ce1
   class class-default
    shape average 6000000 bps
   !
   end-policy-map
  !
  interface GigabitEthernet 0/0/0/3.77
   description Link to CE / ce1 - GigabitEthernet0/1
   ipv4 address 192.168.1.6 255.255.255.252
   service-policy output volvo-ce1
   vrf         volvo
   encapsulation dot1q 77
  exit
  router bgp 100
   vrf volvo
    rd 65101:1
    address-family ipv4 unicast
    exit
    neighbor 192.168.1.5
     remote-as 65101
     address-family ipv4 unicast
      as-override
     exit
    exit
   exit
  exit

The device configuration parameters that need to be uniquely configured for each VPN have been marked in bold.

Steps 3 and 4: Model Structure and Integration with other Systems:

When configuring a new MPLS l3vpn in the network we will have to configure all CE routers that should be interconnected by the VPN, as well as the PE routers they connect to.

However, when creating a new l3vpn service instance in NSO it would be ideal if only the endpoints (CE routers) are needed as parameters to avoid having knowledge about PE routers in a northbound order management system. This means a way to use topology information is needed to derive or compute what PE router a CE router is connected to. This makes the input parameters for a new service instance very simple. It also makes the entire service very flexible, since we can move CE and PE routers around, without modifying the service configuration.

Resulting YANG Service Model:

container vpn {

  list l3vpn {
    tailf:info "Layer3 VPN";

    uses ncs:service-data;
    ncs:servicepoint l3vpn-servicepoint;

    key name;
    leaf name {
      tailf:info "Unique service id";
      type string;
    }
    leaf as-number {
      tailf:info "MPLS VPN AS number.";
      mandatory true;
      type uint32;
    }

    list endpoint {
      key id;
      leaf id {
        tailf:info "Endpoint identifier";
        type string;
      }
      leaf ce-device {
         mandatory true;
         type leafref {
           path "/ncs:devices/ncs:device/ncs:name";
         }
      }
      leaf ce-interface {
        mandatory true;
        type string;
      }
      leaf ip-network {
        tailf:info “private IP network”;
        mandatory true;
        type inet:ip-prefix;
      }
      leaf bandwidth {
        tailf:info "Bandwidth in bps";
        mandatory true;
        type uint32;
      }
    }
  }
}

The snipped above contains the l3vpn service model. The structure of the model is very simple. Every VPN has a name, an as-number, and a list of all the endpoints in the VPN. Each endpoint has:

• A unique ID.

• A reference to a device (a CE router in our case).

• A pointer to the LAN local interface on the CE router. This is kept as a string since we want this to work in a multi-vendor environment.

• LAN private IP network.

• Bandwidth on the VPN connection.

To be able to derive the CE to PE connections we use a very simple topology model. Notice that this YANG snippet does not contain any service point, which means that this is not a service model but rather just a YANG schema letting us store information in CDB.

container topology {
  list connection {
    key name;
    leaf name {
      type string;
    }
    container endpoint-1 {
      tailf:cli-compact-syntax;
      uses connection-grouping;
    }
    container endpoint-2 {
      tailf:cli-compact-syntax;
      uses connection-grouping;
    }
    leaf link-vlan {
      type uint32;
    }
  }
}

grouping connection-grouping {
  leaf device {
    type leafref {
      path "/ncs:devices/ncs:device/ncs:name";
    }
  }
  leaf interface {
    type string;
  }
  leaf ip-address {
    type tailf:ipv4-address-and-prefix-length;
  }
}

The model basically contains a list of connections, where each connection points out the device, interface, and IP address in each of the connections.

Defining the Mapping

Since we need to look up which PE routers to configure using the topology model in the mapping logic it is not possible to use a declarative configuration template-based mapping. Using Java and configuration templates together is the right approach.

The Java logic lets you set a list of parameters that can be consumed by the configuration templates. One huge benefit of this approach is that all the parameters set in the Java code are completely vendor-agnostic. When writing the code, there is no need for knowledge of what kind of devices or vendors exist in the network, thus creating an abstraction of vendor-specific configuration. This also means that in to create the configuration template there is no need to have knowledge of the service logic in the Java code. The configuration template can instead be created and maintained by subject matter experts, the network engineers.

With this service mapping approach, it makes sense to modularize the service mapping by creating configuration templates on a per-feature level, creating an abstraction for a feature in the network. In this example means, we will create the following templates:

• CE router

• PE router

This is both to make services easier to maintain and create but also to create components that are reusable from different services. This can of course be even more detailed with templates with for example BGP or interface configuration if needed.

Since the configuration templates are decoupled from the service logic it is also possible to create and add additional templates in a running NSO system. You can for example add a CE router from a new vendor to the layer3 VPN service by only creating a new configuration template, using the set of parameters from the service logic, to a running NSO system without changing anything in the other logical layers.

The Java Code

The Java code part for the service mapping is very simple and follows the following pseudo code steps:

READ topology
FOR EACH endpoint
        USING topology
DERIVE connected-pe-router
                READ ce-pe-connection
        SET pe-parameters
        SET ce-parameters
        APPLY TEMPLATE l3vpn-ce
        APPLY TEMPLATE l3vpn-pe

This section will go through relevant parts of Java outlined by the pseudo-code above. The code starts with defining the configuration templates and reading the list of endpoints configured and the topology. The Navu API is used for navigating the data models.

Template peTemplate = new Template(context, "l3vpn-pe");
        Template ceTemplate = new Template(context,"l3vpn-ce");
        NavuList endpoints = service.list("endpoint");
        NavuContainer topology = ncsRoot.getParent().
                container("http://com/example/l3vpn").
                container("topology");

The next step is iterating over the VPN endpoints configured in the service, finding out connected PE router using small helper methods navigating the configured topology.

 for(NavuContainer endpoint : endpoints.elements()) {
            try {
                String ceName =  endpoint.leaf("ce-device").valueAsString();
                // Get the PE connection for this endpoint router
                NavuContainer conn =
                    getConnection(topology,
                                  endpoint.leaf("ce-device").valueAsString());
                NavuContainer peEndpoint = getConnectedEndpoint(
                                                conn,ceName);
                NavuContainer ceEndpoint = getMyEndpoint(
                                                conn,ceName);

The parameter dictionary is created from the TemplateVariables class and is populated with appropriate parameters.

TemplateVariables vpnVar = new TemplateVariables();
vpnVar.putQuoted("PE",peEndpoint.leaf("device").valueAsString());
vpnVar.putQuoted("CE",endpoint.leaf("ce-device").valueAsString());
vpnVar.putQuoted("VLAN_ID", vlan.valueAsString());
vpnVar.putQuoted("LINK_PE_ADR",
getIPAddress(peEndpoint.leaf("ip-address").valueAsString()));
vpnVar.putQuoted("LINK_CE_ADR",
                getIPAddress(ceEndpoint. leaf("ip-address").valueAsString()));
vpnVar.putQuoted("LINK_MASK",
                getNetMask(ceEndpoint. leaf("ip-address").valueAsString()));
vpnVar.putQuoted("LINK_PREFIX",
                getIPPrefix(ceEndpoint.leaf("ip-address").valueAsString()));

The last step after all parameters have been set is applying the templates for the CE and PE routers for this VPN endpoint.

peTemplate.apply(service, vpnVar);
ceTemplate.apply(service, vpnVar);

Configuration Templates

The configuration templates are XML templates based on the structure of device YANG models. There is a very easy way to create the configuration templates for the service mapping if NSO is connected to a device with the appropriate configuration on it, using the following steps.

1. Configure the device with the appropriate configuration.

2. Add the device to NSO

3. Sync the configuration to NSO.

4. Display the device configuration in XML format.

5. Save the XML output to a configuration template file and replace configured values with parameters

The commands in NSO give the following output. To make the example simpler, only the BGP part of the configuration is used:

admin@ncs# devices device ce1 sync-from
admin@ncs# show running-config devices device ce1 config \
        ios:router bgp | display xml

<config xmlns="http://tail-f.com/ns/config/1.0">
  <devices xmlns="http://tail-f.com/ns/ncs">
  <device>
    <name>ce1</name>
      <config>
         <router xmlns="urn:ios">
          <bgp>
            <as-no>65101</as-no>
            <neighbor>
              <id>192.168.1.6</id>
              <remote-as>100</remote-as>
              <activate/>
            </neighbor>
            <network>
              <number>10.7.7.0</number>
            </network>
          </bgp>
        </router>
      </config>
  </device>
  </devices>
</config>

The final configuration template with the replaced parameters marked in bold is shown below. If the parameter starts with a $-sign, it's taken from the Java parameter dictionary; otherwise, it is a direct xpath reference to the value from the service instance.

<config-template xmlns="http://tail-f.com/ns/config/1.0">
  <devices xmlns="http://tail-f.com/ns/ncs">
    <device tags="nocreate">
      <name>{$CE}</name>
      <config>
       <router xmlns="urn:ios" tags="merge">
          <bgp>
            <as-no>{/as-number}</as-no>
            <neighbor>
              <id>{$LINK_PE_ADR}</id>
              <remote-as>100</remote-as>
              <activate/>
            </neighbor>
            <network>
              <number>{$LOCAL_CE_NET}</number>
            </network>
          </bgp>
        </router>
      </config>
    </device>
  </devices>
</config-template>

Progress tracing in NSO provides developers with useful information for debugging, diagnostics, and profiling. This information can be used both during development cycles and after the release of the software. The system overhead for progress tracing is usually negligible.

When a transaction or action is applied, NSO emits progress events. These events can be displayed and recorded in a number of different ways. The easiest way is to pipe an action to details in the CLI.

admin@ncs% commit | details
Possible completions:
  debug  verbose  very-verbose
admin@ncs% commit | details

As seen by the details output, all events are recorded with a timestamp and in some cases with the duration. All phases of the transaction, service, and device communication are printed.

applying transaction for running datastore usid=41 tid=1761 trace-id=d7f06482-41ad-4151-938d-7a8bc7b3ce33
entering validate phase
 2021-05-25T17:28:12.267 taking transaction lock... ok (0.000 s)
 2021-05-25T17:28:12.267 holding transaction lock...
 2021-05-25T17:28:12.268 creating rollback file... ok (0.004 s)
 2021-05-25T17:28:12.272 run transforms and transaction hooks...
 2021-05-25T17:28:12.273 run pre-transform validation... ok (0.000 s)
 2021-05-25T17:28:12.275 service-manager: service /ordserv[name='o2']: run service... ok (0.035 s)
 2021-05-25T17:28:12.311 run transforms and transaction hooks: ok (0.038 s)
 2021-05-25T17:28:12.311 mark inactive... ok (0.000 s)
 2021-05-25T17:28:12.311 pre validate... ok (0.000 s)
 2021-05-25T17:28:12.311 run validation over the changeset... ok (0.000 s)
 2021-05-25T17:28:12.312 run dependency-triggered validation... ok (0.000 s)
 2021-05-25T17:28:12.312 check configuration policies... ok (0.000 s)
leaving validate phase (0.045 s)
entering write-start phase
 2021-05-25T17:28:12.312 cdb: write-start
 2021-05-25T17:28:12.313 check data kickers... ok (0.000 s)
leaving write-start phase (0.001 s)
entering prepare phase
 2021-05-25T17:28:12.314 cdb: prepare
 2021-05-25T17:28:12.314 device-manager: prepare
leaving prepare phase (0.003 s)
entering commit phase
 2021-05-25T17:28:12.317 cdb: commit
 2021-05-25T17:28:12.318 service-manager: commit
 2021-05-25T17:28:12.318 device-manager: commit
 2021-05-25T17:28:12.320 holding transaction lock: ok (0.033 s)
leaving commit phase (0.002 s)
applying transaction for running datastore usid=41 tid=1761 trace-id=d7f06482-41ad-4151-938d-7a8bc7b3ce33 (0.053 s)

Some actions (usually those involving device communication) also produce progress data.

admin@ncs% request devices device ce0 sync-from dry-run | details very-verbose
running action /devices/device\[name='ce0'\]/sync-from usid=41 tid=1800 trace-id=fff4d4b0-5688-42f9-b5f7-53b7c3f70d35
 2021-05-25T17:31:31.222 device ce0: sync-from...
 2021-05-25T17:31:31.222 device ce0: taking device lock... ok (0.000 s)
 2021-05-25T17:28:12.267 device ce0: holding device lock...
 2021-05-25T17:31:31.227 device ce0: connect... ok (0.013 s)
 2021-05-25T17:31:31.240 device ce0: show... ok (0.001 s)
 2021-05-25T17:31:31.242 device ce0: get-trans-id... ok (0.000 s)
 2021-05-25T17:31:31.242 device ce0: close... ok (0.000 s)
...
 2021-05-25T17:28:12.320 device ce0: holding device lock: ok (0.033 s)
 2021-05-25T17:31:31.249 device ce0: sync-from: ok (0.026 s)
running action /devices/device\[name='ce0'\]/sync-from usid=41 tid=1800 trace-id=fff4d4b0-5688-42f9-b5f7-53b7c3f70d35 (0.053 s)

Configuring Progress Trace

The pipe details in the CLI are useful during development cycles of for example a service, but not as useful when tracing calls from other northbound interfaces or events in a released running system. Then it's better to configure a progress trace to be outputted to a file or operational data which can be retrieved through a northbound interface.

Unhide Progress Trace

The top-level container progress is by default invisible due to a hidden attribute. To make progress visible in the CLI, two steps are required:

1. First, the following XML snippet must be added to ncs.conf :\

   Copy

   <hide-group>
      <name>debug</name>
   </hide-group>

2. Then, the unhide command is used in the CLI session:

   Copy

   admin@ncs% unhide debug

Log to File

Progress data can be outputted to a given file. This is useful when the data is to be analyzed in some third-party software like a spreadsheet application.

admin@ncs% set progress trace test destination file event.csv format csv

The file can be formatted as a comma-separated values file defined by RFC 4180 or in a pretty printed log file with each event on a single line.

The location of the file is the directory of /ncs-config/logs/progress-trace/dir in ncs.conf.

Log as Operational Data

When the data is to be retrieved through a northbound interface, it is more useful to output the progress events as operational data.

admin@ncs% set progress trace test destination oper-data

This will log non-persistent operational data to the /progress:progress/trace/event list. As this list might grow rapidly there is a maximum size of it (defaults to 1000 entries). When the maximum size is reached, the oldest list entry is purged.

admin@ncs% set progress trace test max-size 2000

Using the /progress:progress/trace/purge action the event list can be purged.

admin# request progress trace test purge

Log as Notification Events

Progress events can be subscribed to as Notifications events. See NOTIF API for further details.

Verbosity

The verbosity parameter is used to control the level of output. The following levels are available:

Additional debug tracing can be turned on for various parts. These are consciously left out of the normal debug level due to the high amount of output and should only be turned on during development.

Using Filters

By default, all transaction and action events with the given verbosity level will be logged. To get a more selective choice of events, filters can be used.

admin@ncs% show progress trace filter
Possible completions:
  all-devices  - Only log events for devices.
  all-services - Only log events for services.
  context      - Only log events for the specified context.
  device       - Only log events for the specified device(s).
  device-group - Only log events for devices in this group.
  local-user   - Only log events for the specified local user.
  service-type - Only log events for the specified service type.

The context filter can be used to only log events that originate through a specific northbound interface. The context is either one of netconf, cli, webui, snmp, rest, system or it can be any other context string defined through the use of MAAPI.

admin@ncs% set progress trace test filter context netconf

Report Progress Events from User Code

API methods to report progress events exist for Java, Python, and C. There also exist specific methods to report progress events for services.

...
Maapi maapi = service.context().getMaapi();
int tHandle = service.context().getMaapiHandle();
ConfPath servicePath = new ConfPath(service.getKeyPath());
maapi.reportServiceProgress(tHandle, Maapi.Verbosity.VERBOSE,
                            "service test", servicePath);

With an increasing number of services and managed devices in NSO, performance becomes a more important aspect of the system. At the same time, other aspects, such as the way you organize code, also start playing an important role when using NSO on a bigger scale.

The following section examines these concerns and presents the available options for scaling your NSO automation solution.

Understanding Your Use Case

NSO allows you to tackle different automation challenges and every solution has its own specifics. Therefore, the best approach to scaling depends on the way the solution is implemented. What works in one case may be useless, or effectively degrade performance, for another. You must first analyze and understand how your particular use case behaves, which will then allow you to take the right approach to scaling.

When trying to improve the performance, a very good, possibly even the best starting point is to inspect the tracing data. Tracing is further described in Progress Trace. Yet a simple commit | details command already provides a lot of useful data.

admin@ncs(config-mysvc-test)# commit | details
 2022-09-16T09:17:48.977 applying transaction...
entering validate phase for running usid=54 tid=225 trace-id=3a4a3b7f-a09f-4f9d-b05e-1656310ea5b6
 2022-09-16T09:17:48.977 creating rollback checkpoint... ok (0.000 s)
 2022-09-16T09:17:48.978 creating rollback file... ok (0.004 s)
 2022-09-16T09:17:48.983 creating pre-transform checkpoint... ok (0.000 s)
 2022-09-16T09:17:48.983 run pre-transform validation... ok (0.000 s)
 2022-09-16T09:17:48.983 creating transform checkpoint... ok (0.000 s)
 2022-09-16T09:17:48.983 run transforms and transaction hooks...
 2022-09-16T09:17:48.985 taking service write lock... ok (0.000 s)
 2022-09-16T09:17:48.985 holding service write lock...
 2022-09-16T09:17:48.986 service /mysvc[name='test']: run service... ok (0.012 s)
 2022-09-16T09:17:48.999 run transforms and transaction hooks: ok (0.016 s)
 2022-09-16T09:17:48.999 creating validation checkpoint... ok (0.000 s)
 2022-09-16T09:17:49.000 mark inactive... ok (0.000 s)
 2022-09-16T09:17:49.001 pre validate... ok (0.000 s)
 2022-09-16T09:17:49.001 run validation over the changeset... ok (0.000 s)
 2022-09-16T09:17:49.002 run dependency-triggered validation... ok (0.000 s)
 2022-09-16T09:17:49.003 check configuration policies... ok (0.000 s)
 2022-09-16T09:17:49.003 check for read-write conflicts... ok (0.000 s)
 2022-09-16T09:17:49.004 taking transaction lock... ok (0.000 s)
 2022-09-16T09:17:49.004 holding transaction lock...
 2022-09-16T09:17:49.004 check for read-write conflicts... ok (0.000 s)
 2022-09-16T09:17:49.004 applying service meta-data... ok (0.000 s)
leaving validate phase for running usid=54 tid=225 trace-id=3a4a3b7f-a09f-4f9d-b05e-1656310ea5b6 (0.028 s)
entering write-start phase for running usid=54 tid=225 trace-id=3a4a3b7f-a09f-4f9d-b05e-1656310ea5b6
 2022-09-16T09:17:49.005 cdb: write-start
 2022-09-16T09:17:49.006 ncs-internal-service-mux: write-start
 2022-09-16T09:17:49.006 ncs-internal-device-mgr: write-start
 2022-09-16T09:17:49.007 cdb: match subscribers... ok (0.000 s)
 2022-09-16T09:17:49.007 cdb: create pre commit running... ok (0.000 s)
 2022-09-16T09:17:49.007 cdb: write changeset... ok (0.000 s)
 2022-09-16T09:17:49.008 check data kickers... ok (0.000 s)
leaving write-start phase for running usid=54 tid=225 trace-id=3a4a3b7f-a09f-4f9d-b05e-1656310ea5b6 (0.003 s)
entering prepare phase for running usid=54 tid=225 trace-id=3a4a3b7f-a09f-4f9d-b05e-1656310ea5b6
 2022-09-16T09:17:49.009 cdb: prepare
 2022-09-16T09:17:49.009 ncs-internal-device-mgr: prepare
 2022-09-16T09:17:49.022 device ex1: push configuration...
leaving prepare phase for running usid=54 tid=225 trace-id=3a4a3b7f-a09f-4f9d-b05e-1656310ea5b6 (0.121 s)
entering commit phase for running usid=54 tid=225 trace-id=3a4a3b7f-a09f-4f9d-b05e-1656310ea5b6
 2022-09-16T09:17:49.130 cdb: commit
 2022-09-16T09:17:49.130 cdb: switch to new running... ok (0.000 s)
 2022-09-16T09:17:49.132 ncs-internal-device-mgr: commit
 2022-09-16T09:17:49.149 device ex1: push configuration: ok (0.126 s)
 2022-09-16T09:17:49.151 holding service write lock: ok (0.166 s)
 2022-09-16T09:17:49.151 holding transaction lock: ok (0.147 s)
leaving commit phase for running usid=54 tid=225 trace-id=3a4a3b7f-a09f-4f9d-b05e-1656310ea5b6 (0.021 s)
 2022-09-16T09:17:49.151 applying transaction: ok (0.174 s)
Commit complete.
admin@ncs(config-mysvc-test)#

Pay attention to the time NSO spends doing specific tasks. For a simple service, these are mainly:

• Validate service data (pre-transform validation)

• Run service mapping logic

• Validate produced configuration (changeset)

• Push changes to affected devices

• Commit the new configuration

Tracing data can often quickly reveal a bottleneck, a hidden delay, or some other unexpected inefficiency in your code. The best strategy is to first address any such concerns if they show up since only well-performing code is a good candidate for further optimization. Otherwise, you might find yourself optimizing the wrong parameters and hitting a dead end. Visualizing the progress trace is often helpful in identifying bottlenecks. See Measuring Transaction Throughput.

Analyzing the service in isolation can yield useful insight. But it may also lead you in the wrong direction because some issues only manifest under load and the data from a live system can surprise you. That is why NSO supports different ways of exposing tracing information, including operational data and notification events. Remember to always verify that your observations and assumptions hold for a live, production system, too.

Where to Start?

The times for different parts of the transaction, as reported by the tracing data, are very useful in determining where to focus your efforts.

For example, if your service data model uses a very broad must or similar XPath statement, then NSO may potentially need to evaluate thousands of data entries. Such evaluation requires a considerable amount of additional processing and is, in turn, reflected in increased time spent in validation. The solution in this case is to limit the scope of the data referenced in the YANG constraint, which you can often achieve with a more specific XPath expression.

Similarly, if a significant amount of time is spent constructing a service mapping, perhaps there is some redundant work occurring that you could optimize? Sometimes, however, provisioning requires calls to other systems or some computationally expensive operation, which you cannot easily manage without. Then you might want to consider splitting the provisioning process into smaller pieces, using nano services, for example. See Simplify the Per-Device Concurrent Transaction Creation Using a Nano Service for an example use-case and references to the Nano service documentation.

In general, your own code for a single transaction with no additional load on NSO should execute quickly (sub-second, as a rule of thumb). The faster each service or action code is, the better the overall system performance. Using a service design pattern to both improve performance and scale and avoid conflicts is described in Design to Minimize Conflicts.

Divide the Work Correctly

Things such as reading external data or large computations should not be done inside the create code. Consider using an action to encapsulate these functions. An action does not run under the lock unless it triggers a transaction and can perform side effects as desired.

There are several ways to utilize an action:

• An action is allowed to perform side effects.

• An action can read operational data from devices or external systems.

• An action can write values to operational data in CDB, for later use from the service.

• An action can write configuration to CDB, potentially triggering a service.

Actions can be used together with nano services, see Simplify the Per-Device Concurrent Transaction Creation Using a Nano Service.

Optimizing Device Communication

With the default configuration, one of the first things you might notice standing out in the tracing data is that pushing device configuration takes a significant amount of time compared to other parts of service provisioning. Why is that?

All changes in NSO happen inside a transaction. Network devices participate in the transaction, which gives you the all-or-nothing behavior, to ensure correctness and consistency across the network. But network communication is not instantaneous and a transaction in NSO holds a lock while waiting for devices to process the change. This way, changes to network devices are serialized, even when there are multiple simultaneous transactions. However, a lock blocks other transactions from proceeding, ultimately limiting the overall NSO transaction rate.

So, in many cases, the NSO system is not really resource-constrained but merely experiencing lock contention. Therefore, making locks as short as possible is the best way to improve performance. In the example trace from the section Understanding Your Use Case, most of the time is spent in the prepare phase, where configuration changes are propagated to the network devices. Change propagation requires a management session with each participating device, as well as updating and validating the new configuration on the device side. Understandably, all of these tasks take time.

NSO allows you to influence this behavior. Take a look at Commit Queue on how to avoid long device locks with commit queues and the trade-offs they bring. Usually, enabling the commit queue feature is the first and the most effective step to significantly improving transaction times.

Improving Subscribers

The CDB subscriber mechanism is used to notify the application code about CDB changes and runs at the end of the transaction commit, inside a global lock. Due to this fact, the number and configuration of subscribers affect performance and should be investigated early in your performance optimization efforts.

A badly implemented subscriber prolongs the time the transaction holds the lock, preventing other transactions from completing, in addition to the original transaction taking more time to commit. There are mainly two reasons for suboptimal operation: either the subscriber is too broad and must process too many (irrelevant) changes, or it performs more work inside the lock as necessary. As a recommended practice, the subscriber should only note the changes and schedule the processing to be done later, in order to return and release the lock as quickly as possible.

Moreover, subscribers incur processing overhead regardless of their implementation because NSO needs to communicate with the custom subscriber code, typically written in Java or Python.

That is why modern, performant code in NSO should use the kicker mechanism instead of implementing custom subscribers. While it is still possible to create a badly performing kicker, you are less likely to do so inadvertently. In most situations, kickers are also easier to implement and troubleshoot. You can read more on kickers in Kicker.

Minimizing Concurrency Conflicts

The time it takes to complete a transaction is certainly an important performance metric. However, after a certain point, it gets increasingly hard or even impossible to get meaningful improvement from optimizing each individual transaction. As it turns out, on a busy system, there are usually multiple outstanding requests. So, instead of trying to process each as fast as possible one after another, the system might process them in parallel.

In practice and as the figure shows, some parts must still be processed sequentially to ensure transactional properties. However, there is a significant gain in the overall time it takes to process all transactions in a busy system, even though each might take a little longer individually due to the concurrency overhead.

Throughput then becomes a more relevant metric. It is the number of requests or transactions that the system can process in a given time unit. While throughput is still related to individual transaction times, other factors also come into play. An important one is the way in which NSO implements concurrency and the interaction between the transaction system and your, user, code. Designing for transaction throughput is covered in detail later in this section, and the NSO concurrency model is detailed in NSO Concurrency Model.

The section provides guidance on identifying transaction conflicts and what affects their occurrence, so you can make your code more resistant to producing them. Conflicts arise more frequently on busier systems and negatively affect throughput, which makes them a good candidate for optimization.

Fine-tuning the Concurrency Parameters

Depending on the specifics of the server running NSO, additional performance improvement might be possible by fine-tuning the transaction-limits set of configuration parameters in ncs.conf. Please see the ncs.conf(1) manpage for details.

Enabling Even More Parallelism

If you are experiencing high resource utilization, such as memory and CPU usage, while individual transactions are optimized to execute fast and the rate of conflicts is low, it's possible you are starting to see the level of demand that pushes the limits of this system.

First, you should try adding more resources, in a scale-up manner, if possible. At the same time, you might also have some services that are using an older, less performant user code execution model. For example, the way Python code is executed is controlled by the callpoint-model option, described in The application Component, which you should ensure is set to the most performant setting.

Regardless, a single system cannot scale indefinitely. After you have exhausted all other options, you will need to “scale out,” that is, split the workload across multiple NSO instances. You can achieve this by using the Layered Service Architecture (LSA) approach. But the approach has its trade-offs, so make sure it provides the right benefits in your case. The LSA is further documented in LSA Overview in Layered Service Architecture.

Limit sync-from

In a brownfield environment, where the configuration is not 100% automated and controlled by NSO alone but also written to by other systems or operators, NSO is bound to end up out-of-sync with the device. How to handle synchronization is a big topic, and it is vital to understand what it means to you when things are out of sync. This will help guide your strategy.

If NSO is frequently brought out of sync, it can be tempting to invoke sync-from from the create callback. While it does achieve a higher degree of reliability in the sense that service modifications won't return an out-of-sync error, the impact on performance is usually catastrophic. The typical sync-from operation takes orders of magnitudes longer than the typical service modification, and transactional throughput will suffer greatly.

But other alternatives are often better:

• You can synchronize the configuration from the device when it reports a change rather than when the service is modified by listening for configuration change events from the device, e.g., via RESTCONF or NETCONF notifications, SNMP traps, or Syslog, and invoking sync-from or partial-sync-from when another party (not NSO) has modified the device. See also the section called Partial Sync.

• Using the devices sync-from command does not hold the transaction lock and run across devices concurrently, which reduces the total amount of time spent time synchronizing. This is particularly useful for periodic synchronization to lower the risk of being out-of-sync when committing configuration changes.

• Using the no-overwrite commit flag, you can be more lax about being in sync and focus on not overwriting the modified configuration.

• If the configuration is 100% automated and controlled by NSO alone, using out-of-sync-behaviour accept, you can completely ignore if the device is in sync or not.

• Letting your modification fail with an out-of-sync error and handling that error at the calling side.

Designing for Maximal Transaction Throughput

Maximal transaction throughput refers to the maximum number of transactions a system can handle within a given period. Factors that can influence maximal transaction throughput include:

• Hardware capabilities (e.g., processing power, memory).

• Software efficiency.

• Network bandwidth.

• The complexity of the transactions themselves.

Besides making sure the system hardware capabilities and network bandwidth are not a bottleneck, there are four areas where the NSO user can significantly affect the transaction throughput performance for an NSO node:

• Run multiple transactions concurrently. For example, multiple concurrent RESTCONF or NETCONF edits, CLI commits, MAAPI apply(), nano service re-deploy, etc.

• Design to avoid conflicts and minimize the service create() and validation implementation. For example, in service templates and code mapping to devices or other service instances, YANG must statements with XPath expressions or validation code.

• Using commit queues to exclude the time to push configuration changes to devices from inside the transaction lock.

• Simplify using nano and stacked services. If the processor where NSO with a stacked service runs becomes a severe bottleneck, the added complexity of migrating the stacked service to an LSA setup can be motivated. LSA helps expose only a single service instance when scaling up the number of devices by increasing the number of available CPU cores beyond a single processor.

Measuring Transaction Throughput

Measuring transaction performance includes measuring the total wall-clock time for the service deployment transaction(s) and using the detailed NSO progress trace of the transactions to find bottlenecks. The developer log helps debug the NSO internals, and the XPath trace log helps find misbehaving XPath expressions used in, for example, YANG must statements.

The picture below shows a visualization of the NSO progress trace when running a single transaction for two service instances configuring a device each:

The total RESTCONF edit took ~5 seconds, and the service mapping (“creating service” event) and validation (“run validation ...” event) were done sequentially for the service instances and took 2 seconds each. The configuration push to the devices was done concurrently in 1 second.

For progress trace documentation, see Progress Trace.

Running the perf-trans Example Using a Single Transaction

The perf-trans example from the NSO example set explores the opportunities to improve the wall-clock time performance and utilization, as well as opportunities to avoid common pitfalls.

The example uses simulated CPU loads for service creation and validation work. Device work is simulated with sleep() as it will not run on the same processor in a production system.

The example shows how NSO can benefit from running many transactions concurrently if the service and validation code allow concurrency. It uses the NSO progress trace feature to get detailed timing information for the transactions in the system.

The provided code sets up an NSO instance that exports tracing data to a .csv file, provisions one or more service instances, which each map to a device, and shows different (average) transaction times and a graph to visualize the sequences plus concurrency.

Play with the perf-trans example by tweaking the measure.py script parameters:

-nt NTRANS, --ntrans NTRANS
    The number of transactions updating the same service in parallel. For this
    example, we use NTRANS parallel RESTCONF <tag>plain patch</tag>.
    Default: 1.

-nw NWORK, --nwork NWORK
    Work per transaction in the service creation and validation phases. One
    second of CPU time per work item.
    Default: 3 seconds of CPU time.

-nd 0..10, --ndtrans 0..10
    Number of devices the service will configure per service transaction.
    Default: 1

-dd DDELAY, --ddelay DDELAY
    Transaction delay (simulated by sleeping) on the netsim devices (seconds).
    Default: 0s

-cq {async,sync,bypass,none}, --cqparam {async,sync,bypass,none}
    Commit queue behavior. Select "none" to use the global or device setting.
    Default: none

See the README in the perf-trans example for details.

To run the perf-trans example from the NSO example set and recreate the variant shown in the progress trace above:

cd $NCS_DIR/examples.ncs/development-guide/concurrency-model/perf-trans
make NDEVS=2 python
python3 measure.py --ntrans 1 --nwork 2 --ndtrans 2 --cqparam bypass --ddelay 1
python3 ../common/simple_progress_trace_viewer.py $(ls logs/*.csv)

The following is a sequence diagram and the progress trace of the example, describing the transaction t1. The transaction deploys service configuration to the devices using a single RESTCONF patch request to NSO and then NSO configures the netsim devices using NETCONF:

RESTCONF   service   validate   push config
patch      create    config     ndtrans=2        netsim
ntrans=1   nwork=2   nwork=2    cqparam=bypass   device    ddelay=1
  t1 ------> 2s -----> 2s -----------------------> ex0 -----> 1s
                                    \------------> ex1 -----> 1s
  wall-clock 2s        2s                                     1s = 5s

The only part running concurrently in the example above was configuring the devices. It is the most straightforward option if transaction throughput performance is not a concern or the service creation and validation work are insignificant. A single transaction service deployment will not need to use commit queues as it is the only transaction holding the transaction lock configuring the devices inside the critical section. See the “holding transaction lock” event in the progress trace above.

Stop NSO and the netsim devices:

make stop

Concurrent Transactions

Everything from smartphones and tablets to laptops, desktops, and servers now contain multi-core processors. For maximal throughput, the powerful multi-core systems need to be fully utilized. This way, the wall clock time is minimized when deploying service configuration changes to the network, which is usually equated with performance. Therefore, enabling NSO to spread as much work as possible across all available cores becomes important. The goal is to have service deployments maximize their utilization of the total available CPU time to deploy services faster to the users who ordered them.

Close to full utilization of every CPU core when running under maximal load, for example, ten transactions to ten devices, is ideal, as some process viewer tools such as htop visualize with meters:

    0[|||||||||||||||||||||||||||||||||||||||||||||||||100.0%]
    1[|||||||||||||||||||||||||||||||||||||||||||||||||100.0%]
    2[||||||||||||||||||||||||||||||||||||||||||||||||||99.3%]
    3[||||||||||||||||||||||||||||||||||||||||||||||||||99.3%]
    4[||||||||||||||||||||||||||||||||||||||||||||||||||99.3%]
    5[||||||||||||||||||||||||||||||||||||||||||||||||||99.3%]
    6[||||||||||||||||||||||||||||||||||||||||||||||||||98.7%]
    7[||||||||||||||||||||||||||||||||||||||||||||||||||98.7%]
    8[||||||||||||||||||||||||||||||||||||||||||||||||||98.7%]
    9[||||||||||||||||||||||||||||||||||||||||||||||||||98.7%]
    ...

One transaction per RFS instance and device will allow each NSO transaction to run on a separate core concurrently. Multiple concurrent RESTCONF or NETCONF edits, CLI commits, MAAPI apply(), nano service re-deploy, etc. Keep the number of running concurrent transactions equal to or below the number of cores available in the multi-core processor to avoid performance degradation due to increased contention on system internals and resources. NSO helps by limiting the number of transactions applying changes in parallel to, by default, the number of logical processors (e.g., CPU cores). See ncs.conf(5) in Manual Pages under /ncs-config/transaction-limits/max-transactions for details.

Design to Minimize Conflicts

Conflicts between transactions and how to avoid them are described in Minimizing Concurrency Conflicts and in detail by the NSO Concurrency Model. While NSO can handle transaction conflicts gracefully with retries, retries affect transaction throughput performance. A simple but effective design pattern to avoid conflicts is to update one device with one Resource Facing Service (RFS) instance where service instances do not read each other's configuration changes.

Design to Minimize Service and Validation Processing Time

An overly complex service or validation implementation using templates, code, and XPath expressions increases the processing required and, even if transactions are processed concurrently, will affect the wall-clock time spent processing and, thus, transaction throughput.

When data processing performance is of interest, the best practice rule of thumb is to ensure that must and when statement XPath expressions in YANG models and service templates are only used as necessary and kept as simple as possible.

If a service creates a significant amount of configuration data for devices, it is often significantly faster using a single MAAPI shared_set_values() call instead of using multiple create() and set() calls or a service template.

Running the perf-setvals Example Using a Single Call to MAAPI shared_set_values()

The perf-setvals example writes configuration to an access control list and a route list of a Cisco Adaptive Security Appliance (ASA) device. It uses either MAAPI Python create() and set() calls, Python shared_set_values(), or Java sharedSetValues() to write the configuration in XML format.

To run the perf-setvals example using MAAPI Python create() and set() calls to create 3000 rules and 3000 routes on one device:

cd $NCS_DIR/examples.ncs/development-guide/concurrency-model/perf-setvals
./measure.sh -r 3000 -t py_create -n true

The commit uses the no-networking parameter to skip pushing the configuration to the simulated and un-proportionally slow Cisco ASA netsim device. The resulting NSO progress trace:

Next, run the perf-setvals example using a single MAAPI Python shared_set_values() call to create 3000 rules and 3000 routes on one device:

./measure.sh -r 3000 -t py_setvals_xml -n true

The resulting NSO progress trace:

Using the MAAPI shared_set_values() function, the service create callback is, for this example, ~5x faster than using the MAAPI create() and set() functions. The total wall-clock time for the transaction is more than 2x faster, and the difference will increase for larger transactions.

Stop NSO and the netsim devices:

make stop

Use a Data Kicker Instead of a CDB Subscriber

A kicker triggering on a CDB change, a data-kicker, should be used instead of a CDB subscriber when the action taken does not have to run inside the transaction lock, i.e., the critical section of the transaction. A CDB subscriber will be invoked inside the critical section and, thus, will have a negative impact on the transaction throughput. See Improving Subscribers for more details.

Shorten the Time Used for Writing Configuration to Devices

Writing to devices and other network elements that are slow to configure will stall transaction throughput if you do not enable commit queues, as transactions waiting for the transaction lock to be released cannot start configuring devices before the transaction ahead of them is done writing. For example, if one device is configured using CLI transported with IP over Avian Carriers, the transactions, including such a device, will significantly stall transactions behind it going to devices supporting RESTCONF or NETCONF over a fast optical transport. Where transaction throughput performance is a concern, choosing devices that can be configured efficiently to implement their part of the service configuration is wise.

Running the perf-trans Example Using One Transaction per Device

Dividing the service creation and validation work into two separate transactions, one per device, allows the work to be spread across two CPU cores in a multi-core processor. To run the perf-trans example with the work divided into one transaction per device:

cd $NCS_DIR/examples.ncs/development-guide/concurrency-model/perf-trans
make stop clean NDEVS=2 python
python3 measure.py --ntrans 2 --nwork 1 --ndtrans 1 --cqparam bypass --ddelay 1
python3 ../common/simple_progress_trace_viewer.py $(ls logs/*.csv)

The resulting NSO progress trace:

A sequence diagram with transactions t1 and t2 deploying service configuration to two devices using RESTCONF patch requests to NSO with NSO configuring the netsim devices using NETCONF:

RESTCONF   service   validate   push config
patch      create    config     ndtrans=1       netsim            netsim
ntrans=2   nwork=1   nwork=1    cqparam=bypass  device  ddelay=1  device  ddelay=1
  t1 ------> 1s -----> 1s ---------------------> ex0 ---> 1s
  t2 ------> 1s -----> 1s ---------------------------------------> ex1 ---> 1s
  wall-clock 1s        1s                                 1s                1s = 4s

Note how the service creation and validation work now is divided into 1s per transaction and runs concurrently on one CPU core each. However, the two transactions cannot push the configuration concurrently to a device each as the config push is done inside the critical section, making one of the transactions wait for the other to release the transaction lock. See the two “holding the transaction lock” events in the above progress trace visualization.

To enable transactions to push configuration to devices concurrently, we must enable commit queues.

Using Commit Queues

The concept of a network-wide transaction requires NSO to wait for the managed devices to process the configuration change before exiting the critical section, i.e., before NSO can release the transaction lock. In the meantime, other transactions have to wait their turn to write to CDB and the devices. The commit queue feature avoids waiting for configuration to be written to the device and increases the throughput. For most use cases, commit queues improve transaction throughput significantly.

Writing to a commit queue instead of the device moves the device configuration push outside of the critical region, and the transaction lock can instead be released when the change has been written to the commit queue.

For commit queue documentation, see Commit Queue.

Enabling Commit Queues for the perf-trans Example

Enabling commit queues allows the two transactions to spread the create, validation, and configuration push to devices work across CPU cores in a multi-core processor. Only the CDB write and commit queue write now remain inside the critical section, and the transaction lock is released as soon as the device configuration changes have been written to the commit queues instead of waiting for the config push to the devices to complete. To run the perf-trans example with the work divided into one transaction per device and commit queues enabled:

make stop clean NDEVS=2 python
python3 measure.py --ntrans 2 --nwork 1 --ndtrans 1 --cqparam sync --ddelay 1
python3 ../common/simple_progress_trace_viewer.py $(ls logs/*.csv)

The resulting NSO progress trace:

A sequence diagram with transactions t1 and t2 deploying service configuration to two devices using RESTCONF patch requests to NSO with NSO configuring the netsim devices using NETCONF:

RESTCONF   service   validate   push config
patch      create    config     ndtrans=1        netsim
ntrans=2   nwork=1   nwork=1    cqparam=sync     device    ddelay=1
  t1 ------> 1s -----> 1s --------------[----]---> ex0 -----> 1s
  t2 ------> 1s -----> 1s --------------[----]---> ex1 -----> 1s
  wall-clock 1s        1s                                     1s = 3s

Note how the two transactions now push the configuration concurrently to a device each as the config push is done outside of the critical section. See the two push configuration events in the above progress trace visualization.

Stop NSO and the netsim devices:

make stop

Running the perf-setvals example with two devices and commit queues enabled will produce a similar result.

Simplify the Per-Device Concurrent Transaction Creation Using a Nano Service

The perf-trans example service uses one transaction per service instance where each service instance configures one device. This enables transactions to run concurrently on separate CPU cores in a multi-core processor. The example sends RESTCONF patch requests concurrently to start transactions that run concurrently with the NSO transaction manager. However, dividing the work into multiple processes may not be practical for some applications using the NSO northbound interfaces, e.g., CLI or RESTCONF. Also, it makes a future migration to LSA more complex.

To simplify the NSO manager application, a resource-facing nano service (RFS) can start a process per service instance. The NSO manager application or user can then use a single transaction, e.g., CLI or RESTCONF, to configure multiple service instances where the NSO nano service divides the service instances into transactions running concurrently in separate processes.

The nano service can be straightforward, for example, using a single t3:configured state to invoke a service template or a create() callback. If validation code is required, it can run in a nano service post-action, t3:validated state, instead of a validation point callback to keep the validation code in the process created by the nano service.

See Nano Services for Staged Provisioning and Develop and Deploy a Nano Service for Nano service documentation.

Simplify Using a CFS

A Customer Facing Service (CFS) that is stacked with the RFS and maps to one RFS instance per device can simplify the service that is exposed to the NSO northbound interfaces so that a single NSO northbound interface transaction spawns multiple transactions, for example, one transaction per RFS instance when using the converge-on-re-deploy YANG extension with the nano service behavior tree.

Running the CFS and Nano Service enabled perf-stack Example

The perf-stack example showcases how a CFS on top of a simple resource-facing nano service can be implemented with the perf-trans example by modifying the existing t3 RFS and adding a CFS. Instead of multiple RESTCONF transactions, the example uses a single CLI CFS service commit that updates the desired number of service instances. The commit configures multiple service instances in a single transaction where the nano service runs each service instance in a separate process to allow multiple cores to be used concurrently.

Run as below to start two transactions with a 1-second CPU time workload per transaction in both the service and validation callbacks, each transaction pushing the device configuration to one device, each using a synchronous commit queue, where each device simulates taking 1 second to make the configuration changes to the device:

cd $NCS_DIR/examples.ncs/development-guide/concurrency-model/perf-stack
./showcase.sh -d 2 -t 2 -w 1 -r 1 -q 'True' -y 1

The above progress trace visualization is truncated to fit, but notice how the t3:validated state action callbacks, t3:configured state service creation callbacks and configuration push from the commit queues are running concurrently (on separate CPU cores) when initiating the service deployment with a single transaction started by the CLI commit.

A sequence diagram describing the transaction t1 deploying service configuration to the devices using the NSO CLI:

                                                              config
        CFS             validate  service  push config        change
CLI     create    Nano  config    create   ndtrans=1   netsim subscriber
commit  trans=2   RFS   nwork=1   nwork=1  cq=True     device ddelay=1
                  t1 --> 1s -----> 1s -------[----]---> ex0 ---> 1s
  t -----> t --->
                  t2 --> 1s -----> 1s -------[----]---> ex1 ---> 1s
              wall-clock 1s        1s                            1s=3s

The two transactions run concurrently, deploying the service in ~3 seconds (plus some overhead) of wall-clock time. Like the perf-trans example, you can play around with the perf-stack example by tweaking the parameters.

-d  NDEVS
    The number of netsim (ConfD) devices (network elements) started.
    Default 4

-t  NTRANS
    The number of transactions updating the same service in parallel.
    Default: $NDEVS

-w  NWORK
    Work per transaction in the service creation and validation phases. One
    second of CPU time per work item.
    Default: 3 seconds of CPU time.

-r  NDTRANS
    Number of devices the service will configure per service transaction.
    Default: 1

-c  USECQ
    Use device commit queues.
    Default: True

-y  DEV_DELAY
    Transaction delay (simulated by sleeping) on the netsim devices (seconds).
    Default: 1 second

See the README in the perf-stack example for details. For even more details, see the steps in the showcase script.

Stop NSO and the netsim devices:

make stop

Migrating to and Scale Up Using an LSA Setup

If the processor where NSO runs becomes a severe bottleneck, the CFS can migrate to a layered service architecture (LSA) setup. The perf-stack example implements stacked services, a CFS abstracting the RFS. It allows for easy migration to an LSA setup to scale with the number of devices or network elements participating in the service deployment. While adding complexity, LSA allows exposing a single CFS instance for all processors instead of one per processor.

Running the LSA-enabled perf-lsa Example

The perf-lsa example builds on the perf-stack example and showcases an LSA setup using two RFS NSO instances, lower-nso-1 and lower-nso-2, with a CFS NSO instance, upper-nso.

You can imagine adding more RFS NSO instances, lower-nso-3, lower-nso-4, etc., to the existing two as the number of devices increases. One NSO instance per multi-core processor and at least one CPU core per device (network element) is likely the most performant setup for this simulated work example. See LSA Overview in Layered Service Architecture for more.

As an example, a variant that starts four RFS transactions with a 1-second CPU time workload per transaction in both the service and validation callbacks, each RFS transaction pushing the device configuration to 1 device using synchronous commit queues, where each device simulates taking 1 second to make the configuration changes to the device:

cd $NCS_DIR/examples.ncs/development-guide/concurrency-model/perf-lsa
./showcase.sh -d 2 -t 2 -w 1 -r 1 -q 'True' -y 1

The three NSO progress trace visualizations show NSO on the CFS and the two RFS nodes. Notice how the CLI commit starts a transaction on the CFS node and configures four service instances with two transactions on each RFS node to push the resulting configuration to four devices.

A sequence diagram describing the transactions on RFS 1 t1 t2 and RFS 2 t1 t2. The transactions deploy service configuration to the devices using the NSO CLI:

                                                             config
       CFS             validate  service  push config        change
CLI    create    Nano  config    create   ndtrans=1   netsim subscriber
commit ntrans=2  RFS 1 nwork=1   nwork=1  cq=True     device ddelay=1
  t -----> t ---> t1 --> 1s -----> 1s -------[----]---> ex0 ---> 1s
            \     t2 --> 1s -----> 1s -------[----]---> ex1 ---> 1s
             \   RFS 2
              --> t1 --> 1s -----> 1s -------[----]---> ex2 ---> 1s
                  t2 --> 1s -----> 1s -------[----]---> ex3 ---> 1s
              wall-clock 1s        1s                            1s=3s

The four transactions run concurrently, two per RFS node, performing the work and configuring the four devices in ~3 seconds (plus some overhead) of wall-clock time.

You can play with the perf-lsa example by tweaking the parameters.

-d  LDEVS
    Number of netsim (ConfD) devices (network elements) started per RFS
    NSO instance.
    Default 2 (4 total)

-t  NTRANS
    Number of transactions updating the same service in parallel per RFS
    NSO instance. Here, one per device.
    Default: $LDEVS ($LDEVS * 2 total)

-w  NWORK
    Work per transaction in the service creation and validation phases. One
    second of CPU time per work item.
    Default: 3 seconds of CPU time.

-r  NDTRANS
    Number of devices the service will configure per service transaction.
    Default: 1

-q  USECQ
    Use device commit queues.
    Default: True

-y  DEV_DELAY
    Transaction delay (simulated by sleeping) on the netsim devices (seconds).
    Default: 1 second

See the README in the perf-lsa example for details. For even more details, see the steps in the showcase script.

Stop NSO and the netsim devices:

make stop

Scaling RAM and Disk

NSO contains an internal database called CDB, which stores both configuration and operational state data. Understanding the resource consumption of NSO at a steady state is mostly about understanding CDB, as it typically stands for the vast majority of resource usage.

CDB

Optimized for fast access, CDB is an in-memory database that holds all data in RAM. It also keeps the data on disk for persistence. The in-memory data structure is optimized for navigating tree data but is still a compact and efficient memory structure. The on-disk format uses a log structure, making it fast to write and very compact.

The in-memory structure usually consumes 2 - 3x more than the size of the on-disk format. The on-disk log will grow as more changes are performed in the system. A periodic compaction process compacts the write log and reduces its size. Upon startup of NSO, the on-disk version of CDB will be read, and the in-memory structure will be recreated based on the log. A recently compacted CDB will thus start up faster.

By default, NSO automatically determines when to compact CDB. It is visible in the devel.log when CDB compaction takes place. Compaction may require significant time, during which write transactions cannot be performed. In certain use cases, it may be preferable to disable automatic compaction by CDB and instead trigger compaction manually according to the specific needs. See Compaction for more details.

Services and Devices in CDB

CDB is a YANG-modeled database. By writing a YANG model, it is possible to store any kind of data in NSO and access it via one of the northbound interfaces of NSO. From this perspective, a service or a device's configuration is like most other YANG-modeled data. The number of service instances in NSO in the steady state affects how much space the data consumes in RAM and on disk.

But keep in mind that services tend to be modified from time to time, and with a higher total number of service instances, changes to those services are more likely. A higher number of service instances means more transactions to deploy changes, which means an increased need for optimizing transactional throughput, available CPU processing, RAM, and disk. See Designing for Maximal Transaction Throughput for details.

CDB Stores the YANG Model Schema

In addition to storing instance data, CDB also stores the schema (the YANG models), on disk and reads it into memory on startup. Having a large schema (many or large YANG models) loaded means both disk and RAM will be used, even when starting up an "empty" NSO, i.e., no instance data is stored in CDB.

In particular, device YANG models can be of considerable size. For example, the YANG models in recent versions of Cisco IOS XR have over 750,000 lines. Loading one such NED will consume about 1GB GB of RAM and slightly less disk space. In a mixed vendor network, you would load NEDs for all or some of these device types. With CDM, you can have multiple XR NEDs loaded to support communicating with different versions of XR and similarly for other devices, further consuming resources.

In comparison, most CLI NEDs only model a subset of a device and, are as a result, much smaller, most often under 100,000 lines of YANG.

For small NSO systems, the schema will usually consume more resources than the instance data, and NEDs, in particular, are the most significant contributors to resource consumption. As the system grows and more service and device configurations are added, the percentage of the total resource usage used for NED YANG models will decrease.

Note that the schema is memory mapped into shared memory, so even though multiple Python VMs might be started, memory usage will not increase as it shares memory between different clients. The Java VM uses its own copy of the schema, which is also why we can see that the JVM memory consumption follows the size of the loaded YANG schema.

The Size of CDB

Accurately predicting the size of CDB means accurately modeling its internal data structure. Since the result will depend on the YANG models and what actual values are stored in the database, the easiest way to understand of how the size grows is to start NSO with the schema and data in question and then measure the resource usage.

Performing accurate measurements can be a tedious process or sometimes impossible. When impossible, an estimate can be reached by extrapolating from known data, which is usually much more manageable and accurate enough.

We can look at the disk and RAM used for the running datastore, which stores configuration. On a freshly started NSO, it doesn't occupy much space at all:

# show ncs-state internal cdb datastore running | select ram-size | select disk-size
         DISK
NAME     SIZE      RAM SIZE
------------------------------
running  3.83 KiB  26.27 KiB