Learn basic operational scenarios and common CLI commands.
This section helps you to get started with NSO, learn basic operational scenarios, and get acquainted with the most common CLI commands.
Make sure that you have installed NSO and that you have sourced the ncsrc file in $NCS_DIR. This sets up the paths and environment variables to run NSO. As this must be done every time before running NSO, it is recommended to add it to your profile.
We will use the NSO network simulator to simulate three Cisco IOS routers. NSO will talk Cisco CLI to those devices. You will use the NSO CLI and Web UI to perform the tasks. Sometimes you will use the native Cisco device CLI to inspect configuration or do out-of-band changes.
Note that both the NSO software (NCS) and the simulated network devices run on your local machine.
To start the simulator:
Go to examples.ncs/getting-started/using-ncs/1-simulated-cisco-ios. First of all, we will generate a network simulator with three Cisco devices. They will be called c0, c1, and c2.\
Most of this section follows the procedure in the README file, so it is useful to have it opened as well.
2. Start the simulator.\
3. Run the CLI toward one of the simulated devices.\
The previous step started the simulated Cisco devices. It is now time to start NSO.
The first action is to prepare directories needed for NSO to run and populate NSO with information on the simulated devices. This is all done with the ncs-setup command. Make sure that you are in the examples.ncs/getting-started/using-ncs/1-simulated-cisco-ios directory. (Again ignore the details for the time being).\
Note the . at the end of the command referring to the current directory. What the command does is to create directories needed for NSO in the current directory and populate NSO with devices that are running in netsim. We call this the "run-time" directory.
Start NSO.
Start the NSO CLI as the user admin with a Cisco XR-style CLI.\
NSO also supports a J-style CLI, that is started by using a -J modification to the command like this.\
Throughout this user guide, we will show the commands in Cisco XR style.
At this point, NSO only knows the address, port, and authentication information of the devices. This management information was loaded to NSO by the setup utility. It also tells NSO how to communicate with the devices by using NETCONF, SNMP, Cisco IOS CLI, etc. However, at this point, the actual configuration of the individual devices is unknown.\
Let us analyze the above CLI command. First of all, when you start the NSO CLI it starts in operational mode, so to show configuration data, you have to explicitly run show running-config.
NSO manages a list of devices, each device is reached by the path devices device "name" . You can use standard tab completion in the CLI to learn this.
The address and port fields tells NSO where to connect to the device. For now, they all live in local host with different ports. The device-type structure tells NSO it is a CLI device and the specific CLI is supported by the Network Element Driver (NED) cisco-ios. A more detailed explanation of how to configure the device-type structure and how to choose NEDs will be addressed later in this guide.
So now NSO can try to connect to the devices:
NSO does not need to have the connections active continuously, instead, NSO will establish a connection when needed and connections are pooled to conserve resources. At this time, NSO can read the configurations from the devices and populate the configuration database, CDB.
The following command will synchronize the configurations of the devices with the CDB and respond with true if successful:
The NSO data store, CDB, will store the configuration for every device at the path devices device "name" config, everything after this path is the configuration in the device. NSO keeps this synchronized. The synchronization is managed with the following principles:
At initialization, NSO can discover the configuration as shown above.
The modus operandi when using NSO to perform configuration changes is that the network engineer uses NSO (CLI, WebUI, REST,...) to modify the representation in NSO CDB. The changes are committed to the network as a transaction that includes the actual devices. Only if all changes happen on the actual devices, will it be committed to the NSO data store. The transaction also covers the devices so if any of the devices participating in the transaction fails, NSO will roll back the configuration changes on all modified devices. This works even in the case of devices that do not natively support roll-back like Cisco IOS CLI.
NSO can detect out-of-band changes and reconcile them by either updating the CDB or modifying the configuration on the devices to reflect the currently stored configuration.
NSO only needs to be synchronized with the devices in the event of a change being made outside of NSO. Changes made using NSO will reflected in both the CDB and the devices. The following actions do not need to be taken:
Perform configuration change via NSO.
Perform sync-from action.
The above incorrect (or not necessary) sequence stems from the assumption that the NSO CLI talks directly to the devices. This is not the case; the northbound interfaces in NSO modify the configuration in the NSO data store, NSO calculates a minimum difference between the current configuration and the new configuration, giving only the changes to the configuration to the NEDS that runs the commands to the devices. All this as one single change-set.
View the configuration of the c0 device using the command:
Or, show a particular piece of configuration from several devices:
Or, show a particular piece of configuration from all devices:
The CLI can pipe commands, try TAB after | to see various pipe targets:
The above command shows the router config of all devices as XML and then saves it to a file router.xml.
To change the configuration, enter configure mode.\
Change or add some configuration across the devices, for example:\
It is important to understand how NSO applies configuration changes to the network. At this point, the changes are local to NSO, no configurations have been sent to the devices yet. Since the NSO Configuration Database, CDB is in sync with the network, NSO can calculate the minimum diff to apply the changes to the network.
The command below compares the ongoing changes with the running database:
It is possible to dry-run the changes to see the native Cisco CLI output (in this case almost the same as above):
The changes can be committed to the devices and the NSO CDB simultaneously with a single commit. In the commit command below, we pipe to details to understand the actions being taken.
Changes are committed to the devices and the NSO database as one transaction. If any of the device configurations fail, all changes will be rolled back and the devices will be left in the state that they were in before the commit and the NSO CDB will not be updated.
There are numerous options to the commit command which will affect the behavior of the atomic transactions:
As seen by the details output, NSO stores a roll-back file for every commit so that the whole transaction can be rolled back manually. The following is an example of a rollback file:
(Viewing files as an operational command, prefixing a command in configuration mode with do executes in operational mode.) To perform a manual rollback, first load the rollback file:
apply-rollback-file by default restores to that saved configuration, adding selective as a parameter allows you to just roll back the delta in that specific rollback file. Show the differences:
Commit the rollback:
A trace log can be created to see what is going on between NSO and the device CLI enable trace. Use the following command to enable trace:
Note that the trace settings only take effect for new connections, so is important to disconnect the current connections. Make a change to for example c0:
Note the use of the command commit dry-run outformat native. This will display the net result device commands that will be generated over the native interface without actually committing them to the CDB or the devices. In addition, there is the possibility to append the reverse flag that will display the device commands for getting back to the current running state in the network if the commit is successfully executed.
Exit from the NSO CLI and return to the Unix Shell. Inspect the CLI trace:
As seen above, ranges can be used to send configuration commands to several devices. Device groups can be created to allow for group actions that do not require naming conventions. A group can reference any number of devices. A device can be part of any number of groups, and groups can be hierarchical.
The command sequence below creates a group of core devices and a group with all devices. Note that you can use tab completion when adding the device names to the group. Also, note that it requires configuration mode. (If you are still in the Unix Shell from the steps above, do $ncs_cli -C -u admin).
Note well the do show which shows the operational data for the groups. Device groups have a member attribute that shows all member devices, flattening any group members.
Device groups can contain different devices as well as devices from different vendors. Configuration changes will be committed to each device in its native language without needing to be adjusted in NSO.
You can, for example, at this point use the group to check if all core are in sync:
Assume that we would like to manage permit lists across devices. This can be achieved by defining templates and applying them to device groups. The following CLI sequence defines a tiny template, called community-list :
This can now be applied to a device group:
What if the device group core contained different vendors? Since the configuration is written in IOS the above template would not work on Juniper devices. Templates can be used on different device types (read NEDs) by using a prefix for the device model. The template would then look like:
The above indicates how NSO manages different models for different device types. When NSO connects to the devices, the NED checks the device type and revision and returns that to NSO. This can be inspected (note, in operational mode):
So here we see that c0 uses a tailf-ned-cisco-ios module which tells NSO which data model to use for the device. Every NED package comes with a YANG data model for the device (except for third-party YANG NED for which the YANG device model must be downloaded and fixed before it can be used). This renders the NSO data store (CDB) schema, the NSO CLI, WebUI, and southbound commands.
The model introduces namespace prefixes for every configuration item. This also resolves issues around different vendors using the same configuration command for different configuration elements. Note that every item is prefixed with ios:
Another important question is how to control if the template merges the list or replaces the list. This is managed via tags. The default behavior of templates is to merge the configuration. Tags can be inserted at any point in the template. Tag values are merge, replace, delete, create and nocreate.
Assume that c0 has the following configuration:
If we apply the template the default result would be:
We could change the template in the following way to get a result where the permit list would be replaced rather than merged. When working with tags in templates, it is often helpful to view the template as a tree rather than a command view. The CLI has a display option for showing a curly-braces tree view that corresponds to the data-model structure rather than the command set. This makes it easier to see where to add tags.
Different tags can be added across the template tree. If we now apply the template to the device c0 which already have community lists, the following happens:
Any existing values in the list are replaced in this case. The following tags are available:
merge (default): the template changes will be merged with the existing template.
replace: the template configuration will be replaced by the new configuration.
create: the template will create those nodes that do not exist. If a node already exists this will result in an error.
nocreate: the merge will only affect configuration items that already exist in the template. It will never create the configuration with this tag, or any associated commands inside it. It will only modify existing configuration structures.
delete: delete anything from this point.
Note that a template can have different tags along the tree nodes.
A problem with the above template is that every value is hard-coded. What if you wanted a template where the community-list name and permit-list value are variables passed to the template when applied? Any part of a template can be a variable, (or actually an XPATH expression). We can modify the template to use variables in the following way:
The template now requires two parameters when applied (tab completion will prompt for the variable):
Note, that the replace tag was still part of the template and it would delete any existing community lists, which is probably not the desired outcome in the general case.
The template mechanism described so far is "fire-and-forget". The templates do not have any memory of what happened to the network, or which devices they touched. A user can modify the templates without anything happening to the network until an explicit apply-template action is performed. (Templates are of course, as all configuration changes, applied as a transaction). NSO also supports service templates that are more advanced in many ways, more information on this will be presented later in this guide.
Also, note that device templates have some additional restrictions on the values that can be supplied when applying the template. In particular, a value must either be a number or a single-quoted string. It is currently not possible to specify a value that contains a single quote (').
To make sure that configuration is applied according to site or corporate rules, you can use policies. Policies are validated at every commit, they can be of type error that implies that the change cannot go through or a warning which means that you have to confirm a configuration that gives a warning.
A policy is composed of:
Policy name.
Iterator: loop over a path in the model, for example, all devices, all services of a specific type.
Expression: a boolean expression that must be true for every node returned from the iterator, for example, SNMP must be turned on.
Warning or error: a message displayed to the user. If it is of the type warning, the user can still commit the change, if of type error the change cannot be made.
An example is shown below:
Now, if we try to delete a class-map a, we will get a policy violation:
The {name} variable refers to the node set from the iterator. This node-set will be the list of devices in NSO and the devices have an attribute called 'name'.
To understand the syntax for the expressions a pipe target in the CLI can be used:
To debug policies look at the end of logs/xpath.trace. This file will show all validated XPATH expressions and any errors.
Validation scripts can also be defined in Python, see more about that in Plug-and-Play Scripting.
In reality, network engineers will still modify configurations using other tools like out-of-band CLI or other management interfaces. It is important to understand how NSO manages this. The NSO network simulator supports CLI towards the devices. For example, we can use the IOS CLI on say c0 and delete a permit-list.
From the UNIX shell, start a CLI session towards c0.
Start the NSO CLI again:
NSO detects if its configuration copy in CDB differs from the configuration in the device. Various strategies are used depending on device support: transaction IDs, time stamps, and configuration hash-sums. For example, an NSO user can request a check-sync operation:
NSO can also compare the configurations with the CDB and show the difference:
At this point, we can choose if we want to use the configuration stored in the CDB as the valid configuration or the configuration on the device:
In the above example, we chose to overwrite the device configuration from NSO.
NSO will also detect out-of-sync when committing changes. In the following scenario, a local c0 CLI user adds an interface. Later the NSO user tries to add an interface:
At this point, we have two diffs:
The device and NSO CDB (devices device compare-config).
The ongoing transaction and CDB (show configuration).
To resolve this, you can choose to synchronize the configuration between the devices and the CDB before committing. There is also an option to over-ride the out-of-sync check:
Or:
As noted before, all changes are applied as complete transactions of all configurations on all of the devices. Either all configuration changes are completed successfully or all changes are removed entirely. Consider a simple case where one of the devices is not responding. For the transaction manager, an error response from a device or a non-responding device, are both errors and the transaction should automatically rollback to the state before the commit command was issued.
Stop c0:
Go back to the NSO CLI and perform a configuration change over c0 and c1:
NSO sends commands to all devices in parallel, not sequentially. If any of the devices fail to accept the changes or report an error, NSO will issue a rollback to the other devices. Note, that this works also for non-transactional devices like IOS CLI and SNMP. This works even for non-symmetrical cases where the rollback command sequence is not just the reverse of the commands. NSO does this by treating the rollback as it would any other configuration change. NSO can use the current configuration and previous configuration and generate the commands needed to roll back from the configuration changes.
The diff configuration is still in the private CLI session, it can be restored, modified (if the error was due to something in the config), or in some cases, fix the device.
NSO is not a best-effort configuration management system. The error reporting coupled with the ability to completely rollback failed changes to the devices, ensures that the configurations stored in the CDB and the configurations on the devices are always consistent and that no failed or orphan configurations are left on the devices.
First of all, if the above was not a multi-device transaction, meaning that the change should be applied independently device per device, then it is just a matter of performing the commit between the devices.
Second, NSO has a commit flag commit-queue async or commit-queue sync. The commit queue should primarily be used for throughput reasons when doing configuration changes in large networks. Atomic transactions come with a cost, the critical section of the database is locked when committing the transaction on the network. So, in cases where there are northbound systems of NSO that generate many simultaneous large configuration changes these might get queued. The commit queue will send the device commands after the lock has been released, so the database lock is much shorter. If any device fails, an alarm will be raised.
Go to the UNIX shell, start the device, and monitor the commit queue:
Devices can also be pre-provisioned, this means that the configuration can be prepared in NSO and pushed to the device when it is available. To illustrate this, we can start by adding a new device to NSO that is not available in the network simulator:
Above, we added a new device to NSO with an IP address local host, and port 10030. This device does not exist in the network simulator. We can tell NSO not to send any commands southbound by setting the admin-state to southbound-locked (actually the default). This means that all configuration changes will succeed, and the result will be stored in CDB. At any point in time when the device is available in the network, the state can be changed and the complete configuration pushed to the new device. The CLI sequence below also illustrates a powerful copy configuration command that can copy any configuration from one device to another. The from and to paths are separated by the keyword to.
As shown above, check-sync operations will tell the user that the device is southbound locked. When the device is available in the network, the device can be synchronized with the current configuration in the CDB using the sync-to action.
Different users or management tools can of course run parallel sessions to NSO. All ongoing sessions have a logical copy of CDB. An important case needs to be understood if there is a conflict when multiple users attempt to modify the same device configuration at the same time with different changes. First, let's look at the CLI sequence below, user admin to the left, user joe to the right.
There is no conflict in the above sequence, community is a list so both joe and admin can add items to the list. Note that user joe gets information about the user admin committing.
On the other hand, if two users modify an ordered-by user list in such a way that one user rearranges the list, along with other non-conflicting modifications, and one user deletes the entire list, the following happens:
In this case, joe commits a change to access-list after admin and a conflict message is displayed. Since the conflict is non-resolvable, the transaction has to be reverted. To reapply the changes made by joe to logging in a new transaction, the following commands are entered:
In this case, joe tries to reapply the changes made in the previous transaction and since access-list 10 has been removed, the move command will fail when applied by the reapply-commands command. Since the mode is best-effort, the next command will be processed. The changes to logging will succeed and joe then commits the transaction.
