Perform system management tasks on your NSO deployment.
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Manage purchase and licensing of Cisco software.
Cisco Smart Licensing is a cloud-based approach to licensing and it simplifies the purchase, deployment, and management of Cisco software assets. Entitlements are purchased through a Cisco account via Cisco Commerce Workspace (CCW) and are immediately deposited into a Smart Account for usage. This eliminates the need to install license files on every device. Products that are smart-enabled communicate directly to Cisco to report consumption.
Cisco Smart Software Manager (CSSM) enables the management of software licenses and Smart Account from a single portal. The interface allows you to activate your product, manage entitlements, and renew and upgrade software.
A functioning Smart Account is required to complete the registration process. For detailed information about CSSM, see Cisco Smart Software Manager.
A Virtual Account exists as a sub-account within the Smart Account. Virtual Accounts are a customer-defined structure based on organizational layout, business function, geography, or any defined hierarchy. They are created and maintained by the Smart Account administrator(s).
Visit Cisco Cisco Software Central to learn about how to create and manage Smart Accounts.
The creation of a new Smart Account is a one-time event and subsequent management of users is a capability provided through the tool. To request a Smart Account, visit Cisco Cisco Software Central and take the following steps:
After logging in select Request a Smart Account in the Administration section.
Select the type of Smart Account to create. There are two options: (a) Individual Smart Account requiring agreement to represent your company. By creating this Smart Account you agree to authorization to create and manage product and service entitlements, users, and roles on behalf of your organization. (b) Create the account on behalf of someone else.
Provide the required domain identifier and the preferred account name.
The account request will be pending approval of the Account Domain Identifier. A subsequent email will be sent to the requester to complete the setup process.
Smart Account user management is available in the Administration section of Cisco Cisco Software Central. Take the following steps to add a new user to a Smart Account:
After logging in Select Manage Smart Account in the Administration section.
Choose the Users tab.
Select New User and follow the instructions in the wizard to add a new user.
To create a new token, log into CSSM and select the appropriate Virtual Account.
Click on the Smart Licenses link to enter CSSM.
In CSSM click on New Token.
Follow the dialog to provide a description, expiration, and export compliance applicability before accepting the terms and responsibilities. Click on Create Token to continue.
Click on the new token.
Copy the token from the dialogue window into your clipboard.
Go to the NSO CLI and provide the token to the license smart register idtoken
command:
If ncs.conf
contains configuration for any of java-executable, java-options, override-url/url, or proxy/url under the configure path /ncs-config/smart-license/smart-agent/
any corresponding configuration done via the CLI is ignored.
The smart licensing component of NSO runs its own Java virtual machine. Usually, the default Java options are sufficient:
If you for some reason need to modify the Java options, remember to include the default values as found in the YANG model.
show
and debug
Commandsshow license all
: Displays all information.
show license status
: Displays status information.
show license summary
: Displays summary.
show license tech
: Displays license tech support information.
show license usage
: Displays usage information.
debug smart_lic all
: All available Smart Licensing debug flags.
Perform NSO system management activities.
NSO consists of a number of modules and executable components. These executable components will be referred to by their command-line name, e.g. ncs
, ncs-netsim
, ncs_cli
, etc. ncs
is used to refer to the executable, the running daemon.
When NSO is started, it reads its configuration file and starts all subsystems configured to start (such as NETCONF, CLI, etc.).
By default, NSO starts in the background without an associated terminal. It is recommended to use a System Install when installing NSO for production deployment. This will create an init
script that starts NSO when the system boots, and makes NSO start the service manager.
NSO is licensed using Cisco Smart Licensing. To register your NSO instance, you need to enter a token from your Cisco Smart Software Manager account. For more information on this topic, see Cisco Smart Licensing.
NSO is configured in the following two ways:
Through its configuration file, ncs.conf
.
Through whatever data is configured at run-time over any northbound, for example, turning on trace using the CLI.
ncs.conf
FileThe configuration file ncs.conf
is read at startup and can be reloaded. Below is an example of the most common settings. It is included here as an example and should be self-explanatory. See ncs.conf in Manual Pages for more information. Important configuration settings are:
load-path
: where NSO should look for compiled YANG files, such as data models for NEDs or Services.
db-dir
: the directory on disk that CDB uses for its storage and any temporary files being used. It is also the directory where CDB searches for initialization files. This should be a local disk and not NFS mounted for performance reasons.
Various log settings.
AAA configuration.
Rollback file directory and history length.
Enabling north-bound interfaces like REST, and WebUI.
Enabling of High-Availability mode.
The ncs.conf
file is described in the NSO Manual Pages. There is a large number of configuration items in ncs.conf
, most of them have sane default values. The ncs.conf
file is an XML file that must adhere to the tailf-ncs-config.yang
model. If we start the NSO daemon directly, we must provide the path to the NCS configuration file as in:
However, in a System Install, the init
script is typically used to start NSO, and it will pass the appropriate options to the ncs
command. Thus NSO is started with the command:
It is possible to edit the ncs.conf
file, and then tell NSO to reload the edited file without restarting the daemon as in:
This command also tells NSO to close and reopen all log files, which makes it suitable to use from a system like logrotate
.
In this section, some of the important configuration settings will be described and discussed.
NSO allows access through a number of different interfaces, depending on the use case. In the default configuration, clients can access the system locally through an unauthenticated IPC socket (with the ncs*
family of commands, port 4569) and plain (non-HTTPS) HTTP web server (port 8080). Additionally, the system enables remote access through SSH-secured NETCONF and CLI (ports 2022 and 2024).
We strongly encourage you to review and customize the exposed interfaces to your needs in the ncs.conf
configuration file. In particular, set:
/ncs-config/webui/match-host-name
to true
.
/ncs-config/webui/server-name
to the hostname of the server.
If you decide to allow remote access to the web server, also make sure you use TLS-secured HTTPS instead of HTTP. Not doing so exposes you to security risks.
Using /ncs-config/webui/match-host-name = true
requires you to use the configured hostname when accessing the server. Web browsers do this automatically but you may need to set the Host
header when performing requests programmatically using an IP address instead of the hostname.
To additionally secure IPC access, refer to Restricting Access to the IPC port.
For more details on individual interfaces and their use, see Northbound APIs.
Let's look at all the settings that can be manipulated through the NSO northbound interfaces. NSO itself has a number of built-in YANG modules. These YANG modules describe the structure that is stored in CDB. Whenever we change anything under, say /devices/device
, it will change the CDB, but it will also change the configuration of NSO. We call this dynamic configuration since it can be changed at will through all northbound APIs.
We summarize the most relevant parts below:
tailf-ncs.yang
ModuleThis is the most important YANG module that is used to control and configure NSO. The module can be found at: $NCS_DIR/src/ncs/yang/tailf-ncs.yang
in the release. Everything in that module is available through the northbound APIs. The YANG module has descriptions for everything that can be configured.
tailf-common-monitoring2.yang
and tailf-ncs-monitoring2.yang
are two modules that are relevant to monitoring NSO.
NSO has a built-in SSH server which makes it possible to SSH directly into the NSO daemon. Both the NSO northbound NETCONF agent and the CLI need SSH. To configure the built-in SSH server we need a directory with server SSH keys - it is specified via /ncs-config/aaa/ssh-server-key-dir
in ncs.conf
. We also need to enable /ncs-config/netconf-north-bound/transport/ssh
and /ncs-config/cli/ssh
in ncs.conf
. In a System Install, ncs.conf
is installed in the "config directory", by default /etc/ncs
, with the SSH server keys in /etc/ncs/ssh
.
There are also configuration parameters that are more related to how NSO behaves when talking to the devices. These reside in devices global-settings
.
Users are configured at the path aaa authentication users
.
Access control, including group memberships, is managed using the NACM model (RFC 6536).
Adding a user includes the following steps:
Create the user: admin@ncs(config)# aaa authentication users user <user-name>
.
Add the user to a NACM group: admin@ncs(config)# nacm groups <group-name> admin user-name <user-name>
.
Verify/change access rules.
It is likely that the new user also needs access to work with device configuration. The mapping from NSO users and corresponding device authentication is configured in authgroups
. So, the user needs to be added there as well.
If the last step is forgotten, you will see the following error:
This section describes how to monitor NSO. See also NSO Alarms.
Use the command ncs --status
to get runtime information on NSO.
Checking the overall status of NSO can be done using the shell:
Or, in the CLI:
For details on the output see $NCS_DIR/src/yang/tailf-common-monitoring2.yang
.
Below is an overview of the output:
It is also important to look at the packages that are loaded. This can be done in the CLI with:
NSO runs the following processes:
The daemon: ncs.smp
: this is the NCS process running in the Erlang VM.
Java VM: com.tailf.ncs.NcsJVMLauncher
: service applications implemented in Java run in this VM. There are several options on how to start the Java VM, it can be monitored and started/restarted by NSO or by an external monitor. See the ncs.conf(5) Manual Page and the java-vm
settings in the CLI.
Python VMs: NSO packages can be implemented in Python. The individual packages can be configured to run a VM each or share a Python VM. Use the show python-vm status current
to see current threads and show python-vm status start
to see which threads were started at startup time.
NSO has extensive logging functionality. Log settings are typically very different for a production system compared to a development system. Furthermore, the logging of the NSO daemon and the NSO Java VM/Python VM is controlled by different mechanisms. During development, we typically want to turn on the developer-log
. The sample ncs.conf
that comes with the NSO release has log settings suitable for development, while the ncs.conf
created by a System Install are suitable for production deployment.
NSO logs in /logs
in your running directory, (depends on your settings in ncs.conf
). You might want the log files to be stored somewhere else. See man ncs.conf
for details on how to configure the various logs. Below is a list of the most useful log files:
ncs.log
: NCS daemon log. See Log Messages and Formats. Can be configured to Syslog.
ncserr.log.1
, ncserr.log.idx
, ncserr.log.siz
: if the NSO daemon has a problem. this contains debug information relevant to support. The content can be displayed with ncs --printlog ncserr.log
.
audit.log
: central audit log covering all northbound interfaces. See Log Messages and Formats. Can be configured to Syslog.
localhost:8080.access
: all HTTP requests to the daemon. This is an access log for the embedded Web server. This file adheres to the Common Log Format, as defined by Apache and others. This log is not enabled by default and is not rotated, i.e. use logrotate(8). Can be configured to Syslog.
devel.log
: developer-log is a debug log for troubleshooting user-written code. This log is enabled by default and is not rotated, i.e. use logrotate(8). This log shall be used in combination with the java-vm
or python-vm
logs. The user code logs in the VM logs and the corresponding library logs in devel.log
. Disable this log in production systems. Can be configured to Syslog.
You can manage this log and set its logging level in ncs.conf
.
ncs-java-vm
.log
, ncs-python-vm.log
: logger for code running in Java or Python VM, for example, service applications. Developers writing Java and Python code use this log (in combination with devel.log) for debugging. Both Java and Python log levels can be set from their respective VM settings in, for example, the CLI.
netconf.log
, snmp.log
: Log for northbound agents. Can be configured to Syslog.
rollbackNNNNN
: All NSO commits generate a corresponding rollback file. The maximum number of rollback files and file numbering can be configured in ncs.conf
.
xpath.trace
: XPATH is used in many places, for example, XML templates. This log file shows the evaluation of all XPATH expressions and can be enabled in the ncs.conf
.
To debug XPATH for a template, use the pipe target debug
in the CLI instead.
ned-cisco-ios-xr-pe1.trace
(for example): if device trace is turned on a trace file will be created per device. The file location is not configured in ncs.conf
but is configured when the device trace is turned on, for example in the CLI.
Progress trace log: When a transaction or action is applied, NSO emits specific progress events. These events can be displayed and recorded in a number of different ways, either in CLI with the pipe target details
on a commit, or by writing it to a log file. You can read more about it in the Progress Trace.
Transaction error log: log for collecting information on failed transactions that lead to either a CDB boot error or a runtime transaction failure. The default is false
(disabled). More information about the log is available in the Manual Pages under Configuration Parameters (see logs/transaction-error-log
).
Upgrade log: log containing information about CDB upgrade. The log is enabled by default and not rotated (i.e., use logrotate). With the NSO example set, the following examples populate the log in the logs/upgrade.log
file: examples.ncs/development-guide/ned-upgrade/yang-revision
, examples.ncs/development-guide/high-availability/upgrade-basic
, examples.ncs/development-guide/high-availability/upgrade-cluster
, and examples.ncs/getting-started/developing-with-ncs/14-upgrade-service
. More information about the log is available in the Manual Pages under Configuration Parameters (see logs/upgrade-log)
.
NSO can syslog to a local Syslog. See man ncs.conf
how to configure the Syslog settings. All Syslog messages are documented in Log Messages. The ncs.conf
also lets you decide which of the logs should go into Syslog: ncs.log, devel.log, netconf.log, snmp.log, audit.log, WebUI access log
. There is also a possibility to integrate with rsyslog
to log the NCS, developer, audit, netconf, SNMP, and WebUI access logs to syslog with the facility set to daemon in ncs.conf
. For reference, see the upgrade-l2 example, located in examples.ncs/development-guide/high-availability/hcc
.
Below is an example of Syslog configuration:
Log messages are described on the link below:
NSO generates alarms for serious problems that must be remedied. Alarms are available over all the northbound interfaces and exist at the path /alarms
. NSO alarms are managed as any other alarms by the general NSO Alarm Manager, see the specific section on the alarm manager in order to understand the general alarm mechanisms.
The NSO alarm manager also presents a northbound SNMP view, alarms can be retrieved as an alarm table, and alarm state changes are reported as SNMP Notifications. See the "NSO Northbound" documentation on how to configure the SNMP Agent.
This is also documented in the example /examples.ncs/getting-started/using-ncs/5-snmp-alarm-northbound
.
Alarms are described on the link below:
NSO can issue a unique Trace ID per northbound request, visible in logs and trace headers. This Trace ID can be used to follow the request from service invocation to configuration changes pushed to any device affected by the change. The Trace ID may either be passed in from an external client or generated by NSO.
Trace ID is enabled by default, and can be turned off by adding the following snippet to NSO.conf:
Trace ID is propagated downwards in LSA setups and is fully integrated with commit queues.
Trace ID can be passed to NSO over NETCONF, RESTCONF, JSON-RPC, or CLI as a commit parameter.
If Trace ID is not given as a commit parameter, NSO will generate one if the feature is enabled. This generated Trace ID will be on the form UUID version 4.
For RESTCONF requests, this generated Trace ID will be communicated back to the requesting client as an HTTP header called X-Cisco-NSO-Trace-ID
. The trace-id
query parameter can also be used with RPCs and actions to relay a trace-id from northbound requests.
For NETCONF, the Trace ID will be returned as an attribute called trace-id
.
Trace ID will appear in relevant log entries and trace file headers on the form trace-id=...
.
This section describes a number of disaster scenarios and recommends various actions to take in the different disaster variants.
CDB keeps its data in four files A.cdb
, C.cdb
, O.cdb
and S.cdb
. If NSO is stopped, these four files can be copied, and the copy is then a full backup of CDB.
Furthermore, if neither files exist in the configured CDB directory, CDB will attempt to initialize from all files in the CDB directory with the suffix .xml
.
Thus, there exist two different ways to re-initiate CDB from a previously known good state, either from .xml
files or from a CDB backup. The .xml
files would typically be used to reinstall factory defaults whereas a CDB backup could be used in more complex scenarios.
If the S.cdb
file has become inconsistent or has been removed, all commit queue items will be removed, and devices not yet processed out of sync. For such an event, appropriate alarms will be raised on the devices and any service instance that has unprocessed device changes will be set in the failed state.
When NSO starts and fails to initialize, the following exit codes can occur:
Exit codes 1 and 19 mean that an internal error has occurred. A text message should be in the logs, or if the error occurred at startup before logging had been activated, on standard error (standard output if NSO was started with --foreground --verbose
). Generally, the message will only be meaningful to the NSO developers, and an internal error should always be reported to support.
Exit codes 2 and 3 are only used for the NCS control commands (see the section COMMUNICATING WITH NCS in the ncs(1) in Manual Pages manual page) and mean that the command failed due to timeout. Code 2 is used when the initial connect to NSO didn't succeed within 5 seconds (or the TryTime
if given), while code 3 means that the NSO daemon did not complete the command within the time given by the --timeout
option.
Exit code 10 means that one of the init files in the CDB directory was faulty in some way — further information in the log.
Exit code 11 means that the CDB configuration was changed in an unsupported way. This will only happen when an existing database is detected, which was created with another configuration than the current in ncs.conf
.
Exit code 13 means that the schema change caused an upgrade, but for some reason, the upgrade failed. Details are in the log. The way to recover from this situation is either to correct the problem or to re-install the old schema (fxs
) files.
Exit code 14 means that the schema change caused an upgrade, but for some reason the upgrade failed, corrupting the database in the process. This is rare and usually caused by a bug. To recover, either start from an empty database with the new schema, or re-install the old schema files and apply a backup.
Exit code 15 means that A.cdb
or C.cdb
is corrupt in a non-recoverable way. Remove the files and re-start using a backup or init files.
Exit code 16 means that CDB ran into an unrecoverable file error (such as running out of space on the device while performing journal compaction).
Exit code 20 means that NSO failed to bind a socket.
Exit code 21 means that some NSO configuration file is faulty. More information is in the logs.
Exit code 22 indicates an NSO installation-related problem, e.g., that the user does not have read access to some library files, or that some file is missing.
If the NSO daemon starts normally, the exit code is 0.
If the AAA database is broken, NSO will start but with no authorization rules loaded. This means that all write access to the configuration is denied. The NSO CLI can be started with a flag ncs_cli --noaaa
that will allow full unauthorized access to the configuration.
NSO attempts to handle all runtime problems without terminating, e.g., by restarting specific components. However, there are some cases where this is not possible, described below. When NSO is started the default way, i.e. as a daemon, the exit codes will of course not be available, but see the --foreground
option in the ncs(1) Manual Page.
Out of memory: If NSO is unable to allocate memory, it will exit by calling abort(3). This will generate an exit code, as for reception of the SIGABRT signal - e.g. if NSO is started from a shell script, it will see 134, as the exit code (128 + the signal number).
Out of file descriptors for accept(2): If NSO fails to accept a TCP connection due to lack of file descriptors, it will log this and then exit with code 25. To avoid this problem, make sure that the process and system-wide file descriptor limits are set high enough, and if needed configure session limits in ncs.conf
. The out-of-file descriptors issue may also manifest itself in that applications are no longer able to open new file descriptors.
In many Linux systems, the default limit is 1024, but if we, for example, assume that there are four northbound interface ports, CLI, RESTCONF, SNMP, WebUI/JSON-RPC, or similar, plus a few hundred IPC ports, x 1024 == 5120. But one might as well use the next power of two, 8192, to be on the safe side.
Several application issues can contribute to consuming extra ports. In the scope of an NSO application that could, for example, be a script application that invokes CLI command or a callback daemon application that does not close the connection socket as it should.
A commonly used command for changing the maximum number of open file descriptors is ulimit -n [limit]
. Commands such as netstat
and lsof
can be useful to debug file descriptor-related issues.
When the system is updated, NSO executes a two-phase commit protocol towards the different participating databases including CDB. If a participant fails in the commit()
phase although the participant succeeded in the preparation phase, the configuration is possibly in an inconsistent state.
When NSO considers the configuration to be in an inconsistent state, operations will continue. It is still possible to use NETCONF, the CLI, and all other northbound management agents. The CLI has a different prompt which reflects that the system is considered to be in an inconsistent state and also the Web UI shows this:
The MAAPI API has two interface functions that can be used to set and retrieve the consistency status, those are maapi_set_running_db_status()
and maapi_get_running_db_status()
corresponding. This API can thus be used to manually reset the consistency state. The only alternative to reset the state to a consistent state is by reloading the entire configuration.
All parts of the NSO installation can be backed up and restored with standard file system backup procedures.
The most convenient way to do backup and restore is to use the ncs-backup
command. In that case, the following procedure is used.
NSO Backup backs up the database (CDB) files, state files, config files, and rollback files from the installation directory. To take a complete backup (for disaster recovery), use:
The backup will be stored in the "run directory", by default /var/opt/ncs
, as /var/opt/ncs/backups/ncs-VERSION@DATETIME.backup
.
For more information on backup, refer to the ncs-backup(1) in Manual Pages.
NSO Restore is performed if you would like to switch back to a previous good state or restore a backup.
It is always advisable to stop NSO before performing a restore.
First stop NSO if NSO is not stopped yet.
Restore the backup.
Select the backup to be restored from the available list of backups. The configuration and database with run-time state files are restored in /etc/ncs
and /var/opt/ncs
.
Start NSO.
NSO supports creating rollback files during the commit of a transaction that allows for rolling back the introduced changes. Rollbacks do not come without a cost and should be disabled if the functionality is not going to be used. Enabling rollbacks impacts both the time it takes to commit a change and requires sufficient storage on disk.
Rollback files contain a set of headers and the data required to restore the changes that were made when the rollback was created. One of the header fields includes a unique rollback ID that can be used to address the rollback file independent of the rollback numbering format.
The use of rollbacks from the supported APIs and the CLI is documented in the documentation for the given API.
ncs.conf
Config for RollbackAs described earlier, NSO is configured through the configuration file, ncs.conf
. In that file, we have the following items related to rollbacks:
/ncs-config/rollback/enabled
: If set to true
, then a rollback file will be created whenever the running configuration is modified.
/ncs-config/rollback/directory
: Location where rollback files will be created.
/ncs-config/rollback/history-size
: The number of old rollback files to save.
New users can face problems when they start to use NSO. If you face an issue, reach out to our support team regardless if your problem is listed here or not.
A useful tool in this regard is the ncs-collect-tech-report
tool, which is the Bash script that comes with the product. It collects all log files, CDB backup, and several debug dumps as a TAR file. Note that it works only with a System Install.
Some noteworthy issues are covered here.
If you have trouble starting or running NSO, examples, or the clients you write, here are some troubleshooting tips.
daemon-status
You can see the NSO daemon mode, starting, phase0, phase1, started, stopping. The phase0 and phase1 modes are schema upgrade modes and will appear if you have upgraded any data models.
version
The NSO version.
smp
Number of threads used by the daemon.
ha
The High-Availability mode of the NCS daemon will show up here: secondary
, primary
, relay-secondary
.
internal/callpoints
The next section is callpoints. Make sure that any validation points, etc. are registered. (The ncs-rfs-service-hook
is an obsolete callpoint, ignore this one).
UNKNOWN
code tries to register a call-point that does not exist in a data model.
NOT-REGISTERED
a loaded data model has a call-point but no code has registered.
Of special interest is of course the servicepoints
. All your deployed service models should have a corresponding service-point
. For example:
internal/cdb
The cdb
section is important. Look for any locks. This might be a sign that a developer has taken a CDB lock without releasing it. The subscriber section is also important. A design pattern is to register subscribers to wait for something to change in NSO and then trigger an action. Reactive FASTMAP is designed around that. Validate that all expected subscribers are OK.
loaded-data-models
The next section shows all namespaces and YANG modules that are loaded. If you, for example, are missing a service model, make sure it is loaded.
cli
, netconf
, rest
, snmp
, webui
All northbound agents like CLI, REST, NETCONF, SNMP, etc. are listed with their IP and port. So if you want to connect over REST, for example, you can see the port number here.
patches
Lists any installed patches.
upgrade-mode
If the node is in upgrade mode, it is not possible to get any information from the system over NETCONF. Existing CLI sessions can get system information.
Perform package management tasks.
All user code that needs to run in NSO must be part of a package. A package is basically a directory of files with a fixed file structure or a tar archive with the same directory layout. A package consists of code, YANG modules, etc., that are needed to add an application or function to NSO. Packages are a controlled way to manage loading and versions of custom applications.
Network Element Drivers (NEDs) are also packages. Each NED allows NSO to manage a network device of a specific type. Except for third-party YANG NED packages which do not contain a YANG device model by default (and must be downloaded and fixed before adding to the package), a NED typically contains a device YANG model and the code, specifying how NSO should connect to the device. For NETCONF devices, NSO includes built-in tools to help you build a NED, as described in NED Administration, that you can use if needed. Otherwise, a third-party YANG NED, if available, should be used instead. Vendors, in some cases, provide the required YANG device models but not the entire NED. In practice, all NSO instances use at least one NED. The set of used NED packages depends on the number of different device types the NSO manages.
When NSO starts, it searches for packages to load. The ncs.conf
parameter /ncs-config/load-path
defines a list of directories. At initial startup, NSO searches these directories for packages and copies the packages to a private directory tree in the directory defined by the /ncs-config/state-dir
parameter in ncs.conf
, and loads and starts all the packages found. On subsequent startups, NSO will by default only load and start the copied packages. The purpose of this procedure is to make it possible to reliably load new or updated packages while NSO is running, with a fallback to the previously existing version of the packages if the reload should fail.
In a System Install of NSO, packages are always installed (normally through symbolic links) in the packages
subdirectory of the run directory, i.e. by default /var/opt/ncs/packages
, and the private directory tree is created in the state
subdirectory, i.e. by default /var/opt/ncs/state
.
Loading of new or updated packages (as well as removal of packages that should no longer be used) can be requested via the reload
action - from the NSO CLI:
This request makes NSO copy all packages found in the load path to a temporary version of its private directory, and load the packages from this directory. If the loading is successful, this temporary directory will be made permanent, otherwise, the temporary directory is removed and NSO continues to use the previous version of the packages. Thus when updating packages, always update the version in the load path, and request that NSO does the reload via this action.
If the package changes include modified, added, or deleted .fxs
files or .ccl
files, NSO needs to run a data model upgrade procedure, also called a CDB upgrade. NSO provides a dry-run
option to packages reload
action to test the upgrade without committing the changes. Using a reload dry-run, you can tell if a CDB upgrade is needed or not.
The report all-schema-changes
option of the reload action instructs NSO to produce a report of how the current data model schema is being changed. Combined with a dry run, the report allows you to verify the modifications introduced with the new versions of the packages before actually performing the upgrade.
For a data model upgrade, including a dry run, all transactions must be closed. In particular, users having CLI sessions in configure mode must exit to operational mode. If there are ongoing commit queue items, and the wait-commit-queue-empty
parameter is supplied, it will wait for the items to finish before proceeding with the reload. During this time, it will not allow the creation of any new transactions. Hence, if one of the queue items fails with rollback-on-error
option set, the commit queue's rollback will also fail, and the queue item will be locked. In this case, the reload will be canceled. A manual investigation of the failure is needed in order to proceed with the reload.
While the data model upgrade is in progress, all transactions are closed and new transactions are not allowed. This means that starting a new management session, such as a CLI or SSH connection to the NSO, will also fail, producing an error that the node is in upgrade mode.
By default, the reload
action will (when needed) wait up to 10 seconds for the commit queue to empty (if the wait-commit-queue-empty
parameter is entered) and reload to start.
If there are still open transactions at the end of this period, the upgrade will be canceled and the reload operation will fail. The max-wait-time
and timeout-action
parameters to the action can modify this behavior. For example, to wait for up to 30 seconds, and forcibly terminate any transactions that still remain open after this period, we can invoke the action as:
Thus the default values for these parameters are 10
and fail
, respectively. In case there are no changes to .fxs
or .ccl
files, the reload can be carried out without the data model upgrade procedure, and these parameters are ignored since there is no need to close open transactions.
When reloading packages, NSO will give a warning when the upgrade looks suspicious, i.e., may break some functionality. Note that this is not a strict upgrade validation, but only intended as a hint to the NSO administrator early in the upgrade process that something might be wrong. Currently, the following scenarios will trigger the warnings:
One or more namespaces are removed by the upgrade. The consequence of this is all data belonging to this namespace is permanently deleted from CDB upon upgrade. This may be intended in some scenarios, in which case it is advised to proceed with overriding warnings as described below.
There are source .java
files found in the package, but no matching .class
files in the jars loaded by NSO. This likely means that the package has not been compiled.
There are matching .class
files with modification time older than the source files, which hints that the source has been modified since the last time the package was compiled. This likely means that the package was not re-compiled the last time the source code was changed.
If a warning has been triggered it is a strong recommendation to fix the root cause. If all of the warnings are intended, it is possible to proceed with packages reload force
command.
In some specific situations, upgrading a package with newly added custom validation points in the data model may produce an error similar to no registration found for callpoint NEW-VALIDATION/validate
or simply application communication failure
, resulting in an aborted upgrade. See New Validation Points on how to proceed.
In some cases, we may want NSO to do the same operation as the reload
action at NSO startup, i.e. copy all packages from the load path before loading, even though the private directory copy already exists. This can be achieved in the following ways:
Setting the shell environment variable $NCS_RELOAD_PACKAGES
to true
. This will make NSO do the copy from the load path on every startup, as long as the environment variable is set. In a System Install, NSO must be started via the init
script, and this method must be used, but with a temporary setting of the environment variable:
Giving the option --with-package-reload
to the ncs
command when starting NSO. This will make NSO do the copy from the load path on this particular startup, without affecting the behavior on subsequent startups.
If warnings are encountered when reloading packages at startup using one of the options above, the recommended way forward is to fix the root cause as indicated by the warnings as mentioned before. If the intention is to proceed with the upgrade without fixing the underlying cause for the warnings, it is possible to force the upgrade using NCS_RELOAD_PACKAGES
=force
environment variable or --with-package-reload-force
option.
Always use one of these methods when upgrading to a new version of NSO in an existing directory structure, to make sure that new packages are loaded together with the other parts of the new system.
If it is known in advance that there were no data model changes, i.e. none of the .fxs
or .ccl
files changed, and none of the shared JARs changed in a Java package, and the declaration of the components in the package-meta-data.xml
is unchanged, then it is possible to do a lightweight package upgrade, called package redeploy. Package redeploy only loads the specified package, unlike packages reload which loads all of the packages found in the load-path.
Redeploying a package allows you to reload updated or load new templates, reload private JARs for a Java package, or reload the Python code which is a part of this package. Only the changed part of the package will be reloaded, e.g. if there were no changes to Python code, but only templates, then the Python VM will not be restarted, but only templates reloaded. The upgrade is not seamless however as the old templates will be unloaded for a short while before the new ones are loaded, so any user of the template during this period of time will fail; the same applies to changed Java or Python code. It is hence the responsibility of the user to make sure that the services or other code provided by the package is unused while it is being redeployed.
The package redeploy
will return true
if the package's resulting status after the redeploy is up
. Consequently, if the result of the action is false
, then it is advised to check the operational status of the package in the package list.
Unlike a full packages reload
operation, new NED packages can be loaded into the system without disrupting existing transactions. This is only possible for new packages, since these packages don't yet have any instance data.
The operation is performed through the /packages/add
action. No additional input is necessary. The operation scans all the load-paths for any new NED packages and also verifies the existing packages are still present. If packages are modified or deleted, the operation will fail.
Each NED package defines ned-id
, an identifier that is used in selecting the NED for each managed device. A new NED package is therefore a package with a ned-id value that is not already in use.
In addition, the system imposes some additional constraints, so it is not always possible to add just any arbitrary NED. In particular, NED packages can also contain one or more shared data models, such as NED settings or operational data for private use by the NED, that are not specific to each version of NED package but rather shared between all versions. These are typically placed outside any mount point (device-specific data model), extending the NSO schema directly. So, if a NED defines schema nodes outside any mount point, there must be no changes to these nodes if they already exist.
Adding a NED package with a modified shared data model is therefore not allowed and all shared data models are verified to be identical before a NED package can be added. If they are not, the /packages/add
action will fail and you will have to use the /packages/reload
command.
The command returns true
if the package's resulting status after deployment is up
. Likewise, if the result for a package is false
, then the package was added but its code has not started successfully and you should check the operational status of the package with the show packages package``
PKG
``oper-status
command for additional information. You may then use the /packages/package/redeploy
action to retry deploying the package's code, once you have corrected the error.
In a high-availability setup, you can perform this same operation on all the nodes in the cluster with a single packages ha sync and-add
command.
In a System Install of NSO, management of pre-built packages is supported through a number of actions. This support is not available in a Local Install, since it is dependent on the directory structure created by the System Install. Please refer to the YANG submodule $NCS_DIR/src/ncs/yang/tailf-ncs-software.yang
for the full details of the functionality described in this section.
Actions are provided to list local packages, to fetch packages from the file system, and to install or deinstall packages:
software packages list [...]
: List local packages, categorized into loaded, installed, and installable. The listing can be restricted to only one of the categories - otherwise, each package listed will include the category for the package.
software packages fetch package-from-file``
file
: Fetch a package by copying it from the file system, making it installable.
software packages install package``
package-name
``[...]
: Install a package, making it available for loading via the packages reload
action, or via a system restart with package reload. The action ensures that only one version of the package is installed - if any version of the package is installed already, the replace-existing
option can be used to deinstall it before proceeding with the installation.
software packages deinstall package``
package-name
: Deinstall a package, i.e. remove it from the set of packages available for loading.
There is also an upload
action that can be used via NETCONF or REST to upload a package from the local host to the NSO host, making it installable there. It is not feasible to use in the CLI or Web UI, since the actual package file contents is a parameter for the action. It is also not suitable for very large (more than a few megabytes) packages, since the processing of action parameters is not designed to deal with very large values, and there is a significant memory overhead in the processing of such values.
NSO Packages contain data models and code for a specific function. It might be NED for a specific device, a service application like MPLS VPN, a WebUI customization package, etc. Packages can be added, removed, and upgraded in run-time. A common task is to add a package to NSO to support a new device type or upgrade an existing package when the device is upgraded.
(We assume you have the example up and running from the previous section). Currently installed packages can be viewed with the following command:
So the above command shows that NSO currently has one package, the NED for Cisco IOS.
NSO reads global configuration parameters from ncs.conf
. More on NSO configuration later in this guide. By default, it tells NSO to look for packages in a packages
directory where NSO was started. So in this specific example:
As seen above a package is a defined file structure with data models, code, and documentation. NSO comes with a couple of ready-made packages: $NCS_DIR/packages/
. Also, there is a library of packages available from Tail-f, especially for supporting specific devices.
Assume you would like to add support for Nexus devices to the example. Nexus devices have different data models and another CLI flavor. There is a package for that in $NCS_DIR/packages/neds/nexus
.
We can keep NSO running all the time, but we will stop the network simulator to add the Nexus devices to the simulator.
Add the nexus package to the NSO runtime directory by creating a symbolic link:
The package is now in place, but until we tell NSO to look for package changes nothing happens:
So after the packages reload
operation NSO also knows about Nexus devices. The reload operation also takes any changes to existing packages into account. The data store is automatically upgraded to cater to any changes like added attributes to existing configuration data.
We can now add these Nexus devices to NSO according to the below sequence:
Manage user authentication, authorization, and audit using NSO's AAA mechanism.
Users log into NSO through the CLI, NETCONF, RESTCONF, SNMP, or via the Web UI. In either case, users need to be authenticated. That is, a user needs to present credentials, such as a password or a public key to gain access. As an alternative, for RESTCONF, users can be authenticated via token validation.
Once a user is authenticated, all operations performed by that user need to be authorized. That is, certain users may be allowed to perform certain tasks, whereas others are not. This is called authorization. We differentiate between the authorization of commands and the authorization of data access.
The NSO daemon manages device configuration including AAA information. NSO manages AAA information as well as uses it. The AAA information describes which users may log in, what passwords they have, and what they are allowed to do. This is solved in NSO by requiring a data model to be both loaded and populated with data. NSO uses the YANG module tailf-aaa.yang
for authentication, while ietf-netconf-acm.yang
(NETCONF Access Control Model (NACM), RFC 8341) as augmented by tailf-acm.yang
is used for group assignment and authorization.
The NACM data model is targeted specifically towards access control for NETCONF operations and thus lacks some functionality that is needed in NSO, in particular, support for the authorization of CLI commands and the possibility to specify the context (NETCONF, CLI, etc.) that a given authorization rule should apply to. This functionality is modeled by augmentation of the NACM model, as defined in the tailf-acm.yang
YANG module.
The ietf-netconf-acm.yang
and tailf-acm.yang
modules can be found in $NCS_DIR/src/ncs/yang
directory in the release, while tailf-aaa.yang
can be found in the $NCS_DIR/src/ncs/aaa
directory.
NACM options related to services are modeled by augmentation of the NACM model, as defined in the tailf-ncs-acm.yang
YANG module. The tailf-ncs-acm.yang
can be found in $NCS_DIR/src/ncs/yang
directory in the release.
The complete AAA data model defines a set of users, a set of groups, and a set of rules. The data model must be populated with data that is subsequently used by by NSO itself when it authenticates users and authorizes user data access. These YANG modules work exactly like all other fxs
files loaded into the system with the exception that NSO itself uses them. The data belongs to the application, but NSO itself is the user of the data.
Since NSO requires a data model for the AAA information for its operation, it will report an error and fail to start if these data models cannot be found.
ncs.conf
NSO itself is configured through a configuration file - ncs.conf
. In that file, we have the following items related to authentication and authorization:
/ncs-config/aaa/ssh-server-key-dir
: If SSH termination is enabled for NETCONF or the CLI, the NSO built-in SSH server needs to have server keys. These keys are generated by the NSO install script and by default end up in $NCS_DIR/etc/ncs/ssh
.
It is also possible to use OpenSSH to terminate NETCONF or the CLI. If OpenSSH is used to terminate SSH traffic, this setting has no effect.
/ncs-config/aaa/ssh-pubkey-authentication
: If SSH termination is enabled for NETCONF or the CLI, this item controls how the NSO SSH daemon locates the user keys for public key authentication. See Public Key Login for details.
/ncs-config/aaa/local-authentication/enabled
: The term 'local user' refers to a user stored under /aaa/authentication/users
. The alternative is a user unknown to NSO, typically authenticated by PAM. By default, NSO first checks local users before trying PAM or external authentication.
Local authentication is practical in test environments. It is also useful when we want to have one set of users that are allowed to log in to the host with normal shell access and another set of users that are only allowed to access the system using the normal encrypted, fully authenticated, northbound interfaces of NSO.
If we always authenticate users through PAM, it may make sense to set this configurable to false
. If we disable local authentication, it implicitly means that we must use either PAM authentication or external authentication. It also means that we can leave the entire data trees under /aaa/authentication/users
and, in the case of external authentication, also /nacm/groups
(for NACM) or /aaa/authentication/groups
(for legacy tailf-aaa) empty.
/ncs-config/aaa/pam
: NSO can authenticate users using PAM (Pluggable Authentication Modules). PAM is an integral part of most Unix-like systems.
PAM is a complicated - albeit powerful - subsystem. It may be easier to have all users stored locally on the host, However, if we want to store users in a central location, PAM can be used to access the remote information. PAM can be configured to perform most login scenarios including RADIUS and LDAP. One major drawback with PAM authentication is that there is no easy way to extract the group information from PAM. PAM authenticates users, it does not also assign a user to a set of groups. PAM authentication is thoroughly described later in this chapter.
/ncs-config/aaa/default-group
: If this configuration parameter is defined and if the group of a user cannot be determined, a logged-in user ends up in the given default group.
/ncs-config/aaa/external-authentication
: NSO can authenticate users using an external executable. This is further described later in External Authentication. As an alternative, you may consider using package authentication.
/ncs-config/aaa/external-validation
: NSO can authenticate users by validation of tokens using an external executable. This is very similar to what is further described later in External Token Validation. Where external authentication uses a username and password to authenticate a user, external validation uses a token. The validation script should use the token to authenticate a user and can, optionally, also return a new token to be returned with the result of the request. It is currently only supported for RESTCONF.
/ncs-config/aaa/external-challenge
: NSO has support for multi-factor authentication by sending challenges to a user. Challenges may be sent from any of the external authentication mechanisms but are currently only supported by JSON-RPC and CLI over SSH. This is further described later in External Multi-factor Authentication.
/ncs-config/aaa/package-authentication
: NSO can authenticate users using package authentication. It extends the concept of external authentication by allowing multiple packages to be used for authentication instead of a single executable. This is further described in Package Authentication.
/ncs-config/aaa/single-sign-on
: With this setting enabled, NSO invokes Package Authentication on all requests to HTTP endpoints with the /sso
prefix. This way, Package Authentication packages that require custom endpoints can expose them under the /sso
base route.
For example, a SAMLv2 Single Sign-On (SSO) package needs to process requests to an AssertionConsumerService endpoint, such as /sso/saml/acs
, and therefore requires enabling this setting.
This is a valid authentication method for WEB UI and JSON-RPC interfaces and needs Package Authentication to be enabled as well.
/ncs-config/aaa/single-sign-on/enable-automatic-redirect
: If only one Single Sign-On package is configured (a package with single-sign-on-url
set in package-meta-data.xml
) and also this setting is enabled, NSO automatically redirects all unauthenticated access attempts to the configured single-sign-on-url
.
Depending on the northbound management protocol, when a user session is created in NSO, it may or may not be authenticated. If the session is not yet authenticated, NSO's AAA subsystem is used to perform authentication and authorization, as described below. If the session already has been authenticated, NSO's AAA assigns groups to the user as described in Group Membership, and performs authorization, as described in Authorization.
The authentication part of the data model can be found in tailf-aaa.yang
:
AAA authentication is used in the following cases:
When the built-in SSH server is used for NETCONF and CLI sessions.
For Web UI sessions and REST access.
When the method Maapi.Authenticate()
is used.
NSO's AAA authentication is not used in the following cases:
When NETCONF uses an external SSH daemon, such as OpenSSH.
In this case, the NETCONF session is initiated using the program netconf-subsys
, as described in NETCONF Transport Protocols in Northbound APIs.
When NETCONF uses TCP, as described in NETCONF Transport Protocols in Northbound APIs, e.g. through the command netconf-console
.
When the CLI uses an external SSH daemon, such as OpenSSH, or a telnet daemon.
In this case, the CLI session is initiated through the command ncs_cli. An important special case here is when a user has logged in to the host and invokes the command ncs_cli
from the shell. In NSO deployments, it is crucial to consider this case. If non-trusted users have shell access to the host, the NCS_IPC_ACCESS_FILE
feature as described in Restricting Access to IPC Port must be used.
When SNMP is used, SNMP has its own authentication mechanisms. See NSO SNMP Agent in Northbound APIs.
When the method Maapi.startUserSession()
is used without a preceding call of Maapi.authenticate()
.
When a user logs in over NETCONF or the CLI using the built-in SSH server, with a public key login, the procedure is as follows.
The user presents a username in accordance with the SSH protocol. The SSH server consults the settings for /ncs-config/aaa/ssh-pubkey-authentication
and /ncs-config/aaa/local-authentication/enabled
.
If ssh-pubkey-authentication
is set to local
, and the SSH keys in /aaa/authentication/users/user{$USER}/ssh_keydir
match the keys presented by the user, authentication succeeds.
Otherwise, if ssh-pubkey-authentication
is set to system
, local-authentication
is enabled, and the SSH keys in /aaa/authentication/users/user{$USER}/ssh_keydir
match the keys presented by the user, authentication succeeds.
Otherwise, if ssh-pubkey-authentication
is set to system
and the user /aaa/authentication/users/user{$USER}
does not exist, but the user does exist in the OS password database, the keys in the user's $HOME/.ssh
directory are checked. If these keys match the keys presented by the user, authentication succeeds.
Otherwise, authentication fails.
In all cases the keys are expected to be stored in a file called authorized_keys
(or authorized_keys2
if authorized_keys
does not exist), and in the native OpenSSH format (i.e. as generated by the OpenSSH ssh-keygen
command). If authentication succeeds, the user's group membership is established as described in Group Membership.
This is exactly the same procedure that is used by the OpenSSH server with the exception that the built-in SSH server also may locate the directory containing the public keys for a specific user by consulting the /aaa/authentication/users
tree.
We need to provide a directory where SSH keys are kept for a specific user and give the absolute path to this directory for the /aaa/authentication/users/user/ssh_keydir
leaf. If a public key login is not desired at all for a user, the value of the ssh_keydir
leaf should be set to ""
, i.e. the empty string. Similarly, if the directory does not contain any SSH keys, public key logins for that user will be disabled.
The built-in SSH daemon supports DSA, RSA, and ED25519 keys. To generate and enable RSA keys of size 4096 bits for, say, user "bob", the following steps are required.
On the client machine, as user "bob", generate a private/public key pair as:
Now we need to copy the public key to the target machine where the NETCONF or CLI SSH client runs.
Assume we have the following user entry:
We need to copy the newly generated file id_rsa.pub
, which is the public key, to a file on the target machine called /var/system/users/bob/.ssh/authorized_keys
.
Since the release of OpenSSH 7.0, support of ssh-dss
host and user keys is disabled by default. If you want to continue using these, you may re-enable it using the following options for OpenSSH client:
You can find full instructions at OpenSSH Legacy Options webpage.
Password login is triggered in the following cases:
When a user logs in over NETCONF or the CLI using the built-in SSH server, with a password. The user presents a username and a password in accordance with the SSH protocol.
When a user logs in using the Web UI. The Web UI asks for a username and password.
When the method Maapi.authenticate()
is used.
In this case, NSO will by default try local authentication, PAM, external authentication, and package authentication in that order, as described below. It is possible to change the order in which these are tried, by modifying the ncs.conf
. parameter /ncs-config/aaa/auth-order
. See ncs.conf(5) in Manual Pages for details.
If /aaa/authentication/users/user{$USER}
exists and the presented password matches the encrypted password in /aaa/authentication/users/user{$USER}/password
, the user is authenticated.
If the password does not match or if the user does not exist in /aaa/authentication/users
, PAM login is attempted, if enabled. See PAM for details.
If all of the above fails and external authentication is enabled, the configured executable is invoked. See External Authentication for details.
If authentication succeeds, the user's group membership is established as described in Group Membership.
On operating systems supporting PAM, NSO also supports PAM authentication. Using PAM, authentication with NSO can be very convenient since it allows us to have the same set of users and groups having access to NSO as those that have access to the UNIX/Linux host itself.
If we use PAM, we do not have to have any users or any groups configured in the NSO aaa namespace at all.
PAM is the recommended way to authenticate NSO users.
To configure PAM we typically need to do the following:
Remove all users and groups from the AAA initialization XML file.
Enable PAM in ncs.conf
by adding the following to the AAA section in ncs.conf
. The service
name specifies the PAM service, typically a file in the directory /etc/pam.d
, but may alternatively, be an entry in a file /etc/pam.conf
depending on OS and version. Thus, it is possible to have a different login procedure for NSO than for the host itself.
If PAM is enabled and we want to use PAM for login, the system may have to run as root
. This depends on how PAM is configured locally. However, the default system authentication will typically require root
, since the PAM libraries then read /etc/shadow
. If we don't want to run NSO as root, the solution here is to change the owner of a helper program called $NCS_DIR/lib/ncs/lib/core/pam/priv/epam
and also set the setuid
bit.
As an example, say that we have a user test in /etc/passwd
, and furthermore:
Thus, the test
user is part of the admin
and the operator
groups and logging in to NSO as the test
user through CLI SSH, Web UI, or NETCONF, renders the following in the audit log.
Thus, the test
user was found and authenticated from /etc/passwd
, and the crucial group assignment of the test user was done from /etc/group
.
If we wish to be able to also manipulate the users, their passwords, etc on the device, we can write a private YANG model for that data, store that data in CDB, set up a normal CDB subscriber for that data, and finally when our private user data is manipulated, our CDB subscriber picks up the changes and changes the contents of the relevant /etc
files.
A common situation is when we wish to have all authentication data stored remotely, not locally, for example on a remote RADIUS or LDAP server. This remote authentication server typically not only stores the users and their passwords but also the group information.
If we wish to have not only the users but also the group information stored on a remote server, the best option for NSO authentication is to use external authentication.
If this feature is configured, NSO will invoke the executable configured in /ncs-config/aaa/external-authentication/executable
in ncs.conf
, and pass the username and the clear text password on stdin
using the string notation: "[user;password;]\n"
.
For example, if the user bob
attempts to log in over SSH using the password 'secret', and external authentication is enabled, NSO will invoke the configured executable and write "[bob;secret;]\n"
on the stdin
stream for the executable. The task of the executable is then to authenticate the user and also establish the username-to-groups mapping.
For example, the executable could be a RADIUS client which utilizes some proprietary vendor attributes to retrieve the groups of the user from the RADIUS server. If authentication is successful, the program should write accept
followed by a space-separated list of groups that the user is a member of, and additional information as described below. Again, assuming that bob's password indeed was 'secret', and that bob is a member of the admin
and the lamers
groups, the program should write accept admin lamers $uid $gid $supplementary_gids $HOME
on its standard output and then exit.
There is a general limit of 16000 bytes of output from the externalauth
program.
Thus, the format of the output from an externalauth
program when authentication is successful should be:
"accept $groups $uid $gid $supplementary_gids $HOME\n"
Where:
$groups
is a space-separated list of the group names the user is a member of.
$uid
is the UNIX integer user ID that NSO should use as a default when executing commands for this user.
$gid
is the UNIX integer group ID that NSO should use as a default when executing commands for this user.
$supplementary_gids
is a (possibly empty) space-separated list of additional UNIX group IDs the user is also a member of.
$HOME
is the directory that should be used as HOME for this user when NSO executes commands on behalf of this user.
It is further possible for the program to return a token on successful authentication, by using "accept_token"
instead of "accept"
:
"accept_token $groups $uid $gid $supplementary_gids $HOME $token\n"
Where:
$token
is an arbitrary string. NSO will then, for some northbound interfaces, include this token in responses.
It is also possible for the program to return additional information on successful authentication, by using "accept_info"
instead of "accept"
:
"accept_info $groups $uid $gid $supplementary_gids $HOME $info\n"
Where:
$info
is some arbitrary text. NSO will then just append this text to the generated audit log message (CONFD_EXT_LOGIN).
Yet another possibility is for the program to return a warning that the user's password is about to expire, by using "accept_warning"
instead of "accept"
:
"accept_warning $groups $uid $gid $supplementary_gids $HOME $warning\n"
Where:
$warning
is an appropriate warning message. The message will be processed by NSO according to the setting of /ncs-config/aaa/expiration-warning
in ncs.conf
.
There is also support for token variations of "accept_info"
and "accept_warning"
namely "accept_token_info"
and "accept_token_warning"
. Both "accept_token_info"
and "accept_token_warning"
expect the external program to output exactly the same as described above with the addition of a token after $HOME
:
"accept_token_info $groups $uid $gid $supplementary_gids $HOME $token $info\n"
"accept_token_warning $groups $uid $gid $supplementary_gids $HOME $token $warning\n"
If authentication failed, the program should write "reject"
or "abort"
, possibly followed by a reason for the rejection, and a trailing newline. For example, "reject Bad password\n"
or just "abort\n"
. The difference between "reject"
and "abort"
is that with "reject"
, NSO will try subsequent mechanisms configured for /ncs-config/aaa/auth-order
in ncs.conf
(if any), while with "abort"
, the authentication fails immediately. Thus "abort"
can prevent subsequent mechanisms from being tried, but when external authentication is the last mechanism (as in the default order), it has the same effect as "reject"
.
Supported by some northbound APIs, such as JSON-RPC and CLI over SSH, the external authentication may also choose to issue a challenge:
"challenge $challenge-id $challenge-prompt\n"
The challenge-prompt may be multi-line, why it must be base64 encoded.
For more information on multi-factor authentication, see External Multi-Factor Authentication.
When external authentication is used, the group list returned by the external program is prepended by any possible group information stored locally under the /aaa
tree. Hence when we use external authentication it is indeed possible to have the entire /aaa/authentication
tree empty. The group assignment performed by the external program will still be valid and the relevant groups will be used by NSO when the authorization rules are checked.
When username and password authentication is not feasible, authentication by token validation is possible. Currently, only RESTCONF supports this mode of authentication. It shares all properties of external authentication, but instead of a username and password, it takes a token as input. The output is also almost the same, the only difference is that it is also expected to output a username.
If this feature is configured, NSO will invoke the executable configured in /ncs-config/aaa/external-validation/executable
in ncs.conf
, and pass the token on stdin
using the string notation: "[token;]\n"
.
For example if the user bob
attempts to log over RESTCONF using the token topsecret
, and external validation is enabled, NSO will invoke the configured executable and write "[topsecret;]\n"
on the stdin
stream for the executable.
The task of the executable is then to validate the token, thereby authenticating the user and also establishing the username and username-to-groups mapping.
For example, the executable could be a FUSION client that utilizes some proprietary vendor attributes to retrieve the username and groups of the user from the FUSION server. If token validation is successful, the program should write accept
followed by a space-separated list of groups that the user is a member of, and additional information as described below. Again, assuming that bob
's token indeed was topsecret
, and that bob
is a member of the admin
and the lamers
groups, the program should write accept admin lamers $uid $gid $supplementary_gids $HOME $USER
on its standard output and then exit.
There is a general limit of 16000 bytes of output from the externalvalidation
program.
Thus the format of the output from an externalvalidation
program when token validation authentication is successful should be:
"accept $groups $uid $gid $supplementary_gids $HOME $USER\n"
Where:
$groups
is a space-separated list of the group names the user is a member of.
$uid
is the UNIX integer user ID NSO should use as a default when executing commands for this user.
$gid
is the UNIX integer group ID NSO should use as a default when executing commands for this user.
$supplementary_gids
is a (possibly empty) space-separated list of additional UNIX group IDs the user is also a member of.
$HOME
is the directory that should be used as HOME for this user when NSO executes commands on behalf of this user.
$USER
is the user derived from mapping the token.
It is further possible for the program to return a new token on successful token validation authentication, by using "accept_token"
instead of "accept"
:
"accept_token $groups $uid $gid $supplementary_gids $HOME $USER $token\n"
Where:
$token
is an arbitrary string. NSO will then, for some northbound interfaces, include this token in responses.
It is also possible for the program to return additional information on successful token validation authentication, by using "accept_info"
instead of "accept"
:
"accept_info $groups $uid $gid $supplementary_gids $HOME $USER $info\n"
Where:
$info
is some arbitrary text. NSO will then just append this text to the generated audit log message (CONFD_EXT_LOGIN).
Yet another possibility is for the program to return a warning that the user's password is about to expire, by using "accept_warning"
instead of "accept"
:
"accept_warning $groups $uid $gid $supplementary_gids $HOME $USER $warning\n"
Where:
$warning
is an appropriate warning message. The message will be processed by NSO according to the setting of /ncs-config/aaa/expiration-warning
in ncs.conf
.
There is also support for token variations of "accept_info"
and "accept_warning"
namely "accept_token_info"
and "accept_token_warning"
. Both "accept_token_info"
and "accept_token_warning"
expect the external program to output exactly the same as described above with the addition of a token after $USER
:
"accept_token_info $groups $uid $gid $supplementary_gids $HOME $USER $token $info\n"
"accept_token_warning $groups $uid $gid $supplementary_gids $HOME $USER $token $warning\n"
If token validation authentication fails, the program should write "reject"
or "abort"
, possibly followed by a reason for the rejection and a trailing newline. For example "reject Bad password\n"
or just "abort\n"
. The difference between "reject"
and "abort"
is that with "reject"
, NSO will try subsequent mechanisms configured for /ncs-config/aaa/validation-order
in ncs.conf
(if any), while with "abort"
, the token validation authentication fails immediately. Thus "abort"
can prevent subsequent mechanisms from being tried. Currently, the only available token validation authentication mechanism is the external one.
Supported by some northbound APIs, such as JSON-RPC and CLI over SSH, the external validation may also choose to issue a challenge:
"challenge $challenge-id $challenge-prompt\n"
The challenge prompt may be multi-line, why it must be base64 encoded.
For more information on multi-factor authentication, see External Multi-Factor Authentication.
When username, password, or token authentication is not enough, a challenge may be sent from any of the external authentication mechanisms to the user. A challenge consists of a challenge ID and a base64 encoded challenge prompt, and a user is supposed to send a response to the challenge. Currently, only JSONRPC and CLI over SSH support multi-factor authentication. Responses to challenges of multi-factor authentication have the same output as the token authentication mechanism.
If this feature is configured, NSO will invoke the executable configured in /ncs-config/aaa/external-challenge/executable
in ncs.conf
, and pass the challenge ID and response on stdin
using the string notation: "[challenge-id;response;]\n"
.
For example, a user bob
has received a challenge from external authentication, external validation, or external challenge and then attempts to log in over JSON-RPC with a response to the challenge using challenge ID "22efa",response:"ae457b"
. The external challenge mechanism is enabled, NSO will invoke the configured executable and write "[22efa;ae457b;]\n"
on the stdin
stream for the executable.
The task of the executable is then to validate the challenge ID, and response combination, thereby authenticating the user and also establishing the username and username-to-groups mapping.
For example, the executable could be a RADIUS client which utilizes some proprietary vendor attributes to retrieve the username and groups of the user from the RADIUS server. If challenge ID, response validation is successful, the program should write "accept "
followed by a space-separated list of groups the user is a member of, and additional information as described below. Again, assuming that bob
's challenge ID, the response combination indeed was "22efa", "ae457b"
and that bob
is a member of the admin
and the lamers
groups, the program should write "accept admin lamers $uid $gid $supplementary_gids $HOME $USER\n"
on its standard output and then exit.
There is a general limit of 16000 bytes of output from the externalchallenge
program.
Thus the format of the output from an externalchallenge
program when challenge-based authentication is successful should be:
"accept $groups $uid $gid $supplementary_gids $HOME $USER\n"
Where:
$groups
is a space-separated list of the group names the user is a member of.
$uid
is the UNIX integer user ID NSO should use as a default when executing commands for this user.
$gid
is the UNIX integer group ID NSO should use as a default when executing commands for this user.
$supplementary_gids
is a (possibly empty) space-separated list of additional UNIX group IDs the user is also a member of.
$HOME
is the directory that should be used as HOME for this user when NSO executes commands on behalf of this user.
$USER
is the user derived from mapping the challenge ID, response.
It is further possible for the program to return a token on successful authentication, by using "accept_token"
instead of "accept"
:
"accept_token $groups $uid $gid $supplementary_gids $HOME $USER $token\n"
Where:
$token
is an arbitrary string. NSO will then, for some northbound interfaces, include this token in responses.
It is also possible for the program to return additional information on successful authentication, by using "accept_info"
instead of "accept"
:
"accept_info $groups $uid $gid $supplementary_gids $HOME $USER $info\n"
Where:
$info
is some arbitrary text. NSO will then just append this text to the generated audit log message (CONFD_EXT_LOGIN).
Yet another possibility is for the program to return a warning that the user's password is about to expire, by using "accept_warning"
instead of "accept"
:
"accept_warning $groups $uid $gid $supplementary_gids $HOME $USER $warning\n"
Where:
$warning
is an appropriate warning message. The message will be processed by NSO according to the setting of /ncs-config/aaa/expiration-warning
in ncs.conf
.
There is also support for token variations of "accept_info"
and "accept_warning"
namely "accept_token_info"
and "accept_token_warning"
. Both "accept_token_info"
and "accept_token_warning"
expects the external program to output exactly the same as described above with the addition of a token after $USER
:
"accept_token_info $groups $uid $gid $supplementary_gids $HOME $USER $token $info\n"
"accept_token_warning $groups $uid $gid $supplementary_gids $HOME $USER $token $warning\n"
If authentication fails, the program should write "reject"
or "abort"
, possibly followed by a reason for the rejection and a trailing newline. For example "reject Bad challenge response\n"
or just "abort\n"
. The difference between "reject"
and "abort"
is that with "reject"
, NSO will try subsequent mechanisms configured for /ncs-config/aaa/challenge-order
in ncs.conf
(if any), while with "abort"
, the challenge-response authentication fails immediately. Thus "abort"
can prevent subsequent mechanisms from being tried. Currently, the only available challenge-response authentication mechanism is the external one.
Supported by some northbound APIs, such as JSON-RPC and CLI over SSH, the external challenge may also choose to issue a new challenge:
"challenge $challenge-id $challenge-prompt\n"
The challenge prompt may be multi-line, so it must be base64 encoded.
Note that when using challenges with the CLI over SSH, the /ncs-config/cli/ssh/use-keyboard-interactive>
need to be set to true for the challenges to be sent correctly to the client.
The configuration of the SSH client used may need to be given the option to allow a higher number of allowed number of password prompts, e.g. -o NumberOfPasswordPrompts
, else the default number may introduce an unexpected behavior when the client is presented with multiple challenges.
The Package Authentication functionality allows for packages to handle the NSO authentication in a customized fashion. Authentication data can e.g. be stored remotely, and a script in the package is used to communicate with the remote system.
Compared to external authentication, the Package Authentication mechanism allows specifying multiple packages to be invoked in the order they appear in the configuration. NSO provides implementations for LDAP, SAMLv2, and TACACS+ protocols with packages available in $NCS_DIR/packages/auth/
. Additionally, you can implement your own authentication packages as detailed below.
Authentication packages are NSO packages with the required content of an executable file scripts/authenticate
. This executable basically follows the same API, and limitations, as the external auth script, but with a different input format and some additional functionality. Other than these requirements, it is possible to customize the package arbitrarily.
Package authentication is supported for Single Sign-On (see Single Sign-On in Web UI), JSON-RPC, and RESTCONF. Note that Single Sign-On and (non-batch) JSON-RPC allow all functionality while the RESTCONF interface will treat anything other than a "accept_username
" reply from the package as if authentication failed!
Package authentication is enabled by setting the ncs.conf
options /ncs-config/aaa/package-authentication/enabled
to true, and adding the package by name in the /ncs-config/aaa/package-authentication/packages
list. The order of the configured packages is the order that the packages will be used when attempting to authenticate a user. See ncs.conf(5) in Manual Pages for details.
If this feature is configured in ncs.conf
, NSO will for each configured package invoke script/authenticate
, and pass username, password, and original HTTP request (i.e. the user-supplied next
query parameter), HTTP request, HTTP headers, HTTP body, client source IP, client source port, northbound API context, and protocol on stdin
using the string notation: "[user;password;orig_request;request;headers;body;src-ip;src-port;ctx;proto;]\n"
.
The fields user, password, orig_request, request, headers, and body are all base64 encoded.
If the body length exceeds the partial_post_size
of the RESTCONF server, the body passed to the authenticate script will only contain the string '==nso_package_authentication_partial_body==
'.
The original request will be prefixed with the string ==nso_package_authentication_next==
before the base64 encoded part. This means supplying the next
query parameter value /my-location
will pass the following string to the authentication script: ==nso_package_authentication_next==L215LWxvY2F0aW9u
.
For example, if an unauthenticated user attempts to start a single sign-on process over northbound HTTP-based APIs with the cisco-nso-saml2-auth package, package authentication is enabled and configured with packages, and also single sign-on is enabled, NSO will, for each configured package, invoke the executable scripts/authenticate
and write "[;;;R0VUIC9zc28vc2FtbC9sb2dpbi8gSFRUUC8xLjE=;;;127.0.0.1;59226;webui;https;]\n"
. on the stdin
stream for the executable.
For clarity, the base64 decoded contents sent to stdin
presented: "[;;;GET /sso/saml/login/ HTTP/1.1;;;127.0.0.1;54321;webui;https;]\n"
.
The task of the package is then to authenticate the user and also establish the username-to-groups mapping.
For example, the package could support a SAMLv2 authentication protocol which communicates with an Identity Provider (IdP) for authentication. If authentication is successful, the program should write either "accept"
, or "accept_username"
, depending on whether the authentication is started with a username or if an external entity handles the entire authentication and supplies the username for a successful authentication. (SAMLv2 uses accept_username
, since the IdP handles the entire authentication.) The "accept_username " is followed by a username and then followed by a space-separated list of groups the user is a member of, and additional information as described below. If authentication is successful and the authenticated user bob
is a member of the groups admin
and wheel
, the program should write "accept_username bob admin wheel 1000 1000 100 /home/bob\n"
on its standard output and then exit.
There is a general limit of 16000 bytes of output from the "packageauth" program.
Thus the format of the output from a packageauth
program when authentication is successful should be either the same as from externalauth
(see External Authentication) or the following:
"accept_username $USER $groups $uid $gid $supplementary_gids $HOME\n"
Where:
$USER
is the user derived during the execution of the "packageauth" program.
$groups
is a space-separated list of the group names the user is a member of.
$uid
is the UNIX integer user ID NSO should use as a default when executing commands for this user.
$gid
is the UNIX integer group ID NSO should use as a default when executing commands for this user.
$supplementary_gids
is a (possibly empty) space-separated list of additional UNIX group IDs the user is also a member of.
$HOME
is the directory that should be used as HOME for this user when NSO executes commands on behalf of this user.
In addition to the externalauth
API, the authentication packages can also return the following responses:
unknown '
reason
'
- (reason
being plain-text) if they can't handle authentication for the supplied input.
redirect '
url
'
- (url
being base64 encoded) for an HTTP redirect.
content '
content-type
' '
content
'
- (content-type
being plain-text mime-type and content
being base64 encoded) to relay supplied content.
accept_username_redirect url $USER $groups $uid $gid $supplementary_gids $HOME
- which combines the accept_username
and redirect
.
It is also possible for the program to return additional information on successful authentication, by using "accept_info"
instead of "accept"
:
"accept_info $groups $uid $gid $supplementary_gids $HOME $info\n"
Where:
$info
is some arbitrary text. NSO will then just append this text to the generated audit log message (NCS_PACKAGE_AUTH_SUCCESS).
Yet another possibility is for the program to return a warning that the user's password is about to expire, by using "accept_warning"
instead of "accept"
:
"accept_warning $groups $uid $gid $supplementary_gids $HOME $warning\n"
Where:
$warning
is an appropriate warning message. The message will be processed by NSO according to the setting of /ncs-config/aaa/expiration-warning
in ncs.conf
.
If authentication fails, the program should write "reject"
or "abort"
, possibly followed by a reason for the rejection and a trailing newline. For example "reject 'Bad password'\n"
or just "abort\n"
. The difference between "reject"
and "abort"
is that with "reject"
, NSO will try subsequent mechanisms configured for /ncs-config/aaa/auth-order
, and packages configured for /ncs-config/aaa/package-authentication/packages
in ncs.conf
(if any), while with "abort"
, the authentication fails immediately. Thus "abort"
can prevent subsequent mechanisms from being tried, but when external authentication is the last mechanism (as in the default order), it has the same effect as "reject"
.
When package authentication is used, the group list returned by the package executable is prepended by any possible group information stored locally under the /aaa
tree. Hence when package authentication is used, it is indeed possible to have the entire /aaa/authentication
tree empty. The group assignment performed by the external program will still be valid and the relevant groups will be used by NSO when the authorization rules are checked.
Package authentication will invoke the scripts/authenticate
when a user tries to authenticate using CLI. In this case, only the username, password, client source IP, client source port, northbound API context, and protocol will be passed to the script.
When serving a username/password request, script output other than accept, challenge or abort will be treated as if authentication failed.
When this is enabled, /ncs-config/aaa/package-authentication/package-challenge/enabled
is set to true, packages will also be used to try to resolve challenges sent to the server and are only supported by CLI over SSH. The script script/challenge
will be invoked passing challenge ID, response, client source IP, client source port, northbound API context, and protocol on stdin
using the string notation: "[challengeid;response;src-ip;src-port;ctx;proto;]\n"
. The output should follow that of the authenticate script.
The fields challengeid
and response are base64 encoded when passed to the script.
NSO listens for client connections on the NSO IPC port. See /ncs-config/ncs-ipc-address/ip
in ncs.conf
. Access to this port is by default not authenticated. That means that all users with shell access to the host can connect to this port. So NSO deployment ends up in either of two cases: untrusted users have access to the host(s) where NSO is deployed, or they do not have shell access to the host(s) where NSO is deployed. If all shell users on the deployment host(s) are trusted, no further action is required, however, if untrusted users do have shell access to the hosts, access to the IPC port must be restricted, see Restricting Access to the IPC Port for more information.
If IPC port access is not used, an untrusted shell user can simply invoke the following to impersonate an admin
user:
Or, invoke the following to retrieve the entire configuration:
Once a user is authenticated, group membership must be established. A single user can be a member of several groups. Group membership is used by the authorization rules to decide which operations a certain user is allowed to perform. Thus the NSO AAA authorization model is entirely group-based. This is also sometimes referred to as role-based authorization.
All groups are stored under /nacm/groups
, and each group contains a number of usernames. The ietf-netconf-acm.yang
model defines a group entry:
The tailf-acm.yang
model augments this with a gid
leaf:
A valid group entry could thus look like:
The above XML data would then mean that users bob
and joe
are members of the admin
group. The users need not necessarily exist as actual users under /aaa/authentication/users
in order to belong to a group. If for example PAM authentication is used, it does not make sense to have all users listed under /aaa/authentication/users
.
By default, the user is assigned to groups by using any groups provided by the northbound transport (e.g. via the ncs_cli
or netconf-subsys
programs), by consulting data under /nacm/groups
, by consulting the /etc/group
file, and by using any additional groups supplied by the authentication method. If /nacm/enable-external-groups
is set to "false", only the data under /nacm/groups
is consulted.
The resulting group assignment is the union of these methods, if it is non-empty. Otherwise, the default group is used, if configured ( /ncs-config/aaa/default-group
in ncs.conf
).
A user entry has a UNIX uid and UNIX gid assigned to it. Groups may have optional group IDs. When a user is logged in, and NSO tries to execute commands on behalf of that user, the uid/gid for the command execution is taken from the user entry. Furthermore, UNIX supplementary group IDs are assigned according to the gid
's in the groups where the user is a member.
Once a user is authenticated and group membership is established, when the user starts to perform various actions, each action must be authorized. Normally the authorization is done based on rules configured in the AAA data model as described in this section.
The authorization procedure first checks the value of /nacm/enable-nacm
. This leaf has a default of true
, but if it is set to false
, all access is permitted. Otherwise, the next step is to traverse the rule-list
list:
If the group
leaf-list in a rule-list
entry matches any of the user's groups, the cmdrule
list entries are examined for command authorization, while the rule
entries are examined for RPC, notification, and data authorization.
The tailf-acm.yang
module augments the rule-list
entry in ietf-netconf-acm.yang
with a cmdrule
list:
Each rule has seven leafs. The first is the name
list key, the following three leafs are matching leafs. When NSO tries to run a command, it tries to match the command towards the matching leafs and if all of context
, command
, and access-operations
match, the fifth field, i.e. the action
, is applied.
name
: name
is the name of the rule. The rules are checked in order, with the ordering given by the YANG ordered-by user
semantics, i.e. independent of the key values.
context
: context
is either of the strings cli
, webui
, or *
for a command rule. This means that we can differentiate authorization rules for which access method is used. Thus if command access is attempted through the CLI, the context will be the string cli
whereas for operations via the Web UI, the context will be the string webui
.
command
: This is the actual command getting executed. If the rule applies to one or several CLI commands, the string is a space-separated list of CLI command tokens, for example request system reboot
. If the command applies to Web UI operations, it is a space-separated string similar to a CLI string. A string that consists of just *
matches any command.
In general, we do not recommend using command rules to protect the configuration. Use rules for data access as described in the next section to control access to different parts of the data. Command rules should be used only for CLI commands and Web UI operations that cannot be expressed as data rules.
The individual tokens can be POSIX extended regular expressions. Each regular expression is implicitly anchored, i.e. an ^
is prepended and a $
is appended to the regular expression.
access-operations
: access-operations
is used to match the operation that NSO tries to perform. It must be one or both of the "read" and "exec" values from the access-operations-type
bits type definition in ietf-netconf-acm.yang
, or "*" to match any operation.
action: If all of the previous fields match, the rule as a whole matches and the value of action
will be taken. I.e. if a match is found, a decision is made whether to permit or deny the request in its entirety. If action
is permit
, the request is permitted, if action
is deny
, the request is denied and an entry is written to the developer log.
log-if-permit
: If this leaf is present, an entry is written to the developer log for a matching request also when action
is permit
. This is very useful when debugging command rules.
comment
: An optional textual description of the rule.
For the rule processing to be written to the devel log, the /ncs-config/logs/developer-log-level
entry in ncs.conf
must be set to trace
.
If no matching rule is found in any of the cmdrule
lists in any rule-list
entry that matches the user's groups, this augmentation from tailf-acm.yang
is relevant:
If read
access is requested, the value of /nacm/cmd-read-default
determines whether access is permitted or denied.
If exec
access is requested, the value of /nacm/cmd-exec-default
determines whether access is permitted or denied.
If access
is permitted due to one of these default leafs, the /nacm/log-if-default-permit
has the same effect as the log-if-permit
leaf for the cmdrule
lists.
The rules in the rule
list are used to control access to rpc operations, notifications, and data nodes defined in YANG models. Access to invocation of actions (tailf:action
) is controlled with the same method as access to data nodes, with a request for exec
access. ietf-netconf-acm.yang
defines a rule
entry as:
tailf-acm
augments this with two additional leafs:
Similar to the command access check, whenever a user through some agent tries to access an RPC, a notification, a data item, or an action, access is checked. For a rule to match, three or four leafs must match and when a match is found, the corresponding action is taken.
We have the following leafs in the rule
list entry.
name
: The name of the rule. The rules are checked in order, with the ordering given by the YANG ordered-by user
semantics, i.e. independent of the key values.
module-name
: The module-name
string is the name of the YANG module where the node being accessed is defined. The special value *
(i.e. the default) matches all modules.\
rpc-name / notification-name / path
: This is a choice between three possible leafs that are used for matching, in addition to the module-name
:
rpc-name
: The name of an RPC operation, or *
to match any RPC.
notification-name
: the name of a notification, or *
to match any notification.
path
: A restricted XPath expression leading down into the populated XML tree. A rule with a path specified matches if it is equal to or shorter than the checked path. Several types of paths are allowed.
Tagpaths that do not contain any keys. For example /ncs/live-device/live-status
.
Instantiated key: as in /devices/device[name="x1"]/config/interface
matches the interface configuration for managed device "x1" It's possible to have partially instantiated paths only containing some keys instantiated - i.e. combinations of tagpaths and keypaths. Assuming a deeper tree, the path /devices/device/config/interface[name="eth0"]
matches the eth0
interface configuration on all managed devices.
The wild card at the end as in: /services/web-site/*
does not match the website service instances, but rather all children of the website service instances.\
Thus, the path in a rule is matched against the path in the attempted data access. If the attempted access has a path that is equal to or longer than the rule path - we have a match.
If none of the leafs rpc-name
, notification-name
, or path
are set, the rule matches for any RPC, notification, data, or action access.
context
: context
is either of the strings cli
, netconf
, webui
, snmp
, or *
for a data rule. Furthermore, when we initiate user sessions from MAAPI, we can choose any string we want. Similarly to command rules, we can differentiate access depending on which agent is used to gain access.
access-operations
: access-operations
is used to match the operation that NSO tries to perform. It must be one or more of the "create", "read", "update", "delete" and "exec" values from the access-operations-type
bits type definition in ietf-netconf-acm.yang
, or "*" to match any operation.
action
: This leaf has the same characteristics as the action
leaf for command access.
log-if-permit
: This leaf has the same characteristics as the log-if-permit
leaf for command access.
comment
: An optional textual description of the rule.
If no matching rule is found in any of the rule
lists in any rule-list
entry that matches the user's groups, the data model node for which access is requested is examined for the presence of the NACM extensions:
If the nacm:default-deny-all
extension is specified for the data model node, the access is denied.
If the nacm:default-deny-write
extension is specified for the data model node, and create
, update
, or delete
access is requested, the access is denied.
If examination of the NACM extensions did not result in access being denied, the value (permit
or deny
) of the relevant default leaf is examined:
If read
access is requested, the value of /nacm/read-default
determines whether access is permitted or denied.
If create
, update
, or delete
access is requested, the value of /nacm/write-default
determines whether access is permitted or denied.
If exec
access is requested, the value of /nacm/exec-default
determines whether access is permitted or denied.
If access is permitted due to one of these default leafs, this augmentation from tailf-acm.yang
is relevant:
I.e. it has the same effect as the log-if-permit
leaf for the rule
lists, but for the case where the value of one of the default leafs permits access.
When NSO executes a command, the command rules in the authorization database are searched, The rules are tried in order, as described above. When a rule matches the operation (command) that NSO is attempting, the action of the matching rule is applied - whether permit or deny.
When actual data access is attempted, the data rules are searched. E.g. when a user attempts to execute delete aaa
in the CLI, the user needs delete access to the entire tree /aaa
.
Another example is if a CLI user writes show configuration aaa TAB
it suffices to have read access to at least one item below /aaa
for the CLI to perform the TAB completion. If no rule matches or an explicit deny rule is found, the CLI will not TAB complete.
Yet another example is if a user tries to execute delete aaa authentication users
, we need to perform a check on the paths /aaa
and /aaa/authentication
before attempting to delete the sub-tree. Say that we have a rule for path /aaa/authentication/users
which is a permit rule and we have a subsequent rule for path /aaa
which is a deny rule. With this rule set the user should indeed be allowed to delete the entire /aaa/authentication/users
tree but not the /aaa
tree nor the /aaa/authentication
tree.
We have two variations on how the rules are processed. The easy case is when we actually try to read or write an item in the configuration database. The execution goes like this:
The second case is when we execute TAB completion in the CLI. This is more complicated. The execution goes like this:
The idea is that as we traverse (through TAB) down the XML tree, as long as there is at least one rule that can possibly match later, once we have more data, we must continue. For example, assume we have:
"/system/config/foo" --> permit
"/system/config" --> deny
If we in the CLI stand at "/system/config"
and hit TAB we want the CLI to show foo
as a completion, but none of the other nodes that exist under /system/config
. Whereas if we try to execute delete /system/config
the request must be rejected.
By default, NACM rules are configured for the entire tailf:action
or YANG 1.1 action
statements, but not for input
statement child leafs. To override this behavior, and enable NACM rules on input
leafs, set the following parameter to 'true': /ncs-config/aaa/action-input-rules/enabled
. When enabled all action input leafs given to an action will be validated for NACM rules. If broad 'deny' NACM rules are used, you might need to add 'permit' rules for the affected action input leafs to allow actions to be used with parameters.
By design NACM rules are ignored for changes done by services - FASTMAP, Reactive FASTMAP, or Nano services. The reasoning behind this is that a service package can be seen as a controlled way to provide limited access to devices for a user group that is not allowed to apply arbitrary changes on the devices.
However, there are NSO installations where this behavior is not desired, and NSO administrators want to enforce NACM rules even on changes done by services. For this purpose, the leaf called /nacm/enforce-nacm-on-services
is provided. By default, it is set to false
.
Note however that currently, even with this leaf set to true, there are limitations. Namely, the post-actions for nano-services are run in a user session without any access checks. Besides that, NACM rules are not enforced on the read operations performed in the service callbacks.
It might be desirable to deny everything for a user group and only allow access to a specific service. This pattern could be used to allow an operator to provision the service, but deny everything else. While this pattern works for a normal FASTMAP service, there are some caveats for stacked services, Reactive FASTMAP, and Nano services. For these kinds of services, in addition to the service itself, access should be provided to the user group for the following paths:
In case of stacked services, the user group needs read and write access to the leaf private/re-deploy-counter
under the bottom service. Otherwise, the user will not be able to redeploy the service.
In the case of Reactive FASTMAP or Nano services, the user group needs read and write access to the following:
/zombies
/side-effect-queue
/kickers
In deployments with many devices, it can become cumbersome to handle data authorization per device. To help with this there is a rule type that works on device group membership (for more on device groups, see Device Groups). To do this, devices are added to different device groups, and the rule type device-group-rule
is used.
The IETF NACM rule type is augmented with a new rule type named device-group-rule
which contains a leafref to the device groups. See the following example.
In the example below, we configure two device groups based on different regions and add devices to them.
In the example below, we configure an operator for the us_east
region:
In the example below, we configure the device group rules and refer to the device group and the us_east
group.
In summary device group authorization gives a more compact configuration for deployments where devices can be grouped and authorization can be done on a device group basis.
Modifications on the device-group subtree are recommended to be controlled by a limited set of users.
Assume that we have two groups, admin
and oper
. We want admin
to be able to see and edit the XML tree rooted at /aaa
, but we do not want users who are members of the oper
group to even see the /aaa
tree. We would have the following rule list and rule entries. Note, here we use the XML data from tailf-aaa.yang
to exemplify. The examples apply to all data, for all data models loaded into the system.
If we do not want the members of oper
to be able to execute the NETCONF operation edit-config
, we define the following rule list and rule entries:
To spell it out, the above defines four elements to match. If NSO tries to perform a netconf
operation, which is the operation edit-config
, and the user who runs the command is a member of the oper
group, and finally it is an exec
(execute) operation, we have a match. If so, the action is deny
.
The path
leaf can be used to specify explicit paths into the XML tree using XPath syntax. For example the following:
Explicitly allows the admin
group to change the password for precisely the bob
user when the user is using the CLI. Had path
been /aaa/authentication/users/user/password
the rule would apply to all password elements for all users. Since the path
leaf completely identifies the nodes that the rule applies to, we do not need to give tailf-aaa
for the module-name
leaf.
NSO applies variable substitution, whereby the username of the logged-in user can be used in a path
. Thus:
The above rule allows all users that are part of the admin
group to change their own passwords only.
A member of oper
is able to execute NETCONF operation action
if that member has exec
access on NETCONF RPC action
operation, read
access on all instances in the hierarchy of data nodes that identifies the specific action in the data store, and exec
access on the specific action. For example, an action is defined as below.
To be able to execute double
action through NETCONF RPC, the members of oper
need the following rule list and rule entries.
Or, a simpler rule set as the following.
Finally, if we wish members of the oper
group to never be able to execute the request system reboot
command, also available as a reboot
NETCONF rpc, we have:
Debugging the AAA rules can be hard. The best way to debug rules that behave unexpectedly is to add the log-if-permit
leaf to some or all of the rules that have action
permit
. Whenever such a rule triggers a permit action, an entry is written to the developer log.
Finally, it is worth mentioning that when a user session is initially created it will gather the authorization rules that are relevant for that user session and keep these rules for the life of the user session. Thus when we update the AAA rules in e.g. the CLI the update will not apply to the current session - only to future user sessions.
NSO's AAA subsystem will cache the AAA information in order to speed up the authorization process. This cache must be updated whenever there is a change to the AAA information. The mechanism for this update depends on how the AAA information is stored, as described in the following two sections.
To start NSO, the data models for AAA must be loaded. The defaults in the case that no actual data is loaded for these models allow all read and exec access, while write access is denied. Access may still be further restricted by the NACM extensions, though - e.g. the /nacm
container has nacm:default-deny-all
, meaning that not even read access is allowed if no data is loaded.
The NSO installation ships with an XML initialization file containing AAA configuration. The file is called aaa_init.xml
and is, by default, copied to the CDB directory by the NSO install scripts.
The local installation variant, targeting development only, defines two users, admin
and oper
with passwords set to admin
and oper
respectively for authentication. The two users belong to user groups with NACM rules restricting their authorization level. The system installation aaa_init.xml
variant, targeting production deployment, defines NACM rules only as users are, by default, authenticated using PAM. The NACM rules target two user groups, ncsadmin
and ncsoper
. Users belonging to the ncsoper
group are limited to read-only access.
The default aaa_init.xml
file provided with the NSO system installation must not be used as-is in a deployment without reviewing and verifying that every NACM rule in the file matches
Normally the AAA data will be stored as configuration in CDB. This allows for changes to be made through NSO's transaction-based configuration management. In this case, the AAA cache will be updated automatically when changes are made to the AAA data. If changing the AAA data via NSO's configuration management is not possible or desirable, it is alternatively possible to use the CDB operational data store for AAA data. In this case, the AAA cache can be updated either explicitly e.g. by using the maapi_aaa_reload()
function, see the confd_lib_maapi(3) in the Manual Pages manual page, or by triggering a subscription notification by using the subscription lock when updating the CDB operational data store, see Using CDB in Development.
Some applications may not want to expose the AAA data to end users in the CLI or the Web UI. Two reasonable approaches exist here and both rely on the tailf:export
statement. If a module has tailf:export none
it will be invisible to all agents. We can then either use a transform whereby we define another AAA model and write a transform program that maps our AAA data to the data that must exist in tailf-aaa.yang
and ietf-netconf-acm.yang
. This way we can choose to export and and expose an entirely different AAA model.
Yet another very easy way out, is to define a set of static AAA rules whereby a set of fixed users and fixed groups have fixed access to our configuration data. Possibly the only field we wish to manipulate is the password field.
Learn about Cisco-provided NEDs and how to manage them.
This section provides necessary information on NED (Network Element Driver) administration with a focus on Cisco-provided NEDs. If you're planning to use NEDs not provided by Cisco, refer to the NED Development to build your own NED packages.
NED represents a key NSO component that makes it possible for the NSO core system to communicate southbound with network devices in most deployments. NSO has a built-in client that can be used to communicate southbound with NETCONF-enabled devices. Many network devices are, however, not NETCONF-enabled, and there exist a wide variety of methods and protocols for configuring network devices, ranging from simple CLI to HTTP/REST-enabled devices. For such cases, it is necessary to use a NED to allow NSO communicate southbound with the network device.
Even for NETCONF-enabled devices, it is possible that the NSO's built-in NETCONF client cannot be used, for instance, if the devices do not strictly follow the specification for the NETCONF protocol. In such cases, one must also use a NED to seamlessly communicate with the device. See Managing Cisco-provided third Party YANG NEDs for more information on third-party YANG NEDs.
A NED package is a package that NSO uses to manage a particular type of device. A NED is a piece of code that enables communication with a particular type of managed device. You add NEDs to NSO as a special kind of package, called NED packages.
A NED package must provide a device YANG model as well as define means (protocol) to communicate with the device. The latter can either leverage the NSO built-in NETCONF and SNMP support or use a custom implementation. When a package provides custom protocol implementation, typically written in Java, it is called a CLI NED or a Generic NED.
Cisco provides and supports a number of such NEDs. With these Cisco-provided NEDs, a major category are CLI NEDs which communicate with a device through its CLI instead of a dedicated API.
This NED category is targeted at devices that use CLI as a configuration interface. Cisco-provided CLI NEDs are available for various network devices from different vendors. Many different CLI syntaxes are supported.
The driver element in a CLI NED implemented by the Cisco NSO NED team typically consists of the following three parts:
The protocol client, responsible for connecting to and interacting with the device. The protocols supported are SSH and Telnet.
A fast and versatile CLI parser (+ emitter), usually referred to as the turbo parser.
Various transform engines capable of converting data between NSO and device formats.
The YANG models in a CLI NED are developed and maintained by the Cisco NSO NED team. Usually, the models for a CLI NED are structured to mimic the CLI command hierarchy on the device.
A generic NED is typically used to communicate with non-CLI devices, such as devices using protocols like REST, TL1, Corba, SOAP, RESTCONF, or gNMI as a configuration interface. Even NETCONF-enabled devices in many cases require a generic NED to function properly with NSO.
The driver element in a Generic NED implemented by the Cisco NED team typically consists of the following parts:
The protocol client, responsible for interacting with the device.
Various transform engines capable of converting data between NSO and the device formats, usually JSON and/or XML transformers.
There are two types of Generic NEDs maintained by the Cisco NSO NED team:
NEDs with Cisco-owned YANG models. These NEDs have models developed and maintained by the Cisco NSO NED team.
NEDs targeted at YANG models from third-party vendors, also known as, third-party YANG NEDs.
Generic NEDs belonging to the first category typically handle devices that are model-driven. For instance, devices using proprietary protocols based on REST, SOAP, Corba, etc. The YANG models for such NEDs are usually structured to mimic the messages used by the proprietary protocol of the device.
As the name implies, this NED category is used for cases where the device YANG models are not implemented, maintained, or owned by the Cisco NSO NED team. Instead, the YANG models are typically provided by the device vendor itself, or by organizations like IETF, IEEE, ONF, or OpenConfig.
This category of NEDs has some special characteristics that set them apart from all other NEDs developed by the Cisco NSO NED team:
Targeted for devices supporting model-driven protocols like NETCONF, RESTCONF, and gNMI.
Delivered from the software.cisco.com portal without any device YANG models included. There are several reasons for this, such as legal restrictions that prevent Cisco from re-distributing YANG models from other vendors, or the availability of several different version bundles for open-source YANG, like OpenConfig. The version used by the NED must match the version used by the targeted device.
The NEDs can be bundled with various fixes to solve shortcomings in the YANG models, the download sources, and/or in the device. These fixes are referred to as recipes.
Since the third-party NEDs are delivered without any device YANG models, there are additional steps required to make this category of NEDs operational:
The device models need to be downloaded and copied into the NED package source tree. This can be done by using a special (optional) downloader tool bundled with each third-party YANG NED, or in any custom way.
The NED must be rebuilt with the downloaded YANG models.
This procedure is thoroughly described in Managing Cisco-provided third-Party YANG NEDs.
Recipes
A third-party YANG NED can be bundled with up to three types of recipe modules. These recipes are used by the NED to solve various types of issues related to:
The source of the YANG files.
The YANG files.
The device itself.
The recipes represent the characteristics and the real value of a third-party YANG NED. Recipes are typically adapted for a certain bundle of YANG models and/or certain device types. This is why there exist many different third-party YANG NEDs, each one adapted for a specific protocol, a specific model package, and/or a specific device.
The NSO NED team does not provide any super third-party YANG NEDs, for instance, a super RESTCONF NED that can be used with any models and any device.
Download Recipes
When downloading the YANG files, it is first of all important to know which source to use. In some cases, the source is the device itself. For instance, if the device is enabled for NETCONF and sometimes for RESTCONF (in rare cases).
In other cases, the device does not support model download. This applies to all gNMI-enabled devices and most RESTCONF devices too. In this case, the source can be a public Git repository or an archive file provided by the device vendor.
Another important question is what YANG models and what versions to download. To make this task easier, third-party NEDs can be bundled with the download recipes. These are presets to be used with the downloader tool bundled with the NED. There can be several profiles, each representing a preset that has been verified to work by the Cisco NSO NED team. A profile can point out a certain source to download from. It can also limit the scope of the download so that only certain YANG files are selected.
YANG Recipes (YR)
Third-party YANG files can often contain various types of errors, ranging from real bugs that cause compilation errors to certain YANG constructs that are known to cause runtime issues in NSO. To ensure that the files can be built correctly, the third-party NEDs can be bundled with YANG recipes. These recipes patch the downloaded YANG files before they are built by the NSO compiler. This procedure is performed automatically by the make
system when the NED is rebuilt after downloading the device YANG files. For more information, refer to Rebuilding the NED with a Unique NED ID.
Runtime Recipes (RR)
Many devices enabled for NETCONF, RESTCONF, or gNMI sometimes deviate in their runtime behavior. This can make it impossible to interact properly with NSO. These deviations can be on any level in the runtime behavior, such as:
The configuration protocol is not properly implemented, i.e., the device lacks support for mandatory parts of, for instance, the RESTCONF RFC.
The device returns "dirty" configuration dumps, for instance, JSON or XML containing invalid elements.
Special quirks are required when applying new configuration on a device. May also require additional transforms of the payload before it is relayed by the NED.
The device has aliasing issues, possibly caused by overlapping YANG models. If leaf X in model A is modified, the device will automatically modify leaf Y in model B as well.
A third-party YANG NED can be bundled with runtime recipes to solve these kinds of issues, if necessary. How this is implemented varies from NED to NED. In some cases, a NED has a fixed set of recipes that are always used. Alternatively, a NED can support several different recipes, which can be configured through a NED setting, referred to as a runtime profile. For example, a multi-vendor third-party YANG NED might have one runtime profile for each device type supported:
NED settings are YANG models augmented as configurations in NSO and control the behavior of the NED. These settings are augmented under:
/devices/global-settings/ned-settings
/devices/profiles/ned-settings
/devices/device/ned-settings
Most NEDs are instrumented with a large number of NED settings that can be used to customize the device instance configured in NSO. The README file in the respective NED contains more information on these.
Each managed device in NSO has a device type that informs NSO how to communicate with the device. When managing NEDs, the device type is either cli
or generic
. The other two device types, netconf
and snmp
, are used in NETCONF and SNMP packages and are further described in this guide.
In addition, a special NED ID identifier is needed. Simply put, this identifier is a handle in NSO pointing to the NED package. NSO uses the identifier when it is about to invoke the driver in a NED package. The identifier ensures that the driver of the correct NED package is called for a given device instance. For more information on how to set up a new device instance, see Configuring a device with the new Cisco-provided NED.
Each NED package has a NED ID, which is mandatory. The NED ID is a simple string that can have any format. For NEDs developed by the Cisco NSO NED team, the NED ID is formatted as <NED NAME>-<gen | cli>-<NED VERSION MAJOR>.<NED VERSION MINOR>
.
Examples
onf-tapi_rc-gen-2.0
cisco-iosxr-cli-7.43
The NED ID for a certain NED package stays the same from one version to another, as long as no backward incompatible changes have been done to the YANG models. Upgrading a NED from one version to another, where the NED ID is the same, is simple as it only requires replacing the old NED package with the new one in NSO and then reloading all packages.
Upgrading a NED package from one version to another, where the NED ID is not the same (typically indicated by a change of major or minor number in the NED version), requires additional steps. The new NED package first needs to be installed side-by-side with the old one. Then, a NED migration needs to be performed. This procedure is thoroughly described in NED Migration.
The Cisco NSO NED team ensures that our CLI NEDs, as well as Generic NEDs with Cisco-owned models, have version numbers and NED ID that indicate any possible backward incompatible YANG model changes. When a NED with such an incompatible change is released, the minor digit in the version is always incremented. The case is a bit different for our third-party YANG NEDs since it is up to the end user to select the NED ID to be used. This is further described in Managing Cisco-provided third-Party YANG NEDs.
A NED is assigned a version number consisting of a sequence of numbers separated by dots. The first two numbers represent the major and minor version, and the third number represents the maintenance version.
For example, the number 5.8.1 indicates a maintenance release (1) for the minor release 5.8. Incompatible YANG model changes require either the major or minor version number to be changed. This means that any version within the 5.8.x series is backward compatible with the previous versions.
When a newer maintenance release with the same major/minor version replaces a NED release, NSO can perform a simple data model upgrade to handle stored instance data in the CDB (Configuration Database). This type of upgrade does not pose a risk of data loss.
However, when a NED is replaced by a new major/minor release, it becomes a NED migration. These migrations are complex because the YANG model changes can potentially result in the loss of instance data if not handled correctly.
This section describes the NED installation in NSO for Local and System installs. Consult the README.md
supplied with the NED for the most up-to-date installation description.
This section describes how to install a NED package on a locally installed NSO. See Local Install Steps for more information.
Follow the instructions below to install a NED package:
Download the latest production-grade version of the NED from software.cisco.com using the URLs provided on your NED license certificates. All NED packages are files with the .signed.bin
extension named using the following rule: ncs-<NSO VERSION>-<NED NAME>-<NED VERSION>.signed.bin
. The NED package ncs-6.0-cisco-iosxr-7.43.signed.bin
will be used in the example below. It is assumed the NED package has been downloaded into the directory named /tmp/ned-package-store
. The environment variable NSO_RUNDIR
needs to be configured to point to the NSO runtime directory. Example:
Unpack the NED package and verify its signature.
In case the signature cannot be verified (for instance, if access to internet is down), do as below instead:
The result of the unpacking is a tar.gz
file with the same name as the .bin
file.
Untar the tar.gz
file. The result is a subdirectory named like <NED NAME>-<NED MAJOR VERSION DIGIT>.<NED MINOR VERSION DIGIT>
Install the NED into NSO, using the ncs-setup
tool.
Finally, open an NSO CLI session and load the new NED package like below:
Alternatively, the tar.gz
file can be installed directly into NSO. In this case, skip steps 3 and 4, and do as below instead:
This section describes how to install a NED package on a system-installed NSO. See System Install Steps for more information.
Download the latest production-grade version of the NED from software.cisco.com using the URLs provided on your NED license certificates. All NED packages are files with the .signed.bin
extension named using the following rule: ncs-<NSO_VERSION>-<NED NAME>-<NED VERSION>.signed.bin
. The NED package ncs-6.0-cisco-iosxr-7.43.signed.bin
will be used in the example below. It is assumed that the package has been downloaded into the directory named /tmp/ned-package-store
.
Unpack the NED package and verify its signature.
In case the signature cannot be verified (for instance, if access to internet is down), do as below instead.
The result of the unpacking is a tar.gz
file with the same name as the .bin
file.
Perform an NSO backup before installing the new NED package.
Start an NSO CLI session.
Fetch the NED package.
Install the NED package (add the argument replace-existing if a previous version has been loaded).
Finally, load the NED package.
The basic steps for configuring a device instance using the newly installed NED package are described in this section. Only the most basic configuration steps are covered here.
Many NEDs require additional custom configuration to be operational. This applies in particular to Generic NEDs. Information about such additional configuration can be found in the files README.md
and README-ned-settings.md
bundled with the NED package.
The following info is necessary to proceed with the basic setup of a device instance in NSO:
NED ID of the new NED.
Connection information for the device to connect to (address and port).
Authentication information to the device (username and password).
For CLI NEDs, it is mandatory to specify the protocol to be used, either SSH or Telnet.
The following values will be used for this example:
NED ID: cisco-iosxr-cli-7.43
Address: 10.10.1.1
Port: 22
Protocol: ssh
User: cisco
Password: cisco
Do the CLI NED setup as below:
Start an NSO CLI session.
Enter the configuration mode.
Configure a new authentication group to be used for this device.
Configure the new device instance.
Next, check the README.md
and README-ned-settings.md
bundled with the NED package for further information on additional settings to make the NED fully operational.
Finally, commit the configuration.
In the case of SSH, run also:
This example shows a simple setup of a generic NED.
The following values will be used for this example:
NED ID: onf-tapi_rc-gen-2.0
Address: 10.10.1.2
Port: 443
User: admin
Password: admin
Do the Generic NED setup as below:
Start an NSO CLI session.
Enter the configuration mode.
Configure a new authentication group to be used for this device.
Configure the new device instance.
Next, check the README.md
and README-ned-settings.md
bundled with the NED package for further information on additional settings to make the NED fully operational.
Finally, commit the configuration.
The third-party YANG NED type is a special category of the generic NED type targeted for devices supporting protocols like NETCONF, RESTCONF, and gNMI. As the name implies, this NED category is used for cases where the device YANG models are not implemented or maintained by the Cisco NSO NED Team. Instead, the YANG models are typically provided by the device vendor itself or by organizations like IETF, IEEE, ONF, or OpenConfig.
A third-party YANG NED package is delivered from the software.cisco.com portal without any device YANG models included. It is required that the models are first downloaded, followed by a rebuild and reload of the package, before the NED can become fully operational. This task needs to be performed by the NED user.
This section gives a brief instruction on how to download the device YANG models using the special downloader tool that is bundled with each third-party YANG NED. Each specific NED can contain specific requirements regarding downloading/rebuilding. Before proceeding, check the file README-rebuild.md
bundled with the NED package. Furthermore, it is recommended to use a non-production NSO environment for this task.
Download and install the third-party YANG NED package into NSO, see Local Install of NED in NSO.
Configure a device instance using as usual. See Cisco-provided Generic NED Setup for more information. The device name dev-1
will be used in this example.
Open an NCS CLI session (non-configure mode).
The installed NED is now basically empty. It contains no YANG models except some used by the NED internally. This can be verified with the following CLI commands:
The built-in downloader tool consists of a couple of NSO RPCs defined in one of the NED internal YANG files.
Start with checking the default local directory. This directory will be used as a target for the device YANG models to be downloaded.
This RPC will throw an error if the NED package was installed directly using the tar.gz
file. See NED Installation in NSO for more information.
If this error occurs, it is necessary to unpack the NED package in some other directory and use that as a target for the download. In the example below it is /tmp/ned-package-store/onf-tapi_rc-2.0/src/yang
.
Continue with listing the models supported by the connected device.
The size of the displayed list is device-dependent and so is the level of detail in each list entry. The only mandatory field is the name
. Furthermore, not all devices are actually capable of advertising the models supported. If the currently connected device lacks this support, it is usually emulated by the NED instead. Check the README-rebuild.md
for more information regarding this.
Next, list the download profiles currently supported by the device.
A download profile is a preset for the built-in download tool. Its purpose is to make the download procedure as easy as possible. A profile can, for instance, define a certain source from where the device YANG models will be downloaded. Another usage can be to limit the scope of the YANG files to download. For example, one profile to download the native device models, and another for the OpenConfig models. All download profiles are defined and verified by the Cisco NSO NED team. There is usually at least one profile available, otherwise, check the README-rebuild.md
bundled in the NED package.
Finally, try downloading the YANG models using a profile. In case a non-default local directory is used as a target, it must be explicitly specified.
In case the default local directory is used, no further arguments are needed.
The tool will output a list with each file downloaded. It automatically scans each YANG file for dependencies and tries to download them as well.
Verify that the downloaded files have been stored properly in the configured target directory.
The NED must be rebuilt when the device YANG models have been downloaded and stored properly. Compiling third-party YANG files is often combined with various types of issues caused by bad or odd YANG constructs. Such issues typically cause compiler errors or unwanted runtime errors in NSO. A third-party YANG NED is configured to take care of all currently known build issues. It will automatically patch the problematic files such that they build properly for NSO. This is done using a set of YANG build recipes bundled with the NED package.
Adapting the YANG build recipes is a continuous process. If new issues are found, the Cisco NED team updates the recipes accordingly and releases a new version of the NED.
It is strongly recommended that end users report newly found YANG build issues to the Cisco NSO NED team through a support request.
Before rebuilding the NED, it is important to know the path to the target directory used for the downloaded YANG files. This is the same as the local directory if the built-in NED downloader tool was used, see Downloading with the NED Built-in Download Tool.
This example uses the environment variable NED_YANG_TARGET_DIR
to represent the target directory.
To rebuild the NED with the downloaded YANG file:
Enter the NED build directory, which is the parent directory to the target directory.
Run the make clean all
command. The output from the make command can be massive, depending on the number of YANG files, etc. After this step, the NED is rebuilt with the device YANG models included. Lines like below indicate that the NED has applied a number of YANG recipes (patches) to solve known issues with the YANG files:
This is the final step to make a third-party YANG NED operational. If the NED built-in YANG downloader tool was used together with no local-dir
argument specified (i.e., the default), the only thing required is a package reload in NSO, which you can do by running the packages reload or the packages add command.
If another target directory was used for the YANG file download, it is necessary to first do a proper re-install of the NED package. See NED Installation in NSO.
A common use case is to have many different versions of a certain device type in the network. All devices can be managed by the same third-party YANG NED. However, each device will likely have its unique set of YANG files (or versions) which this NED has to be rebuilt for.
To set up NSO for this kind of scenario, some additional steps need to be taken:
Each flavor of the NED needs to be built in a separate source directory, i.e., untar the third-party YANG NED package at multiple locations.
Each flavor of the re-built NED must have its own unique NED-ID. This will make NSO allow multiple versions of the same NED package to co-exist.
The default NED ID for a third-party YANG NED typically looks like this: <NED NAME>-gen-<NED VERSION MAJOR DIGIT>.<NED VERSION MINOR DIGIT>
The NED build system allows for a customized NED ID by setting one or several of three make variables in any combination when rebuilding the NED:
NED_ID_SUFFIX
NED_ID_MAJOR
NED_ID_MINOR
Do as follows to build each flavor of the third-party YANG NED. Do it in iterations, one at a time:
Unpack the empty NED package as described in NED Installation in NSO.
Unpack the NED package again in a separate location. Rename the NED directory to something unique.
Configure a device instance using the installed NED, as described in Cisco-provided Generic NED Setup. Configure it to connect to the first variant of the device.
Follow the instructions in Downloading with the NED Built-in Download Tool to download the YANG files. Configure local-dir
to point to the location configured in Rebuilding NED with Downloaded YANG Files.
Rebuild a NED package from the location configured in Rebuilding NED with Downloaded YANG Files. Use a suitable combination of the NED_ID_SUFFIX
, NED_ID_MAJOR
, NED_ID_MINOR
.
Example 1:
This will result in the NED ID: onf-tapi_rc_tapi_v2.1.3-gen-2.0
.
Example 2:
This will result in the NED ID: onf-tapi_rc-gen-2.1.3
.
Install the newly built NED package into NSO, side-by-side with the original NED package. See Configuring a Device with the New Cisco-provided NED for further information.
Example:
Configure a new device instance using the newly installed NED package. Configure it to connect to the first variant of the device, as done in step 3.
Verify functionality by executing a sync-from
on the configured device instance.
The NSO procedure to upgrade a NED package to a newer version uses the following approach:
If there are no backward incompatible changes in the schemas (YANG models) of respective NEDs, simply replace the old NED with the new one and reload all packages in NSO.
In case there are backwards incompatible changes present in the schemas, some administration is required: the new NED needs to be installed side-by-side with the old NED, after which a NED migration must be performed to properly update the data in CDB using the new schemas. More information about NED migration is available in NED Migration.
Whether or not there are backward incompatible differences present between two versions of the same NED, is determined by the NED ID. If the versions have the same NED ID, they are fully compatible; otherwise, the NED IDs will differ, typically indicated by the major and/or minor number in the NED ID.
The third-party YANG NEDs add some extra complexity to the NED migration feature. This is because the device YANG models are not included in the NED package. It is up to the end user to select the YANG model versions to use and also to configure the NED ID. If the same NED, at a later stage, needs to be upgraded and rebuilt with newer versions of the YANG model, a decision has to be made regarding the NED ID: Is it safe to use the same NED ID, or should a new one be used?
Using a unique NED ID for each NED package is always the safe option. It minimizes the risk of data loss during package upgrade, etc. However, in some cases, it might be beneficial to use the same NED ID when upgrading a NED package since it minimizes the administration in NSO, i.e., simply replace the old NED package with the new one without any need of NED migration.
This kind of use case can occur when the firmware is upgraded on an NSO-controlled device. For example, assume that we have an optical device that supports the TAPI YANG models from the Open Networking Foundation. Current firmware supports version 2.1.3 of the TAPI bundle. The third-party YANG NED onf-tapi_rc
has been rebuilt accordingly with TAPI version 2.1.3 and the default NED ID onf-tapi_rc-gen-2.0
. This NED package is installed in NSO and a device instance named dev-1
is configured using it. Next, the optical device is upgraded with the new firmware that supports the TAPI bundle version 2.3.1 instead. The onf-tapi_rc
NED needs to be upgraded accordingly. The question is what NED ID to use?
To upgrade a Cisco-provided third-party YANG NED to a newer version:
Unpack a fresh copy of the onf-tapi_rc
NED package.
Download the TAPI models v2.3.1 from the TAPI public Git repository.
Rebuild the NED package with a temporary unique NED ID for this rebuild. Any unique NED ID works for this.
This will generate the NED ID: onf-tapi_rc-gen-2.3.1
.
Install the new onf-tapi_rc
NED package into NSO, side by side with the old one.
Now, execute a dry run of the NSO NED migration feature. This command generates a list of all schema differences found between the two packages, like below:
If the goal is to rebuild the new NED package again using the same NED ID as the old NED package, there are two things to look out for in the list:
Does the list contain any items with backward-compatible false?
If the answer is yes, is the affected schema node relevant for any use case, i.e., referenced by any service code running in NSO?
Any item listed as backward-compatible false can potentially result in data loss if the old NED is simply replaced with the new one. This might however be acceptable if the affected schema node is not relevant for any use case.
If you upgrade a managed device (such as installing a new firmware), the device data model can change in a significant way. If this is the case, you usually need to use a different and newer NED with an updated YANG model.
When the changes in the NED are not backward compatible, the NED is assigned a new ned-id to avoid breaking existing code. On the plus side, this allows you to use both versions of the NED at the same time, so some devices can use the new version and some can use the old one. As a result, there is no need to upgrade all devices at the same time. The downside is, NSO doesn't know the two NEDs are related and will not perform any upgrade on its own due to different ned-ids. Instead, you must manually change the NED of a managed device through a NED migration.
For third-party NEDs, the end user is required to configure the NED ID and also be aware of the backward incompatibilities. See Upgrading a Cisco-provided Third Party YANG NED to a Newer Version for an example.
Migration is required when upgrading a NED and the NED-ID changes, which is signified by a change in either the first or the second number in the NED package version. For example, if you're upgrading the existing router-nc-1.0.1
NED to router-nc-1.2.0
or router-nc-2.0.2
, you must perform NED migration. On the other hand, upgrading to router-nc-1.0.2
or router-nc-1.0.3
retains the same ned-id and you can upgrade the router-1.0.1
package in place, directly replacing it with the new one. However, note that some third-party, non-Cisco packages may not adhere to this standard versioning convention. In that case, you must check the ned-id values to see whether migration is needed.
A potential issue with a new NED is that it can break an existing service or other packages that rely on it. To help service developers and operators verify or upgrade the service code, NSO provides additional options of migration tooling for identifying the paths and service instances that may be impacted. Therefore, ensure that all the other packages are compatible with the new NED before you start migrating devices.
To prepare for the NED migration process, first, load the new NED package into NSO with either packages reload
or packages add
command. Then, use the show packages
command to verify that both NEDs, the new and the old, are present. Finally, you may perform the migration of devices either one by one or multiple at a time.
Depending on your operational policies, this may be done during normal operations and does not strictly require a maintenance window, as the migration only reads from and doesn't write to a network device. Still, it is recommended that you create an NSO backup before proceeding.
Note that changing a ned-id also affects device templates if you use them. To make existing device templates compatible with the new ned-id, you can use the copy
action. It will copy the configuration used for one ned-id to another, as long as the schema nodes used haven't changed between the versions. The following example demonstrates the copy
action usage:
For individual devices, use the /devices/device/migrate
action, with the new-ned-id
parameter. Without additional options, the command will read and update the device configuration in NSO. As part of this process, NSO migrates all the configuration and service meta-data. Use the dry-run
option to see what the command would do and verbose
to list all impacted service instances.
You may also use the no-networking
option to prevent NSO from generating any southbound traffic towards the device. In this case, only the device configuration in the CDB is used for the migration but then NSO can't know if the device is in sync. Afterward, you must use the compare-config or the sync-from action to remedy this.
For migrating multiple devices, use the /devices/migrate
action, which takes the same options. However, with this action, you must also specify the old-ned-id
, which limits the migration to devices using the old NED. You can further restrict the action with the device
parameter, selecting only specific devices.
It is possible for a NED migration to fail if the new NED is not entirely backward compatible with the old one and the device has an active configuration that is incompatible with the new NED version. In such cases, NSO will produce an error with the YANG constraint that is not satisfied. Here, you must first manually adjust the device configuration to make it compatible with the new NED, and then you can perform the migration as usual.
Depending on what changes are introduced by the migration and how these impact the services, it might be good to re-deploy
the affected services before removing the old NED package. It is especially recommended in the following cases:
When the service touches a list key that has changed. As long as the old schema is loaded, NSO is able to perform an upgrade.
When a namespace that was used by the service has been removed. The service diffset, that is, the recorded configuration changes created by the service, will no longer be valid. The diffset is needed for the correct get-modifications
output, deep-check-sync
, and similar operations.
The YANG modeling language supports the notion of a module revision
. It allows users to distinguish between different versions of a module, so the module can evolve over time. If you wish to use a new revision of a module for a managed device, for example, to access new features, you generally need to create a new NED.
When a model evolves quickly and you have many devices that require the use of a lot of different revisions, you will need to maintain a high number of NEDs, which are mostly the same. This can become especially burdensome during NSO version upgrades, when all NEDs may need to be recompiled.
When a YANG module is only updated in a backward-compatible way (following the upgrade rules in RFC6020 or RFC7950), the NSO compiler, ncsc
allows you to pack multiple module revisions into the same package. This way, a single NED with multiple device model revisions can be used, instead of multiple NEDs. Based on the capabilities exchange, NSO will then use the correct revision for communication with each device.
However, there is a major downside to this approach. While the exact revision is known for each communication session with the managed device, the device model in NSO does not have that information. For that reason, the device model always uses the latest revision. When pushing configuration to a device that only supports an older revision, NSO silently drops the unsupported parts. This may have surprising results, as the NSO copy can contain configuration that is not really supported on the device. Use the no-revision-drop
commit parameter when you want to make sure you are not committing config that is not supported by a device.
If you still wish to use this functionality, you can create a NED package with the ncs-make-package --netconf-ned
command as you would otherwise. However, the supplied source YANG directory should contain YANG modules with different revisions. The files should follow the module-or-submodule-name
@
revision-date
.yang
naming convention, as specified in the RFC6020. Some versions of the compiler require you to use the --no-fail-on-warnings
option with the ncs-make-package
command or the build process may fail.
The examples.ncs/development-guide/ned-upgrade/yang-revision
example shows how you can perform a YANG model upgrade. The original, 1.0 version of the router NED uses the router@2020-02-27.yang
YANG model. First, it is updated to the version 1.0.1 router@2020-09-18.yang
using a revision merge approach. This is possible because the changes are backward-compatible.
In the second part of the example, the updates in router@2022-01-25.yang
introduce breaking changes, therefore the version is increased to 1.1 and a different NED-ID is assigned to the NED. In this case, you can't use revision merge and the usual NED migration procedure is required.
Implement redundancy in your deployment using High Availability (HA) setup.
As a single NSO node can fail or lose network connectivity, you can configure multiple nodes in a highly available (HA) setup, which replicates the CDB configuration and operational data across participating nodes. It allows the system to continue functioning even when some nodes are inoperable.
The replication architecture is that of one active primary and a number of secondaries. This means all configuration write operations must occur on the primary, which distributes the updates to the secondaries.
Operational data in the CDB may be replicated or not based on the tailf:persistent
statement in the data model. If replicated, operational data writes can only be performed on the primary, whereas non-replicated operational data can also be written on the secondaries.
Replication is supported in several different architectural setups. For example, two-node active/standby designs as well as multi-node clusters with runtime software upgrade.
This feature is independent of but compatible with the Layered Service Architecture (LSA), which also configures multiple NSO nodes to provide additional scalability. When the following text simply refers to a cluster, it identifies the set of NSO nodes participating in the same HA group, not an LSA cluster, which is a separate concept.
NSO supports the following options for implementing an HA setup to cater to the widest possible range of use cases (only one can be used at a time):
HA Raft: Using a modern, consensus-based algorithm, it offers a robust, hands-off solution that works best in the majority of cases.
Rule-based HA: A less sophisticated solution that allows you to influence the primary selection but may require occasional manual operator action.
External HA: NSO only provides data replication; all other functions, such as primary selection and group membership management, are performed by an external application, using the HA framework (HAFW).
In addition to data replication, having a fixed address to connect to the current primary in an HA group greatly simplifies access for operators, users, and other systems alike. Use Tail-f HCC Package or an external load balancer to manage it.
Raft is a consensus algorithm that reliably distributes a set of changes to a group of nodes and robustly handles network and node failure. It can operate in the face of multiple, subsequent failures, while also allowing a previously failed or disconnected node to automatically rejoin the cluster without risk of data conflicts.
Compared to traditional fail-over HA solutions, Raft relies on the consensus of the participating nodes, which addresses the so-called “split-brain” problem, where multiple nodes assume a primary role. This problem is especially characteristic of two-node systems, where it is impossible for a single node on its own to distinguish between losing network connectivity itself versus the other node malfunctioning. For this reason, Raft requires at least three nodes in the cluster.
Raft achieves robustness by requiring at least three nodes in the HA cluster. Three is the recommended cluster size, allowing the cluster to operate in the face of a single node failure. In case you need to tolerate two nodes failing simultaneously, you can add two additional nodes, for a 5-node cluster. However, permanently having more than five nodes in a single cluster is currently not recommended since Raft requires the majority of the currently configured nodes in the cluster to reach consensus. Without the consensus, the cluster cannot function.
You can start a sample HA Raft cluster using the examples.ncs/high-availability/raft-cluster
example to test it out. The scripts in the example show various aspects of cluster setup and operation, which are further described in the rest of this section.
Optionally, examples using separate containers for each HA Raft cluster member with NSO system installations are available and referenced in the examples.ncs/development-guide/high-availability/hcc
example in the NSO example set.
The Raft algorithm works with the concept of (election) terms. In each term, nodes in the cluster vote for a leader. The leader is elected when it receives the majority of the votes. Since each node only votes for a single leader in a given term, there can only be one leader in the cluster for this term.
Once elected, the leader becomes responsible for distributing the changes and ensuring consensus in the cluster for that term. Consensus means that the majority of the participating nodes must confirm a change before it is accepted. This is required for the system to ensure no changes ever get overwritten and provide reliability guarantees. On the other hand, it also means more than half of the nodes must be available for normal operation.
Changes can only be performed on the leader, that will accept the change after the majority of the cluster nodes confirm it. This is the reason a typical Raft cluster has an odd number of nodes; exactly half of the nodes agreeing on a change is not sufficient. It also makes a two-node cluster (or any even number of nodes in a cluster) impractical; the system as a whole is no more available than it is with one fewer node.
If the connection to the leader is broken, such as during a network partition, the nodes start a new term and a new election. Another node can become a leader if it gets the majority of the votes of all nodes initially in the cluster. While gathering votes, the node has the status of a candidate. In case multiple nodes assume candidate status, a split-vote scenario may occur, which is resolved by starting a fresh election until a candidate secures the majority vote.
If it happens that there aren't enough reachable nodes to obtain a majority, a candidate can stay in the candidate state for an indefinite time. Otherwise, when a node votes for a candidate, it becomes a follower and stays a follower in this term, regardless if the candidate is elected or not.
Additionally, the NSO node can also be in the stalled state, if HA Raft is enabled but the node has not joined a cluster.
Each node in an HA Raft cluster needs a unique name. Names are usually in the ADDRESS
format, where ADDRESS
identifies a network host where the NSO process is running, such as a fully qualified domain name (FQDN) or an IPv4 address.
Other nodes in the cluster must be able to resolve and reach the ADDRESS
, which creates a dependency on the DNS if you use domain names instead of IP addresses.
Limitations of the underlying platform place a constraint on the format of ADDRESS
, which can't be a simple short name (without a dot), even if the system is able to resolve such a name using hosts
file or a similar mechanism.
You specify the node address in the ncs.conf
file as the value for node-address
, under the listen
container. You can also use the full node name (with the “@” character), however, that is usually unnecessary as the system prepends ncsd@
as-needed.
Another aspect in which ADDRESS
plays a role is authentication. The HA system uses mutual TLS to secure communication between cluster nodes. This requires you to configure a trusted Certificate Authority (CA) and a key/certificate pair for each node. When nodes connect, they check that the certificate of the peer validates against the CA and matches the ADDRESS
of the peer.
Consider that TLS not only verifies that the certificate/key pair comes from a trusted source (certificate is signed by a trusted CA), it also checks that the certificate matches the host you are connecting to. Host A may have a valid certificate and key, signed by a trusted CA, however, if the certificate is for another host, say host B, the authentication will fail.
In most cases, this means the ADDRESS
must appear in the node certificate's Subject Alternative Name (SAN) extension, as dNSName
(see RFC2459).
Create and use a self-signed CA to secure the NSO HA Raft cluster. A self-signed CA is the only secure option. The CA should only be used to sign the certificates of the member nodes in one NSO HA Raft cluster. It is critical for security that the CA is not used to sign any other certificates. Any certificate signed by the CA can be used to gain complete control of the NSO HA Raft cluster.
See the examples.ncs/high-availability/raft-cluster
example for one way to set up a self-signed CA and provision individual node certificates. The example uses a shell script gen_tls_certs.sh
that invokes the openssl
command. Consult the section Recipe for a Self-signed CA for using it independently of the example.
Examples using separate containers for each HA Raft cluster member with NSO system installations that use a variant of the gen_tls_certs.sh
script are available and referenced in the examples.ncs/development-guide/high-availability/hcc
example in the NSO example set.
When using an IP address instead of a DNS name for node's ADDRESS
, you must add the IP address to the certificate's dNSName SAN field (adding it to iPAddress field only is insufficient). This is a known limitation in the current version.
The following is a HA Raft configuration snippet for ncs.conf
that includes certificate settings and a sample ADDRESS
:
HA Raft uses a standard TLS protocol with public key cryptography for securing cross-node communication, where each node requires a separate public/private key pair and a corresponding certificate. Key and certificate management is a broad topic and is critical to the overall security of the system.
The following text provides a recipe for generating certificates using a self-signed CA. It uses strong cryptography and algorithms that are deemed suitable for production use. However, it makes a few assumptions that may not be appropriate for all environments. Always consider how they affect your own deployment and consult a security professional if in doubt.
The recipe makes the following assumptions:
You use a secured workstation or server to run these commands and handle the generated keys with care. In particular, you must copy the generated keys to NSO nodes in a secure fashion, such as using scp
.
The CA is used solely for a single NSO HA Raft cluster, with certificates valid for 10 years, and provides no CRL. If a single key or host is compromised, a new CA and all key/certificate pairs must be recreated and reprovisioned in the cluster.
Keys and signatures based on ecdsa-with-sha384/P-384 are sufficiently secure for the vast majority of environments. However, if your organization has specific requirements, be sure to follow those.
To use this recipe, first, prepare a working environment on a secure host by creating a new directory and copying the gen_tls_certs.sh
script from $NCS_DIR/examples.ncs/high-availability/raft-cluster
into it. Additionally, ensure that the openssl
command, version 1.1 or later, is available and the system time is set correctly. Supposing that you have a cluster named lower-west
, you might run:
Including cluster name in the directory name helps distinguish certificates of one HA cluster from another, such as when using an LSA deployment in an HA configuration.
The recipe relies on the gen_tls_certs.sh
script to generate individual certificates. For clusters using FQDN node addresses, invoke the script with full hostnames of all the participating nodes. For example:
Using only hostnames, e.g. node1
, will not work.
If your HA cluster is using IP addresses instead, add the -a
option to the command and list the IPs:
The script outputs the location of the relevant files and you should securely transfer each set of files to the corresponding NSO node. For each node, transfer only the three files: ca.crt
, host
.crt
, and host
.key
.
Once certificates are deployed, you can check their validity with the openssl verify
command:
This command takes into account the current time and can be used during troubleshooting. It can also display information contained in the certificate if you use the openssl x509 -text -in ssl/certs/
node1.example.org
.crt -noout
variant. The latter form allows you to inspect the incorporated hostname/IP address and certificate validity dates.
ncs.conf
PrerequisitesIn addition to the network connectivity required for the normal operation of a standalone NSO node, nodes in the HA Raft cluster must be able to initiate TCP connections from a random ephemeral client port to the following ports on other nodes:
Port 4369
Ports in the range 4370-4399 (configurable)
You can change the ports in the second listed range from the default of 4370-4399. Use the min-port
and max-port
settings of the ha-raft/listen
container.
The Raft implementation does not impose any other hard limits on the network but you should keep in mind that consensus requires communication with other nodes in the cluster. A high round-trip latency between cluster nodes is likely to negatively impact the transaction throughput of the system.
The HA Raft cluster also requires compatible ncs.conf
files among the member nodes. In particular, /ncs-config/cdb/operational/enabled
and /ncs-config/rollback/enabled
values affect replication behavior and must match. Likewise, each member must have the same set of encryption keys and the keys cannot be changed while the cluster is in operation.
To update the ncs.conf
configuration, you must manually update the copy on each member node, making sure the new versions contain compatible values. Then perform the reload on the leader and the follower members will automatically reload their copies of the configuration file as well.
If a node is a cluster member but has been configured with a new, incompatible ncs.conf
file, it gets automatically disabled. See the /ha-raft/status/disabled-reason
for reason. You can re-enable the node with the ha-raft reset
command, once you have reconciled the incompatibilities.
Raft has a notion of cluster configuration, in particular, how many and which members the cluster has. You define member nodes when you first initialize the cluster with the create-cluster
command or use the adjust-membership
command. The member nodes allow the cluster to know how many nodes are needed for consensus and similar.
However, not all cluster members may be reachable or alive all the time. Raft implementation in NSO uses TCP connections between nodes to transport data. The TCP connections are authenticated and encrypted using TLS by default (see Security Considerations). A working connection between nodes is essential for the cluster to function but a number of factors, such as firewall rules or expired/invalid certificates, can prevent the connection from establishing.
Therefore, NSO distinguishes between configured member nodes and nodes to which it has established a working transport connection. The latter are called connected nodes. In a normal, fully working, and properly configured cluster, the connected nodes will be the same as member nodes (except for the current node).
To help troubleshoot connectivity issues without affecting cluster operation, connected nodes will show even nodes that are not actively participating in the cluster but have established a transport connection to nodes in the cluster. The optional discovery mechanism, described next, relies on this functionality.
NSO includes a mechanism that simplifies the initial cluster setup by enumerating known nodes. This mechanism uses a set of seed nodes to discover all connectable nodes, which can then be used with the create-cluster
command to form a Raft cluster.
When you specify one or more nodes with the /ha-raft/seed-nodes/seed-node
setting in the ncs.conf
file, the current node tries to establish a connection to these seed nodes, in order to discover the list of all nodes potentially participating in the cluster. For the discovery to work properly, all other nodes must also use seed nodes and the set of seed nodes must overlap. The recommended practice is to use the same set of seed nodes on every participating node.
Along with providing an autocompletion list for the create-cluster
command, this feature streamlines the discovery of node names when using NSO in containerized or other dynamic environments, where node addresses are not known in advance.
Creating a new HA cluster consists of two parts: configuring the individual nodes and running the create-cluster
action.
First, you must update the ncs.conf
configuration file for each node. All HA Raft configuration comes under the /ncs-config/ha-raft
element.
As part of the configuration, you must:
Enable HA Raft functionality through the enabled
leaf.
Set node-address
and the corresponding TLS parameters (see Node Names and Certificates).
Identify the cluster this node belongs to with cluster-name
.
Reload or restart the NSO process (if already running).
Repeat the preceding steps for every participating node.
Invoke the create-cluster
action.
The cluster name is simply a character string that uniquely identifies this HA cluster. The nodes in the cluster must use the same cluster name or they will refuse to establish a connection. This setting helps prevent mistakenly adding a node to the wrong cluster when multiple clusters are in operation, such as in an LSA setup.
With all the nodes configured and running, connect to the node that you would like to become the initial leader and invoke the ha-raft create-cluster
action. The action takes a list of nodes identified by their names. If you have configured seed-nodes
, you will get auto-completion support, otherwise, you have to type in the names of the nodes yourself.
This action makes the current node a cluster leader and joins the other specified nodes to the newly created cluster. For example:
You can use the show ha-raft
command on any node to inspect the status of the HA Raft cluster. The output includes the current cluster leader and members according to this node, as well as information about the local node, such as node name (local-node
) and role. The status/connected-node
list contains the names of the nodes with which this node has active network connections.
In case you get an error, such as the Error: NSO can't reach member node 'ncsd@ADDRESS'.
, please verify all of the following:
The node at the ADDRESS
is reachable. You can use the ping
ADDRESS
command, for example.
The problematic node has the correct ncs.conf
configuration, especially cluster-name
and node-address
. The latter should match the ADDRESS
and should contain at least one dot.
Nodes use compatible configuration. For example, make sure the ncs.crypto_keys
file (if used) or the encrypted-strings
configuration in ncs.conf
is identical across nodes.
HA Raft is enabled, using the show ha-raft command on the unreachable node.
The firewall configuration on the OS and on the network level permits traffic on the required ports (see Network and ncs.conf
Prerequisites).
The node uses a certificate that the CA can validate. For example, copy the certificates to the same location and run openssl verify -CAfile CA_CERT NODE_CERT
to verify this.
Verify the epmd -names
command on each node shows the ncsd process. If not, stop NSO, run epmd -kill
, and then start NSO again.
In addition to the above, you may also examine the logs/raft.log
file for detailed information on the error message and overall operation of the Raft algorithm. The amount of information in the file is controlled by the /ncs-config/logs/raft-log
configuration in the ncs.conf
.
After the initial cluster setup, you can add new nodes or remove existing nodes from the cluster with the help of the ha-raft adjust-membership
action. For example:
When removing nodes using the ha-raft adjust-membership remove-node
command, the removed node is not made aware that it is removed from the cluster and continues signaling the other nodes. This is a limitation in the algorithm, as it must also handle situations, where the removed node is down or unreachable. To prevent further communication with the cluster, it is important you ensure the removed node is shut down. You should shut down the to-be-removed node prior to removal from the cluster, or immediately after it. The former is recommended but the latter is required if there are only two nodes left in the cluster and shutting down prior to removal would prevent the cluster from reaching consensus.
Additionally, you can force an existing follower node to perform a full re-sync from the leader by invoking the ha-raft reset
action with the force
option. Using this action on the leader will make the node give up the leader role and perform a sync with the newly elected leader.
As leader selection during the Raft election is not deterministic, NSO provides the ha-raft handover
action, which allows you to either trigger a new election if called with no arguments or transfer leadership to a specific node. The latter is especially useful when, for example, one of the nodes resides in a different location and more traffic between locations may incur extra costs or additional latency, so you prefer this node is not the leader under normal conditions.
If you have an existing HA cluster using the rule-based built-in HA, you can migrate it to use HA Raft instead. This procedure is performed in four distinct high-level steps:
Ensuring the existing cluster meets migration prerequisites.
Preparing the required HA Raft configuration files.
Switching to HA Raft.
Adding additional nodes to the cluster.
The procedure does not perform an NSO version upgrade, so the cluster remains on the same version. It also does not perform any schema upgrades, it only changes the type of the HA cluster.
The migration procedure is in place, that is, the existing nodes are disconnected from the old cluster and connected to the new one. This results in a temporary disruption of the service, so it should be performed during a service window.
First, you should ensure the cluster meets migration prerequisites. The cluster must use:
NSO 6.1.2 or later
tailf-hcc 6.0 or later (if used)
In case these prerequisites are not met, follow the standard upgrade procedures to upgrade the existing cluster to supported versions first.
Additionally, ensure that all used packages are compatible with HA Raft, as NSO uses some new or updated notifications about HA state changes. Also, verify the network supports the new cluster communications (see Network and ncs.conf
Prerequisites).
Secondly, prepare all the ncs.conf
and related files for each node, such as certificates and keys. Create a copy of all the ncs.conf
files and disable or remove the existing >ha<
section in the copies. Then add the required configuration items to the copies, as described in Initial Cluster Setup and Node Names and Certificates. Do not update the ncs.conf
files used by the nodes yet.
It is recommended but not necessary that you set the seed nodes in ncs.conf
to the designated primary and fail-over primary. Do this for all ncs.conf
files for all nodes.
With the new configurations at hand and verified, start the switch to HA Raft. The cluster nodes should be in their nominal, designated roles. If not, perform a failover first.
On the designated (actual) primary, called node1
, enable read-only mode.
Then take a backup of all nodes.
Once the backup successfully completes, stop the designated fail-over primary (actual secondary) NSO process, update its ncs.conf
and the related (certificate) files for HA Raft, and then start it again. Connect to this node's CLI, here called node2, and verify HA Raft is enabled with the show
ha-raft
command.
Now repeat the same for the designated primary (node1
). If you have set the seed nodes, you should see the fail-over primary show under connected-node
.
On the old designated primary (node1) invoke the ha-raft create-cluster
action and create a two-node Raft cluster with the old fail-over primary (node2
, actual secondary). The action takes a list of nodes identified by their names. If you have configured seed-nodes
, you will get auto-completion support, otherwise you have to type in the name of the node yourself.
In case of errors running the action, refer to Initial Cluster Setup for possible causes and troubleshooting steps.
Raft requires at least three nodes to operate effectively (as described in NSO HA Raft) and currently, there are only two in the cluster. If the initial cluster had only two nodes, you must provision an additional node and set it up for HA Raft. If the cluster initially had three nodes, there is the remaining secondary node, node3
, which you must stop, update its configuration as you did with the other two nodes, and start it up again.
Finally, on the old designated primary and current HA Raft leader, use the ha-raft adjust-membership add-node
action to add this third node to the cluster.
Communication between the NSO nodes in an HA Raft cluster takes place over Distributed Erlang, an RPC protocol transported over TLS (unless explicitly disabled by setting /ncs-config/ha-raft/ssl/enabled
to 'false').
TLS (Transport Layer Security) provides Authentication and Privacy by only allowing NSO nodes to connect using certificates and keys issued from the same Certificate Authority (CA). Distributed Erlang is transported over TLS 1.2. Access to a host can be revoked by the CA through the means of a CRL (Certificate Revocation List). To enforce certificate revocation within an HA Raft cluster, invoke the action /ha-raft/disconnect to terminate the pre-existing connection. A connection to the node can re-establish once the node's certificate is valid.
Please ensure the CA key is kept in a safe place since it can be used to generate new certificates and key pairs for peers.
Distributed Erlang supports for multiple NSO nodes to run on the same host and the node addresses are resolved by the epmd
(Erlang Port Mapper Daemon) service. Once resolved, the NSO nodes communicate directly.
The ports epmd
and the NSO nodes listen to can be found in Network and ncs.conf
Prerequisites. epmd
binds the wildcard IPv4 address 0.0.0.0
and the IPv6 address ::
.
In case epmd
is exposed to a DoS attack, the HA Raft members may be unable to resolve addresses and communication could be disrupted. Please ensure traffic on these ports are only accepted between the HA Raft members by using firewall rules or other means.
Two NSO nodes can only establish a connection if a shared secret "cookie" matches. The cookie is optionally configured from /ncs-config/ha-raft/cluster-name
. Please note the cookie is not a security feature but a way to isolate HA Raft clusters and to avoid accidental misuse.
NSO contains a mechanism for distributing packages to nodes in a Raft cluster, greatly simplifying package management in a highly-available setup.
You perform all package management operations on the current leader node. To identify the leader node, you can use the show ha-raft status leader
command on a running cluster.
Invoking the packages reload
command makes the leader node update its currently loaded packages, identical to a non-HA, single-node setup. At the same time, the leader also distributes these packages to the followers to load. However, the load paths on the follower nodes, such as /var/opt/ncs/packages/
, are not updated. This means, that if a leader election took place, a different leader was elected, and you performed another packages reload
, the system would try to load the versions of the packages on this other leader, which may be out of date or not even present.
The recommended approach is, therefore, to use the packages ha sync and-reload
command instead, unless a load path is shared between NSO nodes, such as the same network drive. This command distributes and updates packages in the load paths on the follower nodes, as well as loading them.
For the full procedure, first, ensure all cluster nodes are up and operational, then follow these steps on the leader node:
Perform a full backup of the NSO instance, such as running ncs-backup
.
Add, replace, or remove packages on the filesystem. The exact location depends on the type of NSO deployment, for example /var/opt/ncs/packages/
.
Invoke the packages ha sync and-reload
or packages ha sync and-add
command to start the upgrade process.
Note that while the upgrade is in progress, writes to the CDB are not allowed and will be rejected.
For a packages ha sync and-reload
example see the raft-upgrade-l2
NSO system installation-based example referenced by the examples.ncs/development-guide/high-availability/hcc
example in the NSO example set.
For more details, troubleshooting, and general upgrade recommendations, see NSO Packages and Upgrade.
Currently, the only supported and safe way of upgrading the Raft HA cluster NSO version requires that the cluster be taken offline since the nodes must, at all times, run the same software version.
Do not attempt an upgrade unless all cluster member nodes are up and actively participating in the cluster. Verify the current cluster state with the show ha-raft status
command. All member nodes must also be present in the connected-node list.
The procedure differentiates between the current leader node versus followers. To identify the leader, you can use the show ha-raft status leader
command on a running cluster.
Procedure 2. Cluster version upgrade
On the leader, first enable read-only mode using the ha-raft read-only mode true
command and then verify that all cluster nodes are in sync with the show ha-raft status log replications state
command.
Before embarking on the upgrade procedure, it's imperative to backup each node. This ensures that you have a safety net in case of any unforeseen issues. For example, you can use the $NCS_DIR/bin/ncs-backup
command.
Delete the $NCS_RUN_DIR/cdb/compact.lock
file and compact the CDB write log on all nodes using, for example, the $NCS_DIR/bin/ncs --cdb-compact $NCS_RUN_DIR/cdb
command.
On all nodes, delete the $NCS_RUN_DIR/state/raft/
directory with a command such as rm -rf $NCS_RUN_DIR/state/raft/
.
Stop NSO on all the follower nodes, for example, invoking the $NCS_DIR/bin/ncs --stop
or /etc/init.d/ncs stop
command on each node.
Stop NSO on the leader node only after you have stopped all the follower nodes in the previous step. Alternatively NSO can be stopped on the nodes before deleting the HA Raft state and compacting the CDB write log without needing to delete the compact.lock
file.
Upgrade the NSO packages on the leader to support the new NSO version.
Install the new NSO version on all nodes.
Start NSO on all nodes.
Re-initialize the HA cluster using the ha-raft create-cluster
action on the node to become the leader.
Finally, verify the cluster's state through the show ha-raft status
command. Ensure that all data has been correctly synchronized across all cluster nodes and that the leader is no longer read-only. The latter happens automatically after re-initializing the HA cluster.
For a standard System Install, the single-node procedure is described in Single Instance Upgrade, but in general depends on the NSO deployment type. For example, it will be different for containerized environments. For specifics, please refer to the documentation for the deployment type.
For an example see the raft-upgrade-l2
NSO system installation-based example referenced by the examples.ncs/development-guide/high-availability/hcc
example in the NSO example set.
If the upgrade fails before or during the upgrade of the original leader, start up the original followers to restore service and then restore the original leader, using backup as necessary.
However, if the upgrade fails after the original leader was successfully upgraded, you should still be able to complete the cluster upgrade. If you are unable to upgrade a follower node, you may provision a (fresh) replacement and the data and packages in use will be copied from the leader.
NSO can manage the HA groups based on a set of predefined rules. This functionality was added in NSO 5.4 and is sometimes referred to simply as the built-in HA. However, since NSO 6.1, HA Raft (which is also built-in) is available as well, and is likely a better choice in most situations.
Rule-based HA allows administrators to:
Configure HA group members with IP addresses and default roles
Configure failover behavior
Configure start-up behavior
Configure HA group members with IP addresses and default roles
Assign roles, join HA group, enable/disable rule-based HA through actions
View the state of the current HA setup
NSO rule-based HA is defined in tailf-ncs-high-availability.yang
, with data residing under the /high-availability/
container.
NSO rule-based HA does not manage any virtual IP addresses, or advertise any BGP routes or similar. This must be handled by an external package. Tail-f HCC 5.x and greater has this functionality compatible with NSO rule-based HA. You can read more about the HCC package in the following chapter.
To use NSO rule-based HA, HA must first be enabled in ncs.conf
- See Mode of Operation.
If the package tailf-hcc with a version less than 5.0 is loaded, NSO rule-based HA will not function. These HCC versions may still be used but NSO built-in HA will not function in parallel.
All HA group members are defined under /high-availability/ha-node
. Each configured node must have a unique IP address configured and a unique HA ID. Additionally, nominal roles and fail-over settings may be configured on a per-node basis.
The HA Node ID is a unique identifier used to identify NSO instances in an HA group. The HA ID of the local node - relevant amongst others when an action is called - is determined by matching configured HA node IP addresses against IP addresses assigned to the host machine of the NSO instance. As the HA ID is crucial to NSO HA, NSO rule-based HA will not function if the local node cannot be identified.
To join a HA group, a shared secret must be configured on the active primary and any prospective secondary. This is used for a CHAP-2-like authentication and is specified under /high-availability/token/
.
In an NSO System Install setup, not only does the shared token need to match between the HA group nodes but the configuration for encrypted strings, default stored in /etc/ncs/ncs.crypto_keys
, need also to match between the nodes in the HA group.
The token configured on the secondary node is overwritten with the encrypted token of type aes-256-cfb-128-encrypted-string
from the primary node when the secondary node connects to the primary. If there is a mismatch between the encrypted-string configuration on the nodes, NSO will not decrypt the HA token to match the token presented. As a result, the primary node denies the secondary node access the next time the HA connection needs to reestablish with a "Token mismatch, secondary is not allowed" error.
See the upgrade-l2
example, referenced from examples.ncs/development-guide/high-availability/hcc
, for an example setup and the Deployment Example for a description of the example.
Also, note that the ncs.crypto_keys
file is highly sensitive. The file contains the encryption keys for all CDB data that is encrypted on disk. Besides the HA token, this often includes passwords for various entities, such as login credentials to managed devices.
NSO can assume HA roles primary
, secondary
and none
. Roles can be assigned directly through actions, or at startup or failover. See HA Framework Requirements for the definition of these roles.
NSO rule-based HA does not support relay-secondaries.
NSO rule-based HA distinguishes between the concepts of nominal role and assigned role. Nominal-role is configuration data that applies when an NSO instance starts up and at failover. The assigned role is the role that the NSO instance has been ordered to assume either by an action or as a result of startup or failover.
Failover may occur when a secondary node loses the connection to the primary node. A secondary may then take over the primary role. Failover behavior is configurable and controlled by the parameters:
/high-availability/ha-node{id}/failover-primary
/high-availability/settings/enable-failover
For automatic failover to function, /high-availability/settings/enable-failover
must be se to true
. It is then possible to enable at most one node with a nominal role secondary as failover-primary, by setting the parameter /high-availability/ha-node{id}/failover-primary
. The failover works in both directions; if a nominal primary is currently connected to the failover-primary as a secondary and loses the connection, then it will attempt to take over as a primary.
Before failover happens, a failover-primary-enabled secondary node may attempt to reconnect to the previous primary before assuming the primary role. This behavior is configured by the parameters denoting how many reconnect attempts will be made, and with which interval, respectively.
/high-availability/settings/reconnect-attempts
/high-availability/settings/reconnect-interval
HA Members that are assigned as secondaries, but are neither failover-primaries nor set with a nominal-role primary, may attempt to rejoin the HA group after losing connection to the primary.
This is controlled by /high-availability/settings/reconnect-secondaries
. If this is true, secondary nodes will query the nodes configured under /high-availability/ha-node
for an NSO instance that currently has the primary role. Any configured nominal roles will not be considered. If no primary node is found, subsequent attempts to rejoin the HA setup will be issued with an interval defined by /high-availability/settings/reconnect-interval
.
In case a net-split provokes a failover it is possible to end up in a situation with two primaries, both nodes accepting writes. The primaries are then not synchronized and will end up in a split brain. Once one of the primaries joins the other as a secondary, the HA cluster is once again consistent but any out-of-sync changes will be overwritten.
To prevent split-brain from occurring, NSO 5.7 or later comes with a rule-based algorithm. The algorithm is enabled by default, it can be disabled or changed from the parameters:
/high-availability/settings/consensus/enabled [true]
/high-availability/settings/consensus/algorithm [ncs:rule-based]
The rule-based algorithm can be used in either of the two HA constellations:
Two nodes: one nominal primary and one nominal secondary configured as failover-primary.
Three nodes: one nominal primary, one nominal secondary configured as failover-primary, and one perpetual secondary.
On failover:
Failover-primary: become primary but enable read-only mode. Once the secondary joins, disable read-only.
Nominal primary: on loss of all secondaries, change role to none. If one secondary node is connected, stay primary.
In certain cases, the rule-based consensus algorithm results in nodes being disconnected and will not automatically rejoin the HA cluster, such as in the example above when the nominal primary becomes none on the loss of all secondaries.
To restore the HA cluster one may need to manually invoke the /high-availability/be-secondary-to
action.
In the case where the failover-primary takes over as primary, it will enable read-only mode, if no secondary connects it will remain read-only. This is done to guarantee consistency.
In a three-node cluster, when the nominal primary takes over as actual primary, it first enables read-only mode and stays in read-only mode until a secondary connects. This is done to guarantee consistency.
The read-write mode can manually be enabled from the /high-availability/read-only
action with the parameter mode passed with value false.
When any node loses connection, this can also be observed in high-availability alarms as either a ha-primary-down
or a ha-secondary-down
alarm.
Startup behavior is defined by a combination of the parameters /high-availability/settings/start-up/assume-nominal-role
and /high-availability/settings/start-up/join-ha
as well as the node's nominal role:
NSO rule-based HA can be controlled through several actions. All actions are found under /high-availability/
. The available actions are listed below:
The current state of NSO rule-based HA can be monitored by observing /high-availability/status/
. Information can be found about the current active HA mode and the current assigned role. For nodes with active mode primary, a list of connected nodes and their source IP addresses is shown. For nodes with assigned role secondary the latest result of the be-secondary operation is listed. All NSO rule-based HA status information is non-replicated operational data - the result here will differ between nodes connected in an HA setup.
The Tail-f HCC package extends the built-in HA functionality by providing virtual IP addresses (VIPs) that can be used to connect to the NSO HA group primary node. HCC ensures that the VIP addresses are always bound by the HA group primary and never bound by a secondary. Each time a node transitions between primary and secondary states HCC reacts by binding (primary) or unbinding (secondary) the VIP addresses.
HCC manages IP addresses at the link layer (OSI layer 2) for Ethernet interfaces, and optionally, also at the network layer (OSI layer 3) using BGP router advertisements. The layer-2 and layer-3 functions are mostly independent and this document describes the details of each one separately. However, the layer-3 function builds on top of the layer-2 function. The layer-2 function is always necessary, otherwise, the Linux kernel on the primary node would not recognize the VIP address or accept traffic directed to it.
Tail-f HCC version 5.x is non-backward compatible with previous versions of Tail-f HCC and requires functionality provided by NSO version 5.4 and greater. For more details, see the following chapter.
Both the HCC layer-2 VIP and layer-3 BGP functionality depend on iproute2
utilities and awk
. An optional dependency is arping
(either from iputils
or Thomas Habets arping
implementation) which allows HCC to announce the VIP to MAC mapping to all nodes in the network by sending gratuitous ARP requests.
The HCC layer-3 BGP functionality depends on the GoBGP
daemon version 2.x being installed on each NSO host that is configured to run HCC in BGP mode.
GoBGP is open-source software originally developed by NTT Communications and released under the Apache License 2.0. GoBGP can be obtained directly from https://osrg.github.io/gobgp/ and is also packaged for mainstream Linux distributions.
The HCC layer-3 DNS Update functionality depends on the command line utility nsupdate
.
Tools Dependencies are listed below:
Same as with built-in HA functionality, all NSO instances must be configured to run in HA mode. See the following instructions on how to enable HA on NSO instances.
GoBGP uses TCP port 179 for its communications and binds to it at startup. As port 179 is considered a privileged port it is normally required to run gobgpd as root.
When NSO is running as a non-root user the gobgpd command will be executed as the same user as NSO and will prevent gobgpd from binding to port 179.
There a multiple ways of handling this and two are listed here.
Set capability CAP_NET_BIND_SERVICE
on the gobgpd
file. May not be supported by all Linux distributions.
Set the owner to root
and the setuid
bit of the gobgpd
file. Works on all Linux distributions.
The vipctl
script, included in the HCC package, uses sudo
to run the ip
and arping
commands when NSO is not running as root. If sudo
is used, you must ensure it does not require password input. For example, if NSO runs as admin
user, the sudoers
file can be edited similarly to the following:
Tail-f HCC 5.x or later does not participate in decisions on which NSO node is primary or secondary. These decisions are taken by NSO's built-in HA and then pushed as notifications to HCC. The NSO built-in HA functionality is available in NSO starting with version 5.4, where older NSO versions are not compatible with the HCC 5.x or later.
HCC 5.x or later operates a GoBGP daemon as a subprocess completely managed by NSO. The old HCC function pack interacted with an external Quagga BGP daemon using a NED interface.
HCC 5.x or later automatically associates VIP addresses with Linux network interfaces using the ip
utility from the iproute2 package. VIP addresses are also treated as /32
without defining a new subnet. The old HCC function pack used explicit configuration to associate VIPs with existing addresses on each NSO host and define IP subnets for VIP addresses.
Since version 5.0, HCC relies on the NSO built-in HA for cluster management and only performs address or route management in reaction to cluster changes. Therefore, no special measures are necessary if using HCC when performing an NSO version upgrade or a package upgrade. Instead, you should follow the standard best practice HA upgrade procedure from NSO HA Version Upgrade.
A reference to upgrade examples can be found in the NSO example set under examples.ncs/development-guide/high-availability/hcc/README
.
The purpose of the HCC layer-2 functionality is to ensure that the configured VIP addresses are bound in the Linux kernel of the NSO primary node only. This ensures that the primary node (and only the primary node) will accept traffic directed toward the VIP addresses.
HCC also notifies the local layer-2 network when VIP addresses are bound by sending Gratuitous ARP (GARP) packets. Upon receiving the Gratuitous ARP, all the nodes in the network update their ARP tables with the new mapping so they can continue to send traffic to the non-failed, now primary node.
HCC binds the VIP addresses as additional (alias) addresses on existing Linux network interfaces (e.g. eth0
). The network interface for each VIP is chosen automatically by performing a kernel routing lookup on the VIP address. That is, the VIP will automatically be associated with the same network interface that the Linux kernel chooses to send traffic to the VIP.
This means that you can map each VIP onto a particular interface by defining a route for a subnet that includes the VIP. If no such specific route exists the VIP will automatically be mapped onto the interface of the default gateway.
To check which interface HCC will choose for a particular VIP address, simply run for example and look at the device dev
in the output, for example eth0
:
The layer-2 functionality is configured by providing a list of IPv4 and/or IPv6 VIP addresses and enabling HCC. The VIP configuration parameters are found under /hcc:hcc
.
Global Layer-2 Configuration:
The purpose of the HCC layer-3 BGP functionality is to operate a BGP daemon on each NSO node and to ensure that routes for the VIP addresses are advertised by the BGP daemon on the primary node only.
The layer-3 functionality is an optional add-on to the layer-2 functionality. When enabled, the set of BGP neighbors must be configured separately for each NSO node. Each NSO node operates an embedded BGP daemon and maintains connections to peers but only the primary node announces the VIP addresses.
The layer-3 functionality relies on the layer-2 functionality to assign the virtual IP addresses to one of the host's interfaces. One notable difference in assigning virtual IP addresses when operating in Layer-3 mode is that the virtual IP addresses are assigned to the loopback interface lo
rather than to a specific physical interface.
HCC operates a GoBGP
subprocess as an embedded BGP daemon. The BGP daemon is started, configured, and monitored by HCC. The HCC YANG model includes basic BGP configuration data and state data.
Operational data in the YANG model includes the state of the BGP daemon subprocess and the state of each BGP neighbor connection. The BGP daemon writes log messages directly to NSO where the HCC module extracts updated operational data and then repeats the BGP daemon log messages into the HCC log verbatim. You can find these log messages in the developer log (devel.log
).
GoBGP must be installed separately. The gobgp
and gobgpd
binaries must be found in paths specified by the $PATH
environment variable. For system install, NSO reads $PATH
in the init script /etc/init.d/ncs. Since tailf-hcc 6.0.2, the path to gobgp/gobgpd is no longer possible to specify from the configuration data leaf /hcc/bgp/node/gobgp-bin-dir
. The leaf has been removed from the tailf-hcc/src/yang/tailf-hcc.yang module.
Upgrades: If BGP is enabled and the gobgp or gobgpd binaries are not found, the tailf-hcc package will fail to load. The user must then install GoBGP and invoke the packages reload
action or restart NSO with NCS_RELOAD_PACKAGES=true /etc/init.d/ncs restart
.
The layer-3 BGP functionality is configured as a list of BGP configurations with one list entry per node. Configurations are separate because each NSO node usually has different BGP neighbors with their own IP addresses, authentication parameters, etc.
The BGP configuration parameters are found under /hcc:hcc/bgp/node{id}
.
Per-Node Layer-3 Configuration:
Each NSO node can connect to a different set of BGP neighbors. For each node, the BGP neighbor list configuration parameters are found under /hcc:hcc/bgp/node{id}/neighbor{address}
.
Per-Neighbor BGP Configuration:
The purpose of the HCC layer-3 DNS Update functionality is to notify a DNS server of the IP address change of the active primary NSO server, allowing the DNS server to update the DNS record for the given domain name.
Geographically redundant NSO setup typically relies on DNS support. To enable this use case, tailf-hcc can dynamically update DNS with the nsupdate
utility on HA status change notification.
The DNS server used should support updates through nsupdate
command (RFC 2136).
HCC listens on the underlying NSO HA notifications stream. When HCC receives a notification about an NSO node being Primary, it updates the DNS Server with the IP address of the Primary NSO for the given hostname. The HCC YANG model includes basic DNS configuration data and operational status data.
Operational data in the YANG model includes the result of the latest DNS update operation.
If the DNS Update is unsuccessful, an error message will be populated in operational data, for example:
The DNS Server must be installed and configured separately, and details are provided to HCC as configuration data. The DNS Server must be configured to update the reverse DNS record.
The layer-3 DNS Update functionality needs DNS-related information like DNS server IP address, port, zone, etc, and information about NSO nodes involved in HA - node, ip, and location.
The DNS configuration parameters are found under /hcc:hcc/dns
.
Layer-3 DNS Configuration:
Each NSO node can be placed in a separate Location/Site/Availability-Zone. This is configured as a list member configuration, with one list entry per node ID. The member list configuration parameters are found under /hcc:hcc/dns/member{node-id}
.
Here is an example configuration for a setup of two dual-stack NSO nodes, node-1 and node-2, that have an IPv4 and an IPv6 address configured. The configuration also sets up an update signing with the specified key.
This section describes basic deployment scenarios for HCC. Layer-2 mode is demonstrated first and then the layer-3 BGP functionality is configured in addition:
A reference to container-based examples for the layer-2 and layer-3 deployment scenarios described here can be found in the NSO example set under examples.ncs/development-guide/high-availability/hcc
.
Both scenarios consist of two test nodes: london
and paris
with a single IPv4 VIP address. For the layer-2 scenario, the nodes are on the same network. The layer-3 scenario also involves a BGP-enabled router
node as the london
and paris
nodes are on two different networks.
The layer-2 operation is configured by simply defining the VIP addresses and enabling HCC. The HCC configuration on both nodes should match, otherwise, the primary node's configuration will overwrite the secondary node configuration when the secondary connects to the primary node.
Addresses:
Configuring VIPs:
Verifying VIP Availability:
Once enabled, HCC on the HA group primary node will automatically assign the VIP addresses to corresponding Linux network interfaces.
On the secondary node, HCC will not configure these addresses.
Layer-2 Example Implementation:
A reference to a container-based example of the layer-2 scenario can be found in the NSO example set. See the examples.ncs/development-guide/high-availability/hcc/README
Layer-3 operation is configured for each NSO HA group node separately. The HCC configuration on both nodes should match, otherwise, the primary node's configuration will overwrite the configuration on the secondary node.
Addresses:
Configuring BGP for Paris Node:
Configuring BGP for London Node:
Check BGP Neighbor Connectivity:
Check neighbor connectivity on the paris
primary node. Note that its connection to neighbor 192.168.31.2 (router
) is ESTABLISHED
.
Check neighbor connectivity on the london
secondary node. Note that the primary node also has an ESTABLISHED
connection to its neighbor 192.168.30.2 (router
). The primary and secondary nodes both maintain their BGP neighbor connections at all times when BGP is enabled, but only the primary node announces routes for the VIPs.
Check Advertised BGP Routes Neighbors:
Check the BGP routes received by the router
.
The VIP subnet is routed to the paris
host, which is the primary node.
Layer-3 BGP Example Implementation:
A reference to a container-based example of the combined layer-2 and layer-3 BGP scenario can be found in the NSO example set. See the examples.ncs/development-guide/high-availability/hcc/README
If enabled prior to the HA being established, HCC will update the DNS server with the IP address of the Primary node once a primary is selected.
If an HA is already operational, and Layer-3 DNS is enabled and configured afterward, HCC will not update the DNS server automatically. An automatic DNS server update will only happen if a HA switchover happens. HCC exposes an update action to manually trigger an update to the DNS server with the IP address of the primary node.
DNS Update Action:
The user can explicitly update DNS from the specific NSO node by running the update action.
Check the result of invoking the DNS update utility using the operational data in /hcc/dns
:
One way to verify DNS server updates is through the nslookup
program. However, be mindful of the DNS caching mechanism, which may cache the old value for the amount of time controlled by the TTL setting.
DNS get-node-location Action:
/hcc/dns/member holds the information about all members involved in HA. The get-node-location
action provides information on the location of an NSO node.
The HCC data model can be found in the HCC package (tailf-hcc.yang
).
As an alternative to the HCC package, NSO built-in HA, either rule-based or HA Raft, can also be used in conjunction with a load balancer device in a reverse proxy configuration. Instead of managing the virtual IP address directly as HCC does, this setup relies on an external load balancer to route traffic to the currently active primary node.
The load balancer uses HTTP health checks to determine which node is currently the active primary. The example, found in the examples.ncs/development-guide/high-availability/load-balancer
directory uses HTTP status codes on the health check endpoint to easily distinguish whether the node is currently primary or not.
In the example, freely available HAProxy software is used as a load balancer to demonstrate the functionality. It is configured to steer connections on localhost to either of the TCP port 2024 (SSH CLI) and TCP port 8080 (web UI and RESTCONF) to the active node in a 2-node HA cluster. The HAProxy software is required if you wish to run this example yourself.
You can start all the components in the example by running the make build start
command. At the beginning, the first node n1
is the active primary. Connecting to the localhost port 2024 will establish a connection to this node:
Then, you can disable the high availability subsystem on n1
to simulate a node failure.
Disconnect and wait a few seconds for the built-in HA to perform the failover to node n2
. The time depends on the high-availability/settings/reconnect-interval
and is set quite aggressively in this example to make the failover in about 6 seconds. Reconnect with the SSH client and observe the connection is now made to the fail-over node which has become the active primary:
Finally, shut down the example with the make stop clean
command.
NSO can be configured for the HA primary to listen on additional ports for the northbound interfaces NETCONF, RESTCONF, the web server (including JSON-RPC), and the CLI over SSH. Once a different node transitions to role primary the configured listen addresses are brought up on that node instead.
When the following configuration is added to ncs.conf
, then the primary HA node will listen(2) and bind(2) port 1830 on the wildcard IPv4 and IPv6 addresses.
A similar configuration can be added for other NB interfaces, see the ha-primary-listen list under /ncs-config/{restconf,webui,cli}
.
If an external HAFW is used, NSO only replicates the CDB data. NSO must be told by the HAFW which node should be primary and which nodes should be secondaries.
The HA framework must also detect when nodes fail and instruct NSO accordingly. If the primary node fails, the HAFW must elect one of the remaining secondaries and appoint it the new primary. The remaining secondaries must also be informed by the HAFW about the new primary situation.
NSO must be instructed through the ncs.conf
configuration file that it should run in HA mode. The following configuration snippet enables HA mode:
Make sure to restart the ncs
process for the changes to take effect.
The IP address and the port above indicate which IP and which port should be used for the communication between the HA nodes. extra-listen
is an optional list of ip:port
pairs that a HA primary also listens to for secondary connections. For IPv6 addresses, the syntax [ip]:port
may be used. If the :port
is omitted, port
is used. The tick-timeout
is a duration indicating how often each secondary must send a tick message to the primary indicating liveness. If the primary has not received a tick from a secondary within 3 times the configured tick time, the secondary is considered to be dead. Similarly, the primary sends tick messages to all the secondaries. If a secondary has not received any tick messages from the primary within the 3 times the timeout, the secondary will consider the primary dead and report accordingly.
A HA node can be in one of three states: NONE
, SECONDARY
or PRIMARY
. Initially, a node is in the NONE
state. This implies that the node will read its configuration from CDB, stored locally on file. Once the HA framework has decided whether the node should be a secondary or a primary the HAFW must invoke either the methods Ha.beSecondary(primary)
or Ha.bePrimary()
When an NSO HA node starts, it always starts up in mode NONE
. At this point, there are no other nodes connected. Each NSO node reads its configuration data from the locally stored CDB and applications on or off the node may connect to NSO and read the data they need. Although write operations are allowed in the NONE
state it is highly discouraged to initiate southbound communication unless necessary. A node in NONE
state should only be used to configure NSO itself or to do maintenance such as upgrades. When in NONE
state, some features are disabled, including but not limited to:
commit queue
NSO scheduler
nano-service side effect queue
This is to avoid situations where multiple NSO nodes are trying to perform the same southbound operation simultaneously.
At some point, the HAFW will command some nodes to become secondary nodes of a named primary node. When this happens, each secondary node tracks changes and (logically or physically) copies all the data from the primary. Previous data at the secondary node is overwritten.
Note that the HAFW, by using NSO's start phases, can make sure that NSO does not start its northbound interfaces (NETCONF, CLI, ...) until the HAFW has decided what type of node it is. Furthermore once a node has been set to the SECONDARY
state, it is not possible to initiate new write transactions towards the node. It is thus never possible for an agent to write directly into a secondary node. Once a node is returned either to the NONE
state or to the PRIMARY
state, write transactions can once again be initiated towards the node.
The HAFW may command a secondary node to become primary at any time. The secondary node already has up-to-date data, so it simply stops receiving updates from the previous primary. Presumably, the HAFW also commands the primary node to become a secondary node or takes it down, or handles the situation somehow. If it has crashed, the HAFW tells the secondary to become primary, restarts the necessary services on the previous primary node, and gives it an appropriate role, such as secondary. This is outside the scope of NSO.
Each of the primary and secondary nodes has the same set of all callpoints and validation points locally on each node. The start sequence has to make sure the corresponding daemons are started before the HAFW starts directing secondary nodes to the primary, and before replication starts. The associated callbacks will however only be executed at the primary. If e.g. the validation executing at the primary needs to read data that is not stored in the configuration and only available on another node, the validation code must perform any needed RPC calls.
If the order from the HAFW is to become primary, the node will start to listen for incoming secondaries at the ip:port
configured under /ncs-config/ha
. The secondaries TCP connect to the primary and this socket is used by NSO to distribute the replicated data.
If the order is to be a secondary, the node will contact the primary and possibly copy the entire configuration from the primary. This copy is not performed if the primary and secondary decide that they have the same version of the CDB database loaded, in which case nothing needs to be copied. This mechanism is implemented by use of a unique token, the transaction id
- it contains the node id of the node that generated it and a time stamp, but is effectively "opaque".
This transaction ID is generated by the cluster primary each time a configuration change is committed, and all nodes write the same transaction ID into their copy of the committed configuration. If the primary dies and one of the remaining secondaries is appointed the new primary, the other secondaries must be told to connect to the new primary. They will compare their last transaction ID to the one from the newly appointed primary. If they are the same, no CDB copy occurs. This will be the case unless a configuration change has sneaked in since both the new primary and the remaining secondaries will still have the last transaction ID generated by the old primary - the new primary will not generate a new transaction ID until a new configuration change is committed. The same mechanism works if a secondary node is simply restarted. No cluster reconfiguration will lead to a CDB copy unless the configuration has been changed in between.
Northbound agents should run on the primary, an agent can't commit write operations at a secondary node.
When an agent commits its CDB data, CDB will stream the committed data out to all registered secondaries. If a secondary dies during the commit, nothing will happen, the commit will succeed anyway. When and if the secondary reconnects to the cluster, the secondary will have to copy the entire configuration. All data on the HA sockets between NSO nodes only go in the direction from the primary to the secondaries. A secondary that isn't reading its data will eventually lead to a situation with full TCP buffers at the primary. In principle, it is the responsibility of HAFW to discover this situation and notify the primary NSO about the hanging secondary. However, if 3 times the tick timeout is exceeded, NSO will itself consider the node dead and notify the HAFW. The default value for tick timeout is 20 seconds.
The primary node holds the active copy of the entire configuration data in CDB. All configuration data has to be stored in CDB for replication to work. At a secondary node, any request to read will be serviced while write requests will be refused. Thus, the CDB subscription code works the same regardless of whether the CDB client is running at the primary or at any of the secondaries. Once a secondary has received the updates associated to a commit at the primary, all CDB subscribers at the secondary will be duly notified about any changes using the normal CDB subscription mechanism.
If the system has been set up to subscribe for NETCONF notifications, the secondaries will have all subscriptions as configured in the system, but the subscription will be idle. All NETCONF notifications are handled by the primary, and once the notifications get written into stable storage (CDB) at the primary, the list of received notifications will be replicated to all secondaries.
We specify in ncs.conf
which IP address the primary should bind for incoming secondaries. If we choose the default value 0.0.0.0
it is the responsibility of the application to ensure that connection requests only arrive from acceptable trusted sources through some means of firewalling.
A cluster is also protected by a token, a secret string only known to the application. The Ha.connect()
method must be given the token. A secondary node that connects to a primary node negotiates with the primary using a CHAP-2-like protocol, thus both the primary and the secondary are ensured that the other end has the same token without ever revealing their own token. The token is never sent in clear text over the network. This mechanism ensures that a connection from an NSO secondary to a primary can only succeed if they both have the same token.
It is indeed possible to store the token itself in CDB, thus an application can initially read the token from the local CDB data, and then use that token in . the constructor for the Ha
class. In this case, it may very well be a good idea to have the token stored in CDB be of type tailf:aes-256-cfb-128-encrypted-string.
If the actual CDB data that is sent on the wire between cluster nodes is sensitive, and the network is untrusted, the recommendation is to use IPSec between the nodes. An alternative option is to decide exactly which configuration data is sensitive and then use the tailf:aes-256-cfb-128-encrypted-string type for that data. If the configuration data is of type tailf:aes-256-cfb-128-encrypted-string the encrypted data will be sent on the wire in update messages from the primary to the secondaries.
There are two APIs used by the HA framework to control the replication aspects of NSO. First, there exists a synchronous API used to tell NSO what to do, secondly, the application may create a notifications socket and subscribe to HA-related events where NSO notifies the application on certain HA-related events such as the loss of the primary, etc. The HA-related notifications sent by NSO are crucial to how to program the HA framework.
The HA-related classes reside in the com.tailf.ha
package. See Javadocs for reference. The HA notifications-related classes reside in the com.tailf.notif
package, See Javadocs for reference.
The configuration parameter /ncs-cfg/ha/tick-timeout
is by default set to 20 seconds. This means that every 20 seconds each secondary will send a tick message on the socket leading to the primary. Similarly, the primary will send a tick message every 20 seconds on every secondary socket.
This aliveness detection mechanism is necessary for NSO. If a socket gets closed all is well, NSO will clean up and notify the application accordingly using the notifications API. However, if a remote node freezes, the socket will not get properly closed at the other end. NSO will distribute update data from the primary to the secondaries. If a remote node is not reading the data, TCP buffer will get full and NSO will have to start to buffer the data. NSO will buffer data for at most tickTime
times 3 time units. If a tick
has not been received from a remote node within that time, the node will be considered dead. NSO will report accordingly over the notifications socket and either remove the hanging secondary or, if it is a secondary that loses contact with the primary, go into the initial NONE
state.
If the HAFW can be really trusted, it is possible to set this timeout to PT0S
, i.e zero, in which case the entire dead-node-detection mechanism in NSO is disabled.
The normal setup of an NSO HA cluster is to have all secondaries connected directly to the primary. This is a configuration that is both conceptually simple and reasonably straightforward to manage for the HAFW. In some scenarios, in particular a cluster with multiple secondaries at a location that is network-wise distant from the primary, it can however be sub-optimal, since the replicated data will be sent to each remote secondary individually over a potentially low-bandwidth network connection.
To make this case more efficient, we can instruct a secondary to be a relay for other secondaries, by invoking the Ha.beRelay()
method. This will make the secondary start listening on the IP address and port configured for HA in ncs.conf
, and handle connections from other secondaries in the same manner as the cluster primary does. The initial CDB copy (if needed) to a new secondary will be done from the relay secondary, and when the relay secondary receives CDB data for replication from its primary, it will distribute the data to all its connected secondaries in addition to updating its own CDB copy.
To instruct a node to become a secondary connected to a relay secondary, we use the Ha.beSecondary()
method as usual, but pass the node information for the relay secondary instead of the node information for the primary. I.e. the "sub-secondary" will in effect consider the relay secondary as its primary. To instruct a relay secondary to stop being a relay, we can invoke the Ha.beSecondary()
method with the same parameters as in the original call. This is a no-op for a "normal" secondary, but it will cause a relay secondary to stop listening for secondary connections, and disconnect any already connected "sub-secondaries".
This setup requires special consideration by the HAFW. Instead of just telling each secondary to connect to the primary independently, it must set up the secondaries that are intended to be relays, and tell them to become relays, before telling the "sub-secondaries" to connect to the relay secondaries. Consider the case of a primary M and a secondary S0 in one location, and two secondaries S1 and S2 in a remote location, where we want S1 to act as relay for S2. The setup of the cluster then needs to follow this procedure:
Tell M to be primary.
Tell S0 and S1 to be secondary with M as primary.
Tell S1 to be relay.
Tell S2 to be secondary with S1 as primary.
Conversely, the handling of network outages and node failures must also take the relay secondary setup into account. For example, if a relay secondary loses contact with its primary, it will transition to the NONE
state just like any other secondary, and it will then disconnect its sub-secondaries which will cause those to transition to NONE
too, since they lost contact with "their" primary. Or if a relay secondary dies in a way that is detected by its sub-secondaries, they will also transition to NONE
. Thus in the example above, S1 and S2 needs to be handled differently. E.g. if S2 dies, the HAFW probably won't take any action, but if S1 dies, it makes sense to instruct S2 to be a secondary of M instead (and when S1 comes back, perhaps tell S2 to be a relay and S1 to be a secondary of S2).
Besides the use of Ha.beRelay()
, the API is mostly unchanged when using relay secondaries. The HA event notifications reporting the arrival or the death of a secondary are still generated only by the "real" cluster primary. If the Ha.HaStatus()
method is used towards a relay secondary, it will report the node state as SECONDARY_RELAY
rather than just SECONDARY
, and the array of nodes will have its primary as the first element (same as for a "normal" secondary), followed by its "sub-secondaries" (if any).
When HA is enabled in ncs.conf
, CDB automatically replicates data written on the primary to the connected secondary nodes. Replication is done on a per-transaction basis to all the secondaries in parallel and is synchronous. When NSO is in secondary mode the northbound APIs are in read-only mode, that is the configuration can not be changed on a secondary other than through replication updates from the primary. It is still possible to read from for example NETCONF or CLI (if they are enabled) on a secondary. CDB subscriptions work as usual. When NSO is in the NONE
state CDB is unlocked and it behaves as when NSO is not in HA mode at all.
Unlike configuration data, operational data is replicated only if it is defined as persistent in the data model (using the tailf:persistent
extension).
Tool | Package | Required | Description |
---|---|---|---|
Parameters | Type | Description |
---|---|---|
Parameters | Type | Description |
---|---|---|
Parameters | Type | Description |
---|---|---|
Parameters | Type | Description |
---|---|---|
Parameter | Type | Description |
---|---|---|
Hostname | Address | Role |
---|---|---|
Hostname | Address | AS | Role |
---|---|---|---|
assume-nominal-role
join-ha
nominal-role
behaviour
true
false
primary
Assume primary role.
true
false
secondary
Attempt to connect as secondary to the node (if any) which has nominal-role primary. If this fails, make no retry attempts and assume none role.
true
false
none
Assume none role
false
true
primary
Attempt to join HA setup as secondary by querying for current primary. Retries will be attempted. Retry attempt interval is defined by /high-availability/settings/reconnect-interval
.
false
true
secondary
Attempt to join HA setup as secondary by querying for current primary. Retries will be attempted. Retry attempt interval is defined by /high-availability/settings/reconnect-interval
. If all retry attempts fail, assume none role.
false
true
none
Assume none role.
true
true
primary
Query HA setup once for a node with primary role. If found, attempt to connect as secondary to that node. If no current primary is found, assume primary role.
true
true
secondary
Attempt to join HA setup as secondary by querying for current primary. Retries will be attempted. Retry attempt interval is defined by /high-availability/settings/reconnect-interval
. If all retry attempts fail, assume none role.
true
true
none
Assume none role.
false
false
-
Assume none role.
be-primary
Order the local node to assume ha role primary
be-none
Order the local node to assume ha role none
be-secondary-to
Order the local node to connect as secondary to the provided HA node. This is an asynchronous operation, result can be found under /high-availability/status/be-secondary-result
local-node-id
Identify the which of the nodes in /high-availability/ha-node
(if any) corresponds to the local NSO instance
enable
Enable NSO rule-based HA and optionally assume a ha role according to /high-availability/settings/start-up/ parameters
disable
Disable NSO rule-based HA and assume a ha role none
ip
iproute2
yes
Adds and deletes the virtual IP from the network interface.
awk
mawk
or gawk
yes
Installed with most Linux distributions.
sed
sed
yes
Installed with most Linux distributions.
arping
iputils
or arping
optional
Installation recommended. Will reduce the propagation of changes to the virtual IP for layer-2 configurations.
gobgpd
and gobgp
GoBGP 2.x
optional
Required for layer-3 configurations. gobgpd is started by the HCC package and advertises the virtual IP using BGP. gobgp is used to get advertised routes.
nsupdate
bind-tools
or knot-dnsutils
optional
Required for layer-3 DNS update functionality and is used to submit Dynamic DNS Update requests to a name server.
enabled
boolean
If set to 'true', the primary node in an HA group automatically binds the set of Virtual IPv[46] addresses.
vip-address
list of inet:ip-address
The list of virtual IPv[46] addresses to bind on the primary node. The addresses are automatically unbound when a node becomes secondary. The addresses can therefore be used externally to reliably connect to the HA group primary node.
node-id
string
Unique node ID. A reference to /ncs:high-availability/ha-node/id
.
enabled
boolean
If set to true
, this node uses BGP to announce VIP addresses when in the HA primary state.
as
inet:as-number
The BGP Autonomous System Number for the local BGP daemon.
router-id
inet:ip-address
The router ID for the local BGP daemon.
address
inet:ip-address
BGP neighbor IP address.
as
inet:as-number
BGP neighbor Autonomous System Number.
ttl-min
uint8
Optional minimum TTL value for BGP packets. When configured enables BGP Generalized TTL Security Mechanism (GTSM).
password
string
Optional password to use for BGP authentication with this neighbor.
enabled
boolean
If set to true
, then an outgoing BGP connection to this neighbor is established by the HA group primary node.
enabled
boolean
If set to true
, DNS updates will be enabled.
fqdn
inet:domain-name
DNS domain-name for the HA primary.
ttl
uint32
Time to live for DNS record, default 86400.
key-file
string
Specifies the file path for nsupdate
keyfile.
server
inet:ip-address
DNS Server IP Address.
port
uint32
DNS Server port, default 53.
zone
inet:host
DNS Zone to update on the server.
timeout
uint32
Timeout for nsupdate
command, default 300.
node-id
string
Unique NSO HA node ID. Valid values are: /high-availability/ha-node when built-in HA is used or /ha-raft/status/member for HA Raft.
ip-address
inet:ip-address
IP where NSO listens for incoming requests to any northbound interfaces.
location
string
Name of the Location/Site/Availability-Zone where node is placed.
paris
192.168.23.99
Paris service node.
london
192.168.23.98
London service node.
vip4
192.168.23.122
NSO primary node IPv4 VIP address.
paris
192.168.31.99
64512
Paris node
london
192.168.30.98
64513
London node
router
192.168.30.2
192.168.31.2
64514
BGP-enabled router
vip4
192.168.23.122
Primary node IPv4 VIP address