Miscellaneous topics connected to NSO development.
Loading...
Loading...
Loading...
Loading...
Loading...
Schedule background tasks in NSO.
NSO includes a native time-based job scheduler suitable for scheduling background work. Tasks can be scheduled to run at particular times or periodically at fixed times, dates, or intervals. It can typically be used to automate system maintenance or administrative tasks.
A standard Vixie Cron expression is used to represent the periodicity in which the task should run. When the task is triggered, the configured action is invoked on the configured action node instance. The action is run as the user that configured the task.
Example: To schedule a task to run sync-from
at 2 AM on the 1st of every month, we do:
If the task was added through an XML init
file, the task will run with the system
user, which implies that AAA rules will not be applied at all. Thus, the task action will not be able to initiate device communication.
If the action node instance is given as an XPath 1.0 expression, the expression is evaluated with the root as the context node, and the expression must return a node-set. The action is then invoked on each node in this node-set.
Optionally, action parameters can be configured in XML format to be passed to the action during invocation.
Once the task has been configured, you can view the next run times of the task:
You could also see if the task is running or not:
A standard Vixie Cron expression is a string comprising five fields separated by white space that represents a set of times. The following rules can be used to create an expression.
The table below shows expression rules.
The following list describes the legal special characters and how you can use them in a Cron expression.
Star (*
). Selects all values within a field. For example, *
in the minute field selects every minute.
Comma (,
). Commas are used to specify additional values. For example, using MON,WED,FRI
in the day of week field.
Hyphen (-
). Hyphens define ranges. For example 1-5
in the day of week field indicates every day between Monday and Friday, inclusive.
Forward slash (/
). Slashes can be combined with ranges to specify increments. For example, */5
in the minutes field indicates every 5 minutes.
Compaction in NSO can take a considerable amount of time, during which transactions could be blocked. To avoid disruption, it might be advantageous to schedule compaction during times of low NSO utilization. This can be done using the NSO scheduler and a service. See examples.ncs/development-guide/periodic-compaction
for an example that demonstrates how to create a periodic compaction service that can be scheduled using the NSO scheduler.
The scheduler can also be used to configure non-recurring tasks that will run at a particular time.
A non-recurring task will by default be removed when it has finished executing. It will be up to the action to raise an alarm if an error occurs. The task can also be kept in the task list by setting the keep
leaf.
In a HA cluster, a scheduled task will by default be run on the primary HA node. By configuring the ha-mode
leaf a task can be scheduled to run on nodes with a particular HA mode, for example scheduling a read-only action on the secondary nodes. More specifically, a task can be configured with the ha-node-id
to only run on a certain node. These settings will not have any effect on a standalone node.
The scheduler is disabled when HA is enabled and when HA mode is NONE
. See Mode of Operation in HA for more details.
Troubleshooting information is covered below.
To find out whether a scheduled task has run successfully or not, the easiest way is to view the history log of the scheduler. It will display the latest runs of the scheduled task.
Detailed information from the XPath evaluator can be enabled and made available in the xpath log. Add the following snippet to ncs.conf
.
Error information is written to the development log. The development log is meant to be used as support while developing the application. It is enabled in ncs.conf
:
While investigating a failure with a scheduled task or performing maintenance on the system, like upgrading, it might be useful to suspend the scheduler temporarily.
When ready the scheduler can be resumed.
Field | Allowed values | Allowed special characters |
---|---|---|
Minutes
0-59
* , - /
Hours
0-23
* , - /
Day of month
1-31
* , - /
Month
1-12 or JAN-DEC
* , - /
Day of week
0-6 or SUN-SAT
* , - /
Send the log data to an external command.
As a development feature, NSO supports sending log data as-is to an external command for reading on standard input. As this is a development feature, there are a few limitations, such as the data sent to the external command is not guaranteed to be processed before the external application is shut down.
The general configuration of the external log processing is done in ncs.conf
. Global and per-device settings controlling the external log processing for NED trace logs are stored in the CDB.
To enable external log processing, set /ncs-config/logs/external
to true
and /ncs-config/logs/command
to the full path of the command that will receive the log data. The same executable will be used for all log types.
External configuration example:
To support the debugging of the external log command behavior, a separate log file is used. This debugging log is configured under /ncs-config/logs/ext-log
. The example below shows the configuration for ./logs/external.log
with the highest log level set:
By default, NED trace output is written to the file preserving backward compatibility. To write NED trace logs to file for all but the device test
, which will use external log processing, the following configuration can be entered in the CLI:
When setting both external
and file
bits without setting /ncs-config/logs/external
to true
, a warning message will be logged to ext-log
. When only setting the external
bit, no logging will be done.
After enabling external log processing, NSO will start one instance of the external command for each configured log destination. Processing of the log data is done by reading from standard input and processing it as required.
The command line arguments provide information about the log that is being processed and in what format the data is sent.
The example below shows how the configured command ./log_processor
would be executed for NETCONF trace data configured to log in raw mode:
Command line argument position and meaning:
version
: Protocol version, always set to 1
. Added for forward compatibility.
action
: The action being performed. Is always set to log
. Added for forward compatibility.
name:
Name of the log being processed.
log-type
: Type of log data being processed. For all but NETCONF and NED trace logs, this is set to system
. Depending on the type of NED one of ned-trace-java
, ned-trace-netconf
and ned-trace-snmp
is used. NETCONF trace is set to netconf-trace
.
log-mode
: Format of log data being sent. For all but NETCONF and NED trace logs, this will be raw
. NETCONF and NED trace logs can be pretty-printed and then the format will be pretty
.
Use NSO's embedded web server to deliver dynamic content.
This page describes an embedded basic web server that can deliver static and Common Gateway Interface (CGI) dynamic content to a web client, commonly a browser. Due to the limitations of this web server, and/or of its configuration capabilities, a proxy server such as Nginx is recommended to address special requirements.
The web server can be configured through settings in ncs.conf
. See the Manual Pages section Configuration Parameters.
Here is a brief overview of what you can configure on the web server:
toggle web server
: the web server can be turned on or off.
toggle transport
: enable HTTP and/or HTTPS, set IPs, ports, redirects, certificates, etc.
hostname
: set the hostname of the web server and decide whether to block requests for other hostnames.
/
: set the docroot
from where all static content is served.
/login
: set the docroot
from where static content is served for URL paths starting with /login
.
"/custom": set the docroot
from where static content is served for URL paths starting with /custom
.
/cgi
: toggle CGI support and set the docroot
from where dynamic content is served for URL paths starting with /cgi
.
non-authenticated paths
: by default, all URL paths, except those needed for the login page are hidden from non-authenticated users; authentication is done by calling the JSON-RPC login
method.
allow symlinks
: Allow symlinks from under the docroot
.
cache
: set the cache time window for static content.
log
: several logs are available to configure in terms of file paths - an access log, a full HTTP traffic/trace log, and a browser/JavaScript log.
custom headers
: set custom headers across all static and dynamic content, including requests to /jsonrpc
.
In addition to what is configurable, the web server also GZip-compresses responses automatically if the browser handles such responses, either by compressing the response on the fly or, if requesting a static file, like /bigfile.txt
, by responding with the contents of /bigfile.txt.gz
, if there is such a file.
The web server includes CGI functionality, disabled by default. Once you enable it in ncs.conf
(see Configuration Parameters in manual Pages), you can write CGI scripts, that will be called with the following NSO environment variables prefixed with NCS_ when a user has logged in via JSON-RPC:
JSONRPC_SESSIONID
: the JSON-RPC session id (cookie).
JSONRPC_START_TIME
: the start time of the JSON-RPC session.
JSONRPC_END_TIME
: the end time of the JSON-RPC session.
JSONRPC_READ
: the latest JSON-RPC read transaction.
JSONRPC_READS
: a comma-separated list of JSON-RPC read transactions.
JSONRPC_WRITE
: the latest JSON-RPC write transaction.
JSONRPC_WRITES
: a comma-separated of JSON-RPC write transactions.
MAAPI_USER
: the MAAPI username.
MAAPI_GROUPS
: a comma-separated list of MAAPI groups.
MAAPI_UID
: the MAAPI UID.
MAAPI_GID
: the MAAPI GID.
MAAPI_SRC_IP
: the MAAPI source IP address.
MAAPI_SRC_PORT
: the MAAPI source port.
MAAPI_USID
: the MAAPI USID.
MAAPI_READ
: the latest MAAPI read transaction.
MAAPI_READS
: a comma-separated list of MAAPI read transactions.
MAAPI_WRITE
: the latest MAAPI write transaction.
MAAPI_WRITES
: a comma-separated of MAAPI write transactions.
Server or HTTP-specific information is also exported as environment variables:
SERVER_SOFTWARE:
SERVER_NAME:
GATEWAY_INTERFACE:
SERVER_PROTOCOL:
SERVER_PORT:
REQUEST_METHOD:
REQUEST_URI:
DOCUMENT_ROOT:
DOCUMENT_ROOT_MOUNT:
SCRIPT_FILENAME:
SCRIPT_TRANSLATED:
PATH_INTO:
PATH_TRANSLATED:
SCRIPT_NAME:
REMOTE_ADDR:
REMOTE_HOST:
SERVER_ADDR:
LOCAL_ADDR:
QUERY_STRING:
CONTENT_TYPE:
CONTENT_LENGTH:
HTTP_*"
: HTTP headers e.g. "Accept" value exported as HTTP_ACCEPT
.
The tailf-tls.yang
YANG module defines a structure to store TLS data in the database. It is possible to store the private key, the private key's passphrase, the public key certificate, and CA certificates.
To enable the web server to fetch TLS data from the database, ncs.conf
needs to be configured.
Note that the options key-file
, cert-file
, and ca-cert-file
, are ignored when read-from-db
is set to true. See the ncs.conf.5 man page for more details.
The database is populated with TLS data by configuring the /tailf-tls:tls/private-key, /tailf-tls:tls/certificate
, and, optionally, /tailf-tls/ca-certificates
. It is possible to use password-protected private keys, then the passphrase leaf in the private-key
container needs to be set to the password of the encrypted private key. Unencrypted private key data can be supplied in both PKCS#8 and PKCS#1 format, while encrypted private key data needs to be supplied in PKCS#1 format.
In the following example a password-protected private key, the passphrase, a public key certificate, and two CA certificates are configured with the CLI.
The SHA256 fingerprints of the public key certificate and the CA certificates can be accessed as operational data. The fingerprint is shown as a hex string. The first octet identifies what hashing algorithm is used, 04 is SHA256, and the following octets is the actual fingerprint.
When the database is populated, NSO needs to be reloaded.
After configuring NSO, populating the database, and reloading, the TLS transport is usable.
The web server includes support for uploading packages to /package-upload
using HTTP POST
from the local host to the NSO host, making them installable there. It is disabled by default but can be enabled in ncs.conf
; see Configuration Parameters in Manual Pages.
By default, only uploading 1 file per request will be processed and any remaining file parts after that will result in an error and its content will be ignored. To allow multiple files in a request you can increase /ncs-config/webui/package-upload/max-files
.
The AAA infrastructure can be used to restrict access to library functions using command rules:
Note how the command is prefixed with ::webui::
. This tells the AAA engine to apply the command rule to WebUI API functions. You can read more about command rules in AAA infrastructure.
Configure NSO to receive SNMP notifications.
NSO can act as an SNMP notification receiver (v1, v2c, v3) for its managed devices. The application can register notification handlers and react to the notifications, for example by mapping SNMP notifications to NSO alarms.
The notification receiver is started in the Java VM by application code, as described below. The application code registers the handlers, which are invoked when a notification is received from a managed device. The NSO operator can enable and disable the notification receiver as needed. The notification receiver is configured in the /snmp-notification-receiver
subtree.
By default, nothing happens with SNMP Notifications. You need to register a function to listen to traps and do something useful with the traps. First of all, SNMP var-binds are typically sparse in information and in many cases, you want to do enrichment of the information and map the notification to some meaningful state. Sometimes a notification indicates an alarm state change, sometimes it indicates that the configuration of the device has changed. The action based on the two above examples is very different; in the first case, you want to interpret the notification for meaningful alarm information and submit a call to the NSO Alarm Manager. In the second case, you probably want to initiate a check-sync, compare-config, sync action
sequence.
The NSO operator must enable the SNMP notification receiver, and configure the addresses NSO will use to listen for notifications. The primary parameters for the Notification receiver are shown below.
The notification reception can be turned on and off using the enabled lead. NSO will listen to notifications at the end-points configured in listen
. There is no need to manually configure the NSO engine-id
. NSO will do this automatically using the algorithm described in RFC 3411. However, it can be assigned an engine-id
manually by setting this leaf.
The managed devices must also be configured to send notifications to the NSO addresses.
NSO silently ignores any notification received from unknown devices. By default, NSO uses the /devices/device/address
leaf, but this can be overridden by setting /devices/device/snmp-notification-address
.
There are some standard built-in filters for the SNMP notification receiver that perform standard tasks:
Standard filter for suppression of received SNMP events that are not of type TRAP
, NOTIFICATION
, or INFORM
.
Standard filter for suppression of notifications emanating from IP addresses outside the defined set of addresses. This filter determines the source IP address first from the snmpTrapAddress
1.3.6.1.6.3.18.1.3 varbind if this is set in the PDU, or otherwise from the emanating peer IP address. If the resulting IP address does not match either the snmp-notification-address
or the address
leaf of any device in the device model, this notification is discarded.
Standard filter that will acknowledge INFORM notification automatically.
NSO uses the Java package SNMP4J to parse the SNMP PDUs.
Notification Handlers are user-supplied Java classes that implement the com.tailf.snmp.snmp4j.NotificationHandler
interface. The processPDU
method is expected to react on the SNMP4J event, e.g. by mapping the PDU to an NSO alarm. The handlers are registered in the NotificationReceiver
. The NotificationReceiver
is the main class that in addition to maintaining the handlers also has the responsibility to read the NSO SNMP notification configuration, and set up SNMP4J
listeners accordingly.
An example of a notification handler can be found at $NCS_DIR/examples.ncs/snmp-notification-receiver
. This example handler receives notifications and sets an alarm text if the notification is an IF-MIB::linkDown trap
.
The instantiation and start of the NotificationReceiver
as well as registration of notification handlers are all expected to be done in the same application component of some NSO package. The following is an example of such an application component:
Manage and work with NSO encryption keys.
By using the tailf:des3-cbc-encrypted-string
, tailf:aes-cfb-128-encrypted-string
or the tailf:aes-256-cfb-128-encrypted-string
built-in types, it is possible to store encrypted values in NSO. The keys used to encrypt these values are configured in ncs.conf
and default stored in ncs.crypto_keys
.
NSO supports reading encryption keys using an external command instead of storing them in ncs.conf
to allow for use with external key management systems. To use this feature, set /ncs-config/encrypted-strings/external-keys/command
to an executable command that will output the keys following the rules described in the following sections. The command will be executed on startup and when NSO reloads the configuration.
If the external command fails during startup, the startup will abort. If the command fails during a reload, the error will be logged and the previously loaded keys will be kept in the system.
The process of providing encryption keys to NSO can be described by the following three steps:
Read the configuration from the environment.
Read encryption keys.
Write encryption keys or the error on standard output.
The value of /ncs-config/encrypted-strings/external-keys/command-argument
is available in the command as the environment variable NCS_EXTERNAL_KEYS_ARGUMENT
. The value of this configuration is only used by the configured command.
The external command should return the encryption keys on standard output using the names as shown in the table below. The encryption key values are in hexadecimal format, just as in ncs.conf
. See the example below for details.
The following table shows the mapping from the name to the path in the configuration.
Name | Configuration path |
---|---|
To signal an error, including ERROR=message
is preferred. A non-zero exit code or unsupported line content will also trigger an error. Any form of error will be logged to the development log and no encryption keys will be available in the system.
Example output providing all supported encryption key configuration settings:
Example error output:
Below is a complete example of an application written in Python providing encryption keys from a plain text file. The application is included in the example crypto/external_keys
:
DES3CBC_KEY1
/ncs-config/encrypted-strings/DES3CBC/key1
DES3CBC_KEY2
/ncs-config/encrypted-strings/DES3CBC/key2
DES3CBC_KEY3
/ncs-config/encrypted-strings/DES3CBC/key3
DES3CBC_IV
/ncs-config/encrypted-strings/DES3CBC/initVector
AESCFB128_KEY
/ncs-config/encrypted-strings/AESCFB128/key
AESCFB128_IV
/ncs-config/encrypted-strings/AESCFB128/initVector
AES256CFB128_KEY
/ncs-config/encrypted-strings/AES256CFB128/key