1. General
1.1 Extract the NED package
1.2 Install the NED package
1.2.1 Local install
1.2.2 System install
1.3 Configure the NED in NSO
2. Optional debug and trace setup
3. Dependencies
4. Sample device configuration
5. Built in live-status actions
6. Built in live-status show
7. Limitations
8. How to report NED issues and feature requests
9. How to rebuild a NED
10. Configure the NED to use ssh multi factor authentication
11. Aflex scripts
12. NED Secrets - Securing your Secrets
1. General
This document describes the a10-acos NED.
Additional README files bundled with this NED package
+---------------------------+------------------------------------------------------------------------------+
| Name | Info |
+---------------------------+------------------------------------------------------------------------------+
| README-ned-settings.md | Information about all run time settings supported by this NED. |
+---------------------------+------------------------------------------------------------------------------+
Common NED Features
+---------------------------+-----------+------------------------------------------------------------------+
| Feature | Supported | Info |
+---------------------------+-----------+------------------------------------------------------------------+
| netsim | yes | Default emulated device: AX Series Advanced Traffic Manager |
| | | v2.6.1-GR1 |
| | | |
| check-sync | yes | check-sing using trans-id |
| | | |
| partial-sync-from | yes | This feature is supported by filtering the needed config from a |
| | | full show (device does not support partial show) |
| | | |
| live-status actions | yes | Supports most of the device needed actions |
| | | |
| live-status show | no | The NED does not implement TTL-based data |
| | | |
| load-native-config | no | |
+---------------------------+-----------+------------------------------------------------------------------+
Custom NED Features
+---------------------------+-----------+------------------------------------------------------------------+
| Feature | Supported | Info |
+---------------------------+-----------+------------------------------------------------------------------+
| device partitions | yes | The NED supports device partitions by using config active- |
| | | partition list |
| | | |
| ned-secrets | yes | NED supports device-enctypted password caching. Please check |
| | | README.md |
+---------------------------+-----------+------------------------------------------------------------------+
admin@ncs(config)# devices device dev-1 sync-from
result true
If the sync-from was not successful, check the NED configuration again.
2. Optional debug and trace setup
It is often desirable to see details from when and how the NED interacts with the device(Example: troubleshooting)
This can be achieved by configuring NSO to generate a trace file for the NED. A trace file
contains information about all interactions with the device. Messages sent and received as well
as debug printouts, depending on the log level configured.
NSO creates one separate trace file for each device instance with tracing enabled.
Stored in the following location:
The log level 'info' is used by default and the 'debug' level is the most verbose.
IMPORTANT:
Tracing shall be used with caution. This feature does increase the number of IPC messages sent
between the NED and NSO. In some cases this can affect the performance in NSO. Hence, tracing should
normally be disabled in production systems.
An alternative method for generating printouts from the NED is to enable the Java logging mechanism.
This makes the NED print log messages to common NSO Java log file.
IMPORTANT:
Java logging does not use any IPC messages sent to NSO. Consequently, NSO performance is not
affected. However, all log printouts from all log enabled devices are saved in one single file.
This means that the usability is limited. Typically single device use cases etc.
3. Dependencies
This NED has the following host environment dependencies:
Java 1.8 (NSO version < 6.2)
Java 17 (NSO version >= 6.2)
Gnu Sed
Dependencies for NED recompile:
Apache Ant
Bash
Gnu Sort
Gnu awk
Grep
Python3 (with packages: re, sys, getopt, subprocess, argparse, os, glob)
4. Sample device configuration
The following is an example of configuration data (CLI NED commands) that can be sent to an a10-acos device:
There are two main categories of commands that can be sent using a10-acos NED: configuration commands (RPC's that are sent from the device configuration) and privileged commands (RPC's that are sent from privileged mode).
Each main RPC category also divides in the following sub-categories: simple commands that does not use additional prompts (eg "show configuration"), interactive commands that uses additional prompts (eg a RPC's that requests username/password or any other prompts) and internal NED commands, that does not interact with the device but with the NED.
The last category is supported but not implemented for a specific feature.
Simple command format is as follows:
action { action-payload "RPC CLI command" }
Interactive command contains the simple command and adds the following list:
action { action-payload "import bw-list bl1 use-mgmt-port scp://11.11.11.11/file" interaction { prompt-pattern "User name.*" value myuser } interaction { prompt-pattern Password.* value mypassword } interaction { prompt-pattern "Do you want to overwrite.*" value yes } interaction { prompt-pattern \"Do you want to save the remote host information.*\" value no } }
In the above command, the interaction list defines each prompt that it is expected from the device, along with its corresponding value. Note that prompt-pattern is compiled in a regular expression, so special characters that are expected from prompts should be escaped.
The order of the interaction list definition is not important, but all the possible expected prompts should be defined. For example, in the above command, the prompt "Do you want to overwrite.*" is only active when the file exists.
Internal commands looks the same as simple commands, but also contain the keyword "internal":
action { action-payload "Internal RPC" internal }
All the above command sub-categories can be chained and sent in the same request, by defining a list of actions:
The new implementation of RPC action execution allows a more structured way of sending multiple RPC's, by using its XML format:
<?xml version="1.0" encoding="UTF-8"?><fragment xmlns="http://www.tailf.com">
<a10-acos-stats:action xmlns:a10-acos-stats="http://tail-f.com/ned/a10-acos-stats">
<a10-acos-stats:action-payload>import bw-list bl1 use-mgmt-port scp://11.11.11.11/file</a10-acos-stats:action-payload>
<a10-acos-stats:interaction>
<a10-acos-stats:prompt-pattern>User name.*</a10-acos-stats:prompt-pattern>
<a10-acos-stats:value>admin</a10-acos-stats:value>
</a10-acos-stats:interaction>
<a10-acos-stats:interaction>
<a10-acos-stats:prompt-pattern>Password.*</a10-acos-stats:prompt-pattern>
<a10-acos-stats:value>admin</a10-acos-stats:value>
</a10-acos-stats:interaction>
<a10-acos-stats:interaction>
<a10-acos-stats:prompt-pattern>Do you want to overwrite.*</a10-acos-stats:prompt-pattern>
<a10-acos-stats:value>yes</a10-acos-stats:value>
</a10-acos-stats:interaction>
<a10-acos-stats:interaction>
<a10-acos-stats:prompt-pattern>Do you want to save the remote host information.*</a10-acos-stats:prompt-pattern>
<a10-acos-stats:value>no</a10-acos-stats:value>
</a10-acos-stats:interaction>
</a10-acos-stats:action>
<a10-acos-stats:action xmlns:a10-acos-stats="http://tail-f.com/ned/a10-acos-stats">
<a10-acos-stats:action-payload>show running</a10-acos-stats:action-payload>
</a10-acos-stats:action>
</fragment>
6. Built in live-status show
The NED does not support TTL-based live-status data
7. Limitations
The NED CLI does not implement the device behavior 1 to 1, since the device is not a netconf-compatible device. Also, the NED may use yang model workarounds (like node alternative naming) in order to support device behavior.
8. How to report NED issues and feature requests
Issues like bugs and errors shall always be reported to the Cisco NSO NED team through
the Cisco Support channel:
The following information is required for the Cisco NSO NED team to be able
to investigate an issue:
- A detailed recipe with steps to reproduce the issue.
- A raw trace file generated when the issue is reproduced.
- SSH/TELNET access to a device where the issue can be reproduced by the Cisco NSO NED team.
This typically means both read and write permissions are required.
Pseudo access via tools like Webex, Zoom etc is not acceptable.
However, it is ok with device access through VPNs, jump servers etc though.
Do as follows to gather the necessary information needed for your device, here named 'dev-1':
Reproduce the found issue using ncs_cli or your NSO service.
Write down each necessary step in a reproduction report.
In addition to this, it helps if you can show how it should work
by manually logging into the device using SSH/TELNET and type
the relevant commands showing a successful operation.
Gather the reproduction report and a copy of the raw trace file
containing data recorded when the issue happened.
Contact the Cisco support and request to open a case. Provide the gathered files
together with access details for a device that can be used by the
Cisco NSO NED when investigating the issue.
Requests for new features and extensions of the NED are handled by the Cisco NSO NED team when
applicable. Such requests shall also go through the Cisco support channel.
The following information is required for feature requests and extensions:
Set the config on the real device including all existing dependent config
and run sync-from to show it in the trace.
Run sync-from # devices device dev-1 sync-from
Attach the raw trace to the ticket
List the config you want implemented in the same syntax as shown on the device
SSH/TELNET access to a device that can be used by the Cisco NSO NED team for testing and verification
of the new feature. This usually means that both read and write permissions are required.
Pseudo access via tools like Webex, Zoom etc is not acceptable. However, it is ok with access
through VPNs, jump servers etc as long as we can connect to the NED via SSH/TELNET.
9. How to rebuild a NED
To rebuild the NED do as follows:
> cd $NED_ROOT_DIR/src
> make clean all
When the NED has been successfully rebuilt, it is necessary to reload the package into NSO.
admin@ncs# packages reload
10. Configure the NED to use ssh multi factor authentication
This NED supports multi factor authentication (MFA) using the ssh authentication
method 'keyboard-interactive'.
Some additional steps are required to enable the MFA support:
Verify that your NSO version supports MFA. This is configurable as additional
settings in the authentication group used by the device instance.
Enter a NSO CLI and enter the following and do tab completion:
> ncs_cli -C -u admin
admin@ncs# show running-config devices authgroups group default default-map <tab>
Possible completions:
action-name The action to call when a notification is received.
callback-node Invoke a standalone action to retrieve login credentials for managed devices on the 'callback-node' instance.
mfa Settings for handling multi-factor authentication towards the device
public-key Use public-key authentication
remote-name Specify device user name
remote-password Specify the remote password
remote-secondary-password Second password for configuration
same-pass Use the local NCS password as the remote password
same-secondary-password Use the local NCS password as the remote secondary password
same-user Use the local NCS user name as the remote user name
If 'mfa' is displayed in the output like above, NSO has MFA support enabled.
In case MFA is not supported it is necessary to upgrade NSO before proceeding.
Implement the authenticator executable. The MFA feature relies on an external executable to take care of the client part
of the multi factor authentication. The NED will automatically call this executable for each challenge presented by the
ssh server and expects to get a proper response in return.
The executable can be a simple shell script or a program implemented in any programming language.
The required behaviour is like this:
read one line from stdin
The line passed from the NED will be a semi colon separated string containing the following info:
[<device name>;<user>;<password>;<opaque>;<ssh server name>;<ssh server instruction>;<ssh server prompt>;]
The elements for device name, user, password and opaque corresponds to what has been configured in NSO.
The ssh server name, instruction and prompt are given by the ssh server during the authentication step.
Each individual element in the semi colon separated list is Base64 encoded.
Extract the challenge based on the contents above.
Print a response matching the challenge to stdout and exit with code 0
In case a matching response can not be given do exit with code 2
Below is a simple example of an MFA authenticator implemented in Python3:
#!/usr/bin/env python3
import sys
import base64
# This is an example on how to implement an external multi factor authentication handler
# that will be called by the NED upon a ssh 'keyboard-interactive' authentication
# The handler is reading a line from stdin with the following expected format:
# [<device name>;<user>;<password>;<opaque>;<ssh server name>;<ssh server instruction>;<ssh server prompt>;]
# All elements are base64 encoded.
def decode(arg):
return str(base64.b64decode(arg))[2:-1]
if __name__ == '__main__':
query_challenges = {
"admin@localhost's password: ":'admin',
'Enter SMS passcode:':'secretSMScode',
'Press secret key: ':'2'
}
# read line from stdin and trim brackets
line = sys.stdin.readline().strip()[1:-1]
args = line.split(';')
prompt = decode(args[6])
if prompt in query_challenges.keys():
print(query_challenges[prompt])
exit(0)
else:
exit(2)
Configure the authentication group used by the device instance to enable MFA. There
are two configurables available:
executable The path to the external multi factor authentication executable (mandatory).
opaque Opaque data that will passed as a cookie element to the executable (optional).
> ncs_cli -C -u admin
admin@ncs# config
Entering configuration mode terminal
admin@ncs(config)# devices authgroups group <name> default-map mfa executable <path to the executable>
admin@ncs(config)# devices authgroups group <name> default-map mfa opaque <some opaque data>
admin@ncs(config)# commit
Try connecting to the device.
10.1 Trouble shooting
In case of connection problems the following steps can help for debugging:
Note: after running this command, it is mandatory to do a sync-from, since the device will introduce the aflex section in the running-config
Once the aflex is activated, aflex scripts can be managed.
Note: since the device aflex scripts are multi-liners, the script payload needs to be properly escaped by replacing newlines - eg '\n' character with '\n'. Also double quote - eg " needs to be escaped: '"' -> '"'
Also, every script should end in a '\n'
Note2: since aflex script is a type string leaf in the NED, the script content needs to be quoted:
For example, to create a new script that would look like this on the device:
vThunder(NOLICENSE)#show aflex test4
Name: test4
Syntax: Check
Virtual port: No
Content:
when HTTP_RESPONSE {
# Check Content-Data to avoid unnecessary collects
if { [HTTP::header "Content-Type"] contains "text" } {
HTTP::collect
}
}
the equivalent config on the NED will look like this:
admin@ncs(config-config)# aflex test2
admin@ncs(config-aflex-test2)# script "when HTTP_RESPONSE {\n # Check Content-Data to avoid unnecessary collects\n if { [HTTP::header \"Content-Type\"] contains \"text\" } {\n HTTP::collect\n }\n}\n"
admin@ncs(config-aflex-test2)# commit dry-run outformat native
native {
device {
name vThunder-8
data aflex create test2
when HTTP_RESPONSE {
# Check Content-Data to avoid unnecessary collects
if { [HTTP::header "Content-Type"] contains "text" } {
HTTP::collect
}
}
.
}
Disabling aflex support is done by the following command:
admin@ncs(config-config)# config-actions action { action-payload "no running-config display aflex" }
12. NED Secrets - Securing your Secrets
It is best practice to avoid storing your secrets (e.g. passwords and
shared keys) in plain-text, either on NSO or on the device. In NSO we
support multiple encrypted datatypes that are encrypted using a local
key, similarly many devices such as ALU-SR supports automatically
encrypting all passwords stored on the device.
Naturally, for security reasons, NSO in general has no way of
encrypting/decrypting passwords with the secret key on the
device. This means that if nothing is done about this we will
become out of sync once we write secrets to the device.
In order to avoid becoming out of sync the NED reads back these elements
immediately after set and stores the encrypted value(s) in a special
`secrets` table in oper data. Later on, when config is read from the
device, the NED replaces all cached encrypted values with their plaintext
values; effectively avoiding all config diffs in this area. If the values
are changed on the device, the new encrypted value will not match the
cached pair and no replacement will take place. This is desired, since out
of band changes should be detected.
This handles the device-side encryption, but passwords are still unencrypted
in NSO. To deal with this we support using NSO-encrypted strings instead of
plaintext passwords in the NSO data model.
--- Handling auto-encryption
Let us say that we have password-encryption on and we want to write a new
user to our device:
system
security
user admin
access console
console
member administrative
exit
password <admin-password>
this will be automatically encrypted by the device
*A:VSR-7750>config# system security user "admin"
*A:VSR-7750>config>system>security>user# info
----------------------------------------------
password "$2y$10$fBSDYG2MHpdpCTDQhq7BE.ojwFR5z10g61PUqWaXb52GXg0Ge8d8W"
access console
console
member "administrative"
exit
----------------------------------------------
But the secrets management will store this new encrypted value in our `secrets` table:
admin@ncs# show devices device dev-1 ned-settings secrets
ID ENCRYPTED REGEX
---------------------------------------------------------------------------------------------------------------
/system/security/user_admin_/password/id $2y$10$fBSDYG2MHpdpCTDQhq7BE.ojwFR5z10g61PUqWaXb52GXg0Ge8d8W -
which means that compare-config or sync-from will not show any
changes and will not result in any updates to CDB". In fact, we can
still see the unencrypted value in the device tree:
admin@ncs(config-config)# show full sys sec user
devices device dev-1
config
system
security
user admin
access console
console
member administrative
!
password <admin-password>
--- Increasing security with NSO-side encryption
We have two alternatives, either we can manually encrypt our values using
one of the NSO-encrypted types (e.g `aes-256-cfb-128-encrypted-string`) and
set them to the tree, or we can recompile the NED to always encrypt secrets.
--- Setting encrypted value
Let us say we know that the NSO-encrypted string
`$2y$10$7ova9fF/bRe9B9GUtjVpA.w5mfeXJXRHyV0KsSfg4XWE9j3Fcq3Qi`, we
can then set it in the device tree as normal
admin@ncs(config-config)# system security user admin password $2y$10$7ova9fF/bRe9B9GUtjVpA.w5mfeXJXRHyV0KsSfg4XWE9j3Fcq3Qi
admin@ncs(config-config)# commit
when commiting this value it will be decrypted and the plaintext will be written to the device.
Unlike the previous example the plaintext is not visible in the device tree:
admin@ncs(config-config)# show full sys sec user
devices device dev-1
config
system
security
user admin
access console
console
member administrative
!
password $2y$10$7ova9fF/bRe9B9GUtjVpA.w5mfeXJXRHyV0KsSfg4XWE9j3Fcq3Qi
On the device side this plaintext value is of course encrypted
with the device key, and just as before we store it in our
`secrets` table:
admin@ncs# show devices device dev-1 ned-settings secrets
ID ENCRYPTED REGEX
---------------------------------------------------------------------------------------------------------------
/system/security/user_admin_/password/id $2y$10$fBSDYG2MHpdpCTDQhq7BE.ojwFR5z10g61PUqWaXb52GXg0Ge8d8W -
We can see that this corresponds to the value set on the device.
--- Auto-encrypting passwords in NSO
To avoid having to pre-encrypt your passwords you can rebuild your NED in your OS
command shell specifying an encrypted type for secrets using a command like:
yourhost:~/ned-folder$ NEDCOM_SECRET_TYPE="tailf:aes-cfb-128-encrypted-string" make -C src/ clean all
Or by adding the line `NED_EXTRA_BUILDFLAGS ?= NEDCOM_SECRET_TYPE=tailf:aes-cfb-128-encrypted-string`
in top of the `Makefile` located in <alu-sr-folder>/src directory.
Doing this means that even if the input to a password is a plaintext string, NSO will always
encrypt it, and you will never see plain text secrets in the device tree.
If we reload our example with the new NED all of the secrets are now encrypted:
admin@ncs(config-config)# show full sys sec user
devices device dev-1
config
system
security
user admin
access console
console
member administrative
!
password $2y$10$7ova9fF/bRe9B9GUtjVpA.w5mfeXJXRHyV0KsSfg4XWE9j3Fcq3Qi
