OpenADR 2.0b Certification Test Harness Users Manual

Quick Start The following steps will get the test harness running in a localhost environment in just a few minutes: Step 1 Install a version of Eclipse on your computer (See Eclipse Configuration on page 9). Step 2 Unzip the Eclipse project OpenADR20bCertTest_v1_x_x into your eclipse working directory. Import the project into your Eclipse environment. Step 3 Start Eclipse and expand the tree in Package Explorer srctestcasespushregisterParty Step 4 Run a VEN registration test case against a VTN test in a self test mode over http as follows (Shortcut: run SelfTest_Register in util): • Run venN0_5020_TH_VTN_1 • Click Resume on first two prompts • Run vtnN0_5020_TH_VEN • Click Resume on prompt • Watch the console windows as test cases execute Step 5 To view a detailed transaction log (starting with TraceLog_xxx), you can look in the log folder off the Eclipse root (you might need to refresh the folder) or point a browser at the logfile.html file in the same folder for a prettier interface. (Shortcut: run LogFiles_View in util) Step 6 Now lets run the same tests over XMPP. Unzip openfire_Windows_xxxxx or openfire_Linux_xxxxx file into a system folder. Step 7 Start Openfire from the bin directory. On Windows use openfire.exe or OpenADR_Start.Bat if you encounter issues. On Linux use.openfire start. Optional Step 8 You can start the Openfire console at 127.0.0.1:9090 from a browser. Enter username as admin and password as password. Ignore warning about RSA cert not matching server domain. Step 9 Modify the selftest properties located in the Eclipse Package Tree at srcproperties vendor.selftest.openadrconfig.properties as follows: Transport_Type=XMPP. Step 10 Repeat steps 3, 4, and 5 to run the same registration test case over XMPP. You have now successfully run test cases across both transports Suggested next steps: • Review Appendix B to get a feel for how the test harness is configured. • Skim the balance of this user manual so you know what information is available. • Set up your device, configure the test harness, and start running tests

OpenADR 2.0b

Certification Test Harness

User's Manual

Version 1.1.1

1.0.5  Updated Appendix D for 1.0.5 release

 Deleted Appendix E and open issues documented here have been resolved

09/11/13 - JZ 1.0.6  Updated Appendix D for 1.0.6 release ECC certificate support is added 03/19/14 - SK 1.0.7  SHA1 removal

Many other edits

11/1/15 – JZ

Table of Contents

Quick Start 5

Overview 6

Installation 9

Java 9

Eclipse IDE for Java Developers 9

Eclipse Configuration 9

Test Harness installation 10

Openfire Installation 11

Configuration 11

Test Harness Configuration Mapping 12

Transport and Security Configuration 13

OpenFire Configuration 14

Running Test Cases 16

Test Case Naming 16

Test Logs 16

Self Test Routines 17

Optional Test Cases 18

Registration DB and Utilities 18

Helpful Hints 19

Active Event Indicator 19

Set polling to 10 seconds 19

Time Synchronization 19

Errors in the Log Window 19

Failure information in test reports 19

Processes Left Running 19

Clearing the event data store 19

Conformance Rule Disable 20

Self Tests 20

Eclipse Getting Confused 20

Firewalls 20

Wireshark 20

Disabling TH Host Verification 20

Disabling DUT Host Verification 20

TLS Console Debug Listings 20

Error when launching browser or log 21

Appendix A – Configuration Tables 22

These following tables use the following addresses to illustrate the configuration options: 22

Pull VEN Configuration 22

Push VEN Configuration 23

Pull VTN Configuration 24

Push VTN Configuration 25

Appendix B – Test Harness Configuration Properties 26

Appendix C – Security Configuration Properties 31

Appendix D – Updated Tests 32

Quick Start

The following steps will get the test harness running in a localhost environment in just a few minutes:

Step 1 - Install a version of Eclipse on your computer (See Eclipse Configuration on page 9).

Step 2 - Unzip the Eclipse project OpenADR20bCertTest_v1_x_x into your eclipse working directory Import the project into your Eclipse environment

Step 3 - Start Eclipse and expand the tree in Package Explorer src/testcases/push/registerParty/

Step 4 - Run a VEN registration test case against a VTN test in a self test mode over http as

follows (Shortcut: run SelfTest_Register in util):

 Run ven/N0_5020_TH_VTN_1

 Click Resume on first two prompts

 Run vtn/N0_5020_TH_VEN

 Click Resume on prompt

 Watch the console windows as test cases execute

Step 5 - To view a detailed transaction log (starting with TraceLog_xxx), you can look in the log

folder off the Eclipse root (you might need to refresh the folder) or point a browser at the logfile.html file in the same folder for a prettier interface (Shortcut: run LogFiles_View in util)

Step 6 - Now let's run the same tests over XMPP Unzip openfire_Windows_xxxxx or

openfire_Linux_xxxxx file into a system folder

Step 7 - Start Openfire from the bin directory On Windows use openfire.exe or

"OpenADR_Start.Bat" if you encounter issues On Linux use"./openfire start"

Optional Step 8 - You can start the Openfire console at 127.0.0.1:9090 from a browser Enter

username as admin and password as password Ignore warning about RSA cert not matching server domain

Step 9 - Modify the selftest properties located in the Eclipse Package Tree at src/properties/

vendor.selftest.openadrconfig.properties as follows: Transport_Type=XMPP

Step 10 - Repeat steps 3, 4, and 5 to run the same registration test case over XMPP.

You have now successfully run test cases across both transports! Suggested next steps:

 Review Appendix B to get a feel for how the test harness is configured

 Skim the balance of this user manual so you know what information is available

 Set up your device, configure the test harness, and start running tests!

OpenADR 2.0b is an application layer message exchange protocol used for two-way

communication of Demand Response (DR), price, and Distributed Energy Resource (DER) signalsbetween the electricity service provider and its customers The technical requirements for the protocol are defined in the OpenADR 2.0b Profile Specification

Devices that implement OpenADR 2.0b must pass a certification test The requirements for the certification test are spelled out in the OpenADR 2.0b Certification Test Specification Each test case defined in the test specification is implemented in the OpenADR 2.0b Certification Test Harness

All OpenADR 2.0b interactions are between a Virtual End Node (VEN) and a Virtual Top Node (VTN) using either a push or pull exchange model The functionality of OpenADR 2.0b is split across 5 services which include EiEvent, EiOpt, EiReport, EiRegisterPart, and OadrPoll When a test is run, the test harness plays the role opposite that of the device under test For instance, when testing a VEN Push implementation, the test harness will play the role of a VTN Push implementation

This document describes how to use the OpenADR 2.0b Certification Test Harness to execute one or more sets of tests, the sum of which represents the OpenADR certification

requirements Test Cases contained in the test harness are organized in a hierarchy as follows:

 src

o testcases

 Exchange Model (push or pull)

 Service (event, report, registerparty, opt, or general)

o Role (VEN or VTN)

 Test case 1

 Test case 2

 Etc

Test cases listed under "DR_PRogram_Guide" are not part of the message exchange

certification and they may be ignored The specific sets of tests required to pass certification are outlined in the OpenADR 2.0b PICS document In order to successfully pass each of the certification tests, the device under test (DUT) must have certain capabilities as outlined in the following sections of the OpenADR 2.0b Test Specification:

 DUT Configuration Requirements

 DUT Implementation Limits

The Test Harness is built on top of the open source Eclipse integrated development

environment The test harness consists of a large body of code, referred to as the Test

Running a test case is as simple as selecting the test case from the Test Harness “Package

Explorer” and clicking the green Run toolbar button, watching the text case execute in the

Console window, and reviewing the test results in a browser The following figure shows the

major functional areas of the Test Harness GUI

Figure 1 – Eclipse IDE

Once a test case has completed execution, a pass/fail indication is shown in the Console window

Detailed transaction logs are recorded for each test execution, and these transaction logs can be used to

This is the test code which the user should not need to alter

During test execution progress indications will appear in the console window

isolate the cause of a test case failure Figure 2 shows the Test Result Summary screen When a tracelog link is clicked, a detailed transaction log will be displayed in a text editor as show in Figure 3.

Figure 2 – Test Result Summary

Figure 3 – Transaction Log

Java

The security test cases require the test harness be run on top of Java 1.7 This can be obtained

at the following site:

http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html

Eclipse IDE for Java Developers

The test harness uses the Eclipse Juno IDE for Java Developers, which can be downloaded from the following site:

http://www.eclipse.org/downloads/packages/eclipse-ide-java-developers/junosr2

Note that the test code is not sensitive to the version of Eclipse that is used as long as it is Juno

or later The narrative in this user’s guide presumes the Juno version of Eclipse is being used

Eclipse Configuration

Two changes need to be made to the Eclipse environment (Fig 4a and Fig 4b) From the Main

Menu select Window, Preferences, Java Expand Compiler, and then select the Errors/Warning option On the right panel under Deprecated API, change the Forbidden reference to Warning Also from Window, Preferences, select General, Workspace, and check Refresh using native

hooks or polling.

Figure 4a – Eclipse Configuration Setting

Figure 4b – Eclipse Configuration Setting

Test Harness installation

The OpenADR 2.0b Test Harness is distributed as an Eclipse project zip file The process for installing the Test Harness in the Eclipse workspace is as follows:

 Extract the project into a temporary directory

 Rename the project folder to something unique, and then copy the project directory into the Eclipse workspace directory Note that the project folder name will become the project name inside Eclipse

 Use the Eclipse File, Import, General, "Existing Project into Workspace" option

 You can then move over any property settings from the previous release and then delete the previous release if desired

It is easier to traverse the test cases when running tests if you use the drop down menu in the Package Explorer to change the package presentation to hierarchical (Fig 5)

Figure 5 – Package Presentation

Openfire Installation

When testing a VEN over XMPP, the test harness VTN configuration includes an XMPP server called Openfire, and this server must also be installed Note that when testing a VTN, the test harness will be communicating with the XMPP server that is part of the device under test

Create a folder on your local system and unzip one of the following Openfire packages to that folder:

 Windows = openfire.zip

 Linux = openfire.tar.gz

Openfire can be started from the Openfire bin directory using openfire.exe (if you encounter issues use "OpenADR_Start.Bat" for Windows) or on Linux with "./openfire start" The WindowsOpenFire, if started with the batch file, will run in a DOS window, as a special startup script was needed to get it to run under Java 1.7 on some systems

Configuration

In order for the test suite to work with a device under test, a number of properties need to be set in the test harness, Openfire server, and the device under test In general the following types of settings need to be configured:

 Information necessary to connect to test harness to the device under test, including IP addresses, port numbers, user names, passwords, and security configuration

 OpenADR specific Information necessary for VENs and VTNs to interoperate, including VEN IDs, market context, available resources, etc.

Appendix A provides detailed information on appropriate settings for various test

combinations Appendix B provides a summary of all configuration settings Appendix C

provides a listing of one of the security profile configuration files, which should normally not need to be changed The following sections focus on how to make the changes, not what the changes themselves should be

Test Harness Configuration Mapping

The test harness configuration properties are located under “src.properties” The configuration

file openadrconfig.properties acts as a pointer to vendor-specific configuration files The

contents of this file looks as follows:

####################################################

# OpenADR Properties

# Test harness configuration properties are read from

# the file point to by the Properties_File key

# listed below

#####################################################

Properties_File= vendor.selftest.openadrconfig.properties

######################################################

To create your own configuration file, copy and paste selftest configuration file

(vendor.selftest.openadrconfig.properties) and make your own edits Each configuration characteristic had a key, such as TH_VEN, and a value, such as "ven_123" The configuration files can be edited from within Eclipse or with any text editor It is helpful to have multiple configuration files for different configurations of transport and security configurations during the testing process

In addition to the properties files noted above, there are 6 security configuration files located in

"src.properties.security" Each of these files has a specific set of settings that represents a particular security profile The security profile used when running a test case is controlled by two settings in the main properties file, explained in more detail below Please note that the security profile files are not normally modified by the user and are not documented beyond an example listing of a representative file in Appendix C

Transport and Security Configuration

By default, the test harness is configured to communicate over http in a localhost environment The following Test Harness property can be used to switch between XMPP and HTTP:

The test harness certificate stores are configured as follows:

 Truststore contains both root and intermediate certs

 Keystores used by the test harness when playing the the role of a VEN or VTN contain the device cert and private key

 The keystore used by the XMPP server contains a complete certificate chain from the root cert through the device cert

VENs being submitted for certification must have host authentication of the x.509 certificate CNfield disabled to avoid needing to reconfigure the test harness server certificates VENs can also avoid the need to reconfigure the test harness XMPP server by using a CN address of

111111111111 for the X.509 client certificate installed in the implementation

VTNs being submitted for certification must have the XMPP Server per-configured for the user name of 111111111111, which the test harness uses to connect to the implementation's VTN XMPP server Note that the user names in the Openfire server MUST match the name in the CN field of the X.509 VEN client certificate

The VTN test certificates used as part of the test harness and Openfire builds have been

generated with a CN name of "vtn" and "127.0.0.1" respectively As long as the device under test has host authentication disabled, these certificates will work correctly in both the test harness and the Openfire server It is beyond the scope of this manual to provide detailed guidance for rebuilding trust and keystore files with new certificates on both the test harness and Openfire server

If you are attempting to test a device remotely, you will need to open your firewall to the various ports used by the test harness, Openfire server, and device under test Most of these ports are listed in the test harness properties file (i.e 8080, 8081, 5222).

OpenFire Configuration

The security configuration of the Openfire server MUST match that of the test harness

XMPP_Security setting In the /bin directory of the Openfire server installation there are 2 batchfiles/shell scripts that allow you to switch between Plain and SHA256 security on the Openfire server The Openfire server must be restarted after changing these settings The batch files are named:

 OpenADR_XMPP_Plain_Security

 OpenADR_XMPP_SHA256_Security

Openfire has an administrative console that can be accessed at 127.0.0.1:9090 with a username

of admin and a password of password From the administrative console, select system

properties (see figure 6)

If you are testing a VEN, the XMPP user name for the DUT VEN must match the CN field of the x.509 certificate for the VEN If the DUT certificate was generated using a CN field of

"111111111111" no action is needed If not, please configure Openfire as show in figure 6a to add a user name matching the CN name of the ven client certificate to Openfire

If you are testing a VTN, the test harness default VEN CN name of "111111111111" must be added as a user to the DUT's XMPP configuration assuming the VTN's XMPP server has the same constraints as OpenFire

If you are testing a VEN, and the test harness, Openfire, and DUT are not all on the same system, you will need to modify the following settings so that the Openfire server IP address is correctly configured in the DUT, Openfire, and Test Harness The following settings will need to

be configured with the new IP base address:

 Openfire system properties (See Figure 6b)

o openadr2.vtn.jid

o xmpp.domain

 Test harness properties

o TH_XMPP_Address or DUT_XMPP_Address (refer to comments in properties for correct configuration)

 VEN DUT

o Set the IP address of the Openfire server

If the test harness, Openfire, and DUT are not all set up pointing at the same IP address for the Openfire server, you will not be able to communicate In particular, you cannot leave the IP address in the openadr2.vtn.jid xmpp.domain Openfire properties set to 127.0.1 if inbound JID addresses are using the actual IP address of the Openfire server

Figure 6b - Openfire System Properties

Running Test Cases

Test Case Naming

Test case numbering uses the following pattern: XY_ZZZZ_TH_WWW_1

X = A letter designating the service or special test type

 E= EiEvent Service Tests

 P=EiOpt Service Tests

 R=EiReport Service Tests

 N=EiRegisterParty Tests

 G=General Tests (includes non-service specific test such as security)

Y = A digit indicating the type of implementation:

 1= Pull Test

 0 = Push Test

ZZZZ = A numeric sequence number for the test case with initial increments of 10 For a given Push or Pull test scenario, each VEN test case must have a matching VTN Test casewith the same 4 digit value

TH_WWW = The "TH" indicates Test Harness and the WWW indicates the role, either

"VEN" or "VTN" This helps the user keep track of the role that the test harness is playing

Note that test cases migrated from the A profile test harness will be prefixed with an

"A_"

Test Logs

Test execution can be observed in the Eclipse Console window, and you will see a pass or fail

indication at the end of test case execution However, detailed transactions records and failure information can be found in a log file generated for each test case in the log directory (in the

"log" folder off the Eclipse project root)

Two test cases under “erc/testcases/util” (LogFiles_Clear and LogFiles_View) will allow you to

log After running a test case, you will need to refresh the browser to see additional test results.You can also click on the logfile.html in the log directory and to open the file with your default browser.

The security tests (Gx_4005, 4007, 4010, 9010) enable the Java network debug output in the console window This debug output is not captured in the test logs, so you may wish to cut and paste these results from the console window into a text file if preserving the debug TLS

exchange details is important The OpenADR application layer exchange that occurs as part of the security tests is captured in the test logs

It should be noted that when the test harness is playing the role of the VTN, the network debug output shows the negotiation over the private connection between the test harness and the Openfire server You will need to use Wireshark to capture the communication between the Openfire server and the VEN DUT if necessary for debugging purposes

Self Test Routines

The B profile test definitions result in set two sets of mirrored test routines, one set each for push and pull exchange models, and within each set a VEN and VTN test routine The result is that each test case definition will result in 4 separate test routines, one each for the following entities: VEN Push, VTN Push, VEN Pull, VTN Pull These mirrored test cases can be run against their counterpart device under test or against each other in a self test scenario

When running test cases against each other, you should always start the test case with the "_1"prefix first Also note that when running self-test routines it is not necessary to configure events

or to wait for specific time periods as requested by the prompts

The B Profile test suite contains a large number of EiEvent service test cases that were ported over from the A Profile test harness These ported test cases contain a set of self test routines that allow you to run test cases without the need for a real device to be attached to the test harness However, these are not implemented as mirrored test cases as noted above, as they are stand-alone pieces of code that can be used to test a particular ported test case These ported A profile self test routines are located as follows:

 testcase.pull.event.selftest_a_ported

 testcase.push.event.selftest_a_ported

These self test simulations have DUT in the test case name Note that the DUT self-test routines

do not simulate time-based state changes, so you can typically just click through prompts on the observation test cases when using these self-test routines Ported A profile test cases can

be identified with the "A_" prefix

To run a self test code against a ported A test case, simply start the test case or the self test routine that has the _1 postfix, then the start the test representing the other side of the

conversation