OpenADR 2.0a Certification Test Harness User Manual

Overview OpenADR 2.0a is an application layer message exchange protocol used for twoway communication of Demand Response (DR), price, and Distributed Energy Resource (DER) signals between the electricity service provider and their customers. The technical requirements for the protocol are defined in the OpenADR 2.0 Profile Specification. Devices that implement OpenADR 2.0a must pass a certification test. The requirements for the certification test are spelled out in the OpenADR 2.0a Certification Test Specification. Each test case defined in the test specification is implemented in the OpenADR 2.0a Certification Test Harness. This document describes how to use the OpenADR 2.0a Certification Test Harness to execute a suite of tests used to certify OpenADR implementations. There are four test suites supported by the Test Harness: • VEN Push Test Suite • VEN Pull Test Suite • VTN Push Test Suite • VTN Pull Test Suite All OpenADR 2.0a interactions are between a Virtual End Node (VEN) and a Virtual Top Node (VTN). When a test is run, the test harness plays the role opposite that of the device under test. For instance, when testing a VEN Push implementation, the test harness will play the role of a VTN Push implementation. In order to successfully pass each of the certification tests, the device under test must have certain capabilities as outlined in the following sections of the OpenADR 2.0a Test Specification: • DUT Configuration Requirements • VTN Test Interface • DUT Implementation Limits The Test Harness is built on top of the open source Eclipse integrated development environment. The test harness consists of a large body of code referred to as the Test Framework that implements the OpenADR 2.0 protocols, and a series of small test case programs that implement the test casespecific logic. Running a test case is as simple as selecting the test case from the Test Harness “Package Explorer” and clicking the green Run toolbar button, watching the text case execute in the Console window, and reviewing the test results in a browser. The following figure shows the major functional areas of the Test Harness GUI. ...

OpenADR 2.0a

Certification Test Harness

User Manual

Version 1.1.1

1.0.4  Update to reflect changes from Test Event 08/28/12 - JZ1.0.5  Update to reflect the NetworkFX security changes 02/26/14 - BD

Table of Contents

Overview 5

Installation 8

Java 8

Eclipse IDE For Java Developers 8

Eclipse Configuration 9

Test Harness 10

Test Cases 11

Test Case Naming 11

Test Logs 11

Self Test Routines 11

Test Case Setup 13

Optional Test Cases 13

Security Configuration 14

Helpful Hints 15

Active Event Indicator 15

Set polling to 10 seconds 15

Time Synchronization 15

Errors in the Log Window 15

Failure information in test reports 15

Processes Left Running 15

Clearing the event data store 15

Conformance Rule Disable 15

Self Tests 15

Clean Project 16

Refresh Project 16

Firewalls 16

Wireshark 16

Disabling TH Host Verification 16

Disabling DUT Host Verification 16

Appendix A – Configuration Tables 17

Pull VEN Configuration 17

Push VEN Configuration 18

Pull VTN Configuration 19

Push VTN Configuration 20

Appendix B – Configuration Properties 21

asyncResponseTimeout 21

createdEventAsynchTimeout 21

DR_MarketContext_1_Name, DR_MarketContext_2_Name 21

Global Conformance Rule Control 21

GroupID 21

logFileNam 21

Login_Name 21

Login_Password 21

logPath 21

Party_ID 21

Report_Header_Info_1, 2, and 3 21

ResourceID 21

SchemaFile 21

Security_Debug 21

HTTP_Security 22

Test Case Specific Configuration Keys 22

testCaseBufferTimeout 22

TestEventString 22

VEN_ID 22

VTN_ID 22

VEN_ID_2 22

VTN_ID_2 22

VenURL 22

VenURL_Port 22

VenURL_Resource 22

VtnURL 22

VtnURL_Port 22

VtnURL_Resource 23

Appendix D – Updated Tests 26

OpenADR 2.0a is an application layer message exchange protocol used for two-way

communication of Demand Response (DR), price, and Distributed Energy Resource (DER) signals between the electricity service provider and their customers The technical requirements for the protocol are defined in the OpenADR 2.0 Profile Specification

Devices that implement OpenADR 2.0a must pass a certification test The requirements for the certification test are spelled out in the OpenADR 2.0a Certification Test Specification Each test case defined in the test specification is implemented in the OpenADR 2.0a Certification Test Harness.

This document describes how to use the OpenADR 2.0a Certification Test Harness to execute a suite of tests used to certify OpenADR implementations There are four test suites supported by the Test Harness:

 VEN Push Test Suite

 VEN Pull Test Suite

 VTN Push Test Suite

 VTN Pull Test Suite

All OpenADR 2.0a interactions are between a Virtual End Node (VEN) and a Virtual Top Node (VTN) When a test is run, the test harness plays the role opposite that of the device under test For instance, when testing a VEN Push implementation, the test harness will play the role of a VTN Push implementation.

In order to successfully pass each of the certification tests, the device under test must have certain capabilities as outlined in the following sections of the OpenADR 2.0a Test Specification:

 DUT Configuration Requirements

 VTN Test Interface

 DUT Implementation Limits

The Test Harness is built on top of the open source Eclipse integrated development

environment The test harness consists of a large body of code referred to as the Test

Framework that implements the OpenADR 2.0 protocols, and a series of small test case

programs that implement the test case-specific logic

Running a test case is as simple as selecting the test case from the Test Harness “Package

Explorer” and clicking the green Run toolbar button, watching the text case execute in the

Console window, and reviewing the test results in a browser

The following figure shows the major functional areas of the Test Harness GUI.

Figure 1 – Eclipse IDE

Once a test case has completed execution, a pass/fail indication is shown in the Console window

Detailed transaction logs are recorded for each test execution, and these transaction logs can be used to

isolate the cause of a test case failure Figure 2 shows the Test Result Summary screen When a tracelog

link is clicked, a detailed transaction log will be displayed in a text editor

This is the test code which the user should not need to alter

During test execution progress indications will appear in the console window

Figure 2 – Test Result Summary

Figure 3 – Transaction Log

Eclipse IDE For Java Developers

The test harness uses the Eclipse Indigo IDE, which can be downloaded from the following site:http://www.eclipse.org/downloads/packages/release/indigo/sr1

Note that the test code is not sensitive to the version of Eclipse that is used as long as it is Juno

or later The narrative in this user’s guide presumes the Juno version of Eclipse is being used

Eclipse Configuration

Two changes needs to be made to the Eclipse environment (Fig 4a and Fig 4b) From the Main

Menu select Window, Preferences, Java Expand Compiler, and then select the Errors/Warning option On the right panel under Deprecated API, change the Forbidden reference to Warning Also from Window, Preferences, select General, Workspace, and check Refresh using native

hooks or polling.

Figure 4a – Eclipse Configuration Setting

Figure 4b – Eclipse Configuration Setting

Test Harness

The OpenADR 2.0a test Harness is distributed as an Eclipse project zip file The process for installing the Test Harness in the Eclipse workspace is as follows:

 Extract the project into a temporary directory

 Rename the project folder to something unique, and then copy the project directory into the Eclipse workspace directory Note that the project folder name will become the project name inside Eclipse.

 Use the Eclipse File, Import, General, "Existing Project into Workspace" option

 You can then move over any property settings from the previous release and then delete the previous release if desired.

The test cases are located under “src” in the following areas:

be ignored

Figure 5 – Package Presentation

Test Cases

Test Case Naming

Test cases names are structured as xx_xxxx_yy_zzz, such as E1_0010_TH_DUT

 xx_xxxx is the test name used in the test specification.

 yy can be “TH” for Test Harness or “DUT” for a device under test (used for self test)

 zzz is the role that the is being played by the test code (VEN or VTN).

When testing a real device, only the TH test cases will be used For instance, to test a real pull VEN you would run the test case E1_0010_TH_VTN In this case the test harness would be playing the role of the VTN.

Test Logs

Test execution can be observed in the Eclipse Console window, and you will see a pass or fail

indication at the end of test case execution However, detailed transactions records and failure information can be found in a log file generated for each test case in the log directory

(configured in Properties)

Two test cases under “util” will allow you to clear the log directory and to launch a browser to see the log file summary, which will show all log files with a pass/fail status Click on the link to see the detailed log After running a test case, you will need to refresh the browser to see additional test results You can also click on the logfile.html in the log directory and to open the file with your default browser

The security tests enable the Java network debug output in the console window This debug output is not captured in the test logs, so users may wish to cut and paste these results from the console window into a text file if preserving the debug TLS exchange details is important Note that the OpenADR exchange that occurs as part of the security tests is captured in the test logs.

Self Test Routines

During development of these test cases we did not have access to real devices, so we wrote simple self-test simulations of the device under test These simulations have DUT in the test case name Note that the DUT self-test routines do not simulate time-based state changes, so you can typically just click through prompts on the observation test cases when using these self-test routines The self-test routines are located as follows:

 testcases.ven.pull.selftest

 testcases.ven.push.selftest

 testcases.vtn.pull.selftest

 testcases.vtn.push.selftest

A typical execution sequence for a Pull VEN test case (the DUT is the VEN) using the self tests would be as follows:

 Run E1_0030_TH_VTN

 Click Resume on the prompt

 Run E1_0030_DUT_VEN in the selftest package (to simulate the device under test)

 Watch Console window You will see a pass/fail indication at the completion of the test

 View the log files with your default browser

When using the self-test routines or real devices, interaction with the device under test must be done in the correct sequence In general, you must make sure that the server receiving the communication is running before causing the client to send events, as shown in the table below:

VEN Pull  Start test case

 Follow directions on prompt and click resume

 If running self test, start self-test routine

 If testing real device, make sure its polling feature is enabled

VEN Push  If running self test, start self-test routine

 If testing real device, make sure its push functionality is enabled

 Start test case

 Follow directions on prompt and click resume

VTN Pull  If running self test, start self-test routine

 If testing real device, make sure device is enabled

 Start test case

 Follow directions on prompt and click resume

VTN Push  Start test case

 Follow directions on prompt and click resume

 If running self test, start self-test routine

Note that when running self-test routines it is not necessary to configure events or to wait for specific time periods as requested by the prompts.

Test Case Setup

In order for the test suite to work with a device under test, a number of properties need to be set in the test suite The Configuration properties are located under “src.properties” The configuration file

openadrconfig.properties acts as a pointer to vendor-specific configuration files The contents of this

file look as follows:

####################################################

# OpenADR Properties

# Test harness configuration properties are read from

# the file point to by the Properties_File key

# listed below

#####################################################

Properties_File=vendor.isy.openadrconfig.properties

######################################################

To create your own configuration file, copy and paste selftest configuration file

(vendor.selftest.openadrconfig.properties) and make your own edits The most important item to set up in your vendor-specific properties is the test report directory Modify the following property to point to thedirectory as shown in the example below:

logPath=C:\\OpenADR_Test_Logs\\

Each time a test case is run, a text file with a time stamp is written to this directory and the file

Logfile.xml and logfile.html is updated

Appendix A provides detailed information on how to set up both the DUT and the test harness for testing Appendix B provides a summary of all configuration settings Appendix C shows a sample of the HTTP_SHA256_Security.properties

Optional Test Cases

Some test cases are dependent on the ability to configure specific values on the Device Under Test If these values are not configurable, the OpenADR 2.0a PICS document provides a list of questions that determine which optional test cases can be skipped This can be found under the heading Optional Test Case Guidelines

The test harness certificate stores are configured as follows:

 Truststore contains both root and intermediate certs

 Keystores used by the test harness when playing the the role of a VEN or VTN contain the device cert and private key

VENs being submitted for certification must have host authentication of the x.509 certificate CN field disabled to avoid needing to reconfigure the test harness server certificates The VTN test certificates used as part of the test harness have been generated with an CN name of "vtn" As long as the device under test has host authentication disabled, these certificates will work correctly in the test harness It isbeyond the scope of this manual to provide detailed guidance for rebuilding trust and keystore files withnew certificates on the test harness

Please review the following helpful hints in the following section prior to doing security testing:

Disabling Host Verification

Helpful Hints

Please note the following:

Active Event Indicator – A number of test cases will ask the test operator to observe the

transition of an event to Active using the Active Event indicator However, if you have access to a DUT

Control Panel that shows the start and end of an event, as well as the eventStatus, it may not be

necessary to actually wait for an event to end if you can see from the DUT console that the event has been scheduled to end at a specific time Judgment must be exercised!

Set polling to 10 seconds – For Pull VENs, set the polling interval to around 10 seconds The test

cases deal in events that are just minutes long, so a longer poll time may result in false failures

Time Synchronization – While all the payload validation is based on the createdDateTime in the

payload, some false failures may occur if there is a delta between the time on the Device Under Test andthe time on the PC executing the test suite Note that there is a built-in conformance rule check that validates that the DUT is reasonably synced with the test harness, which should avoid most issues.

Errors in the Log Window – We shut down the test suite http server after each test case

execution During the shutdown process, an async payload from the DUT may generate an error in the

Console window However, always check the test report in the log directory, as typically you will see that the test case did run successfully even though an error appears at the end of the Console window

trace

Failure information in test reports – The Console window just shows status and an overall

pass/fail status For detailed conformance errors you will need to scan through the test report log file forspecific errors that caused the test to fail

Processes Left Running – If you start and then abort a test case, you may leave processes running If you don’t kill these processes in the Eclipse Debug perspective or allow them to time out,

subsequent test cases may not run correctly

Clearing the event data store – Generally speaking, implicit cancellation when testing VENs

takes care of flushing the DUT at the start of each test case However, it is critical on VTNs that the eventdata store be cleared before the next test VTN Pull test cases will prompt you to clear the data store at the start of each test case The VTN Push test cases will prompt at the end of each test case and also wait 10 seconds for any retries to occur before ending the test case

Conformance Rule Disable – Conformance rules can be disabled selectively by inserting code at

the top of the test() method in test cases using the following format:

ConformanceRuleValidator.setDisableCreatedEvent_ResponseCodeSuccess_Check(true);