OpenADR 2.0b Certification Test Specification

Introduction This purpose of this document is to define a set of test cases that systems implementing the OpenADR 2.0b profile must successfully pass in order to obtain certification. Scope OpenADR 2.0b is an application layer message exchange specification used for twoway communication of Demand Response (DR) , price, and Distributed Energy Resource (DER) signals between the electricity service provider and its customers. OpenADR 2.0b is defined in the following documents: • The OpenADR 2.0b Profile Specification, Revision 1.1 • The OpenADR 2.0b schema The conformance requirements that are not defined by the schema are contained in a set of conformance rules in the OpenADR 2.0b Profile Specification. These rules are referenced in both the OpenADR 2.0 Protocol Implementation Conformance Statement (PICS) and this document. Appendix F provides a mapping between each of these conformance rules and the test cases defined in the test specification. The other narrative sections of the OpenADR B Profile specification are considered informational and the Test Harness assumes that all the testable requirements have been captured in the conformance rules.

Trang 1

OpenADR 2.0b

Certification Test Specification

Version 1.1.0

1

Trang 2

1.0.3  Revised test case N1_0080 and N0_5080 to expect no

response or 463 error in oadrResponse.

 Modified the scope of testing to use test one randomly selected test case and all security test cases while testing multiple transports.

08/12/13 - EP

 S0-303 Diagram has been updated.

08/20/13 - EP

 Displaying Push sequence in Diagram S1-812

 Updated Diagram S0-311 and S1-811 to use oadRegisterReport.

 Changed to VTN Ignore oadrCreatedEvent.

 Modified N0_5080 to also accept appropriate responses.

 Test case A_E3_0680_TH_VEN has been modified to use oadrRequest and not oadrPoll.

 The following test cases have been made optional

o R1_3050_TH_VEN, R0_8050_TH_VEN_1,

R1_3060_TH_VEN, R0_8060_TH_VEN, R1_3120_TH_VTN_1, R0_8120_TH_VTN, R1_3070_TH_VTN_1, R0_8070_TH_VTN_1, R1_3080_TH_VTN_1, R0_8080_TH_VTN_1, R1_3090_TH_VTN_1,R0_8090_TH_VTN_1, R1_3100_TH_VTN_1, R0_8100_TH_VTN_1.

 Corrected typo in N1_0015, N0_5015, N1_0070 and N0_5070.

 Added pulseCount to Metadata_001.

09/06/13 - BD, EP

 Updated prompt in test case A_E3_0655_TH_VEN, E1_1040_TH_VEN, E0_6040_TH_VEN.

 Removed Test Case A_E0_0295_TH_VTN and

2/11/2014 – EP, BD

Trang 3

 Modified E2_0530/E3_0530 prompt.

 Expected result for Test case R1_3130_ and R0_8130_ has been modified Also Prompt_043 has been modified for the same.

 Scenario for test case A_E2_0527 has been modified.

 Added test case E1_1055 & E0_6055 to test Multiple Signals in an Event.

 Split G0_9010 into G0_9005 for RSA and G0_9010 for ECC.

number of intervals.

 Removed all references to startBefore

 Mantis 139: Added adjacent event tests case E1_1090 and E0_6090

 Mantis 144: Test G1_4012 added using event followed by poll for empty queue.

 Mantis 154: Expected result for Test case R1_3130_ and R0_8130_ has been modified.Mantis 157: Conformance rule check modified to flag use of oadrPoll in http push.

 Mantis 163: Added test cases G1_4030 and G0_9030 (both TH_VEN and TH_VTN)

 Mantis 164: Deleted Test Case A_E0_0325/A_E1_0325 - MarketContext can have a wild card, so can't do negative test

 Mantis 165: Modified test case E1_1065 and E0_6065 (both TH_VEN and TH_VTN)

 Mantis 166: Updated conformance rule check to verify dateTime is the same for all events in a

 Mantis 186: Added tests for expired certificates:

G1_4007_TH_VTN and G1_4007_TH_VEN

3

Trang 4

 Mantis 190: Updated tests N1_0020 and N0_5020 to expect oadrRequestEvent to be sent following the exchange of metadata reports during registration

 Revised tests A_E1_0285, A_E0_0285, A_E2_0510, A_E3_0510 and deleted tests A_E2_0520, A_E3_0520, A_E2_530, and A_E3_0530 as the result of changes to rule regarding overlapping events

1.1.0  Added support for wildcard deviceClass enumerations in

conformance rule check.

 Added explicit signalType check for “setpoint” in test cases E1_1025_TH_VEN and E0_6025_THV_VEN

 Added checks to N1_0020_TH_VTN and N0_5020_TH_VTN

to verify that the reports required by the conformance rules are returned by the VEN.

2/22/16 - JZ

Trang 5

Revisions: 2

Introduction 10

Terms and Acronyms 10

Scope 10

Test Methodology 11

Test Case Organization 12

Test Case Numbering 12

B Profile Test Case Template 14

Ported A Profile Test Cases 15

DUT Configuration Requirements 17

DUT Implementation Limits 19

Items with Limited Testing 19

Common Pass and Fail Verdicts 20

Test Case Scenarios 21

EiRegisterParty Service Test Scenarios 21

VEN Registration - Query 21

VEN Registration - Query While Registered 22

VEN Registration - Bootstrap Sequence 23

VEN Registration - Pre-allocated VEN ID 24

VEN Registration - Cancel Registration 25

VTN Registration - Cancel Registration 26

VTN Registration - Request Re-Registration 27

VEN Registration - New Registration while Registered 28

VEN Registration - Re-Registration 29

VTN_VEN Registration - Negative Test Scenarios 30

VEN Registration - Payload While Unregistered 31

EiOpt Service Test Scenarios 32

VEN Opt - New Opt Schedule 32

VEN Opt - Open Ended Opt Schedule 33

5

Trang 6

VEN Opt - Cancel Opt Schedule 34

VEN Opt - Multiple Opt Schedules, Targeting 35

VEN Opt - Multiple Opt Scheduled with Cancel 36

VEN Opt - Negative Test Scenario 37

EiReport Service Test Scenarios 38

VEN Reporting - One Shot Report 38

VEN Reporting - Periodic Report 39

VEN Reporting - Delayed Periodic Report, loadControlState 40

VEN Reporting - Short duration 41

VEN Reporting - Multiple Report Request Payloads 42

VEN Reporting - Cancel Report 43

VEN Reporting - Cancel Open Ended Report, reportToFollow 44

VEN Reporting - Piggy Back Cancellation (Optional VEN) 45

VEN Reporting - Piggy Back Request (Optional VEN) 46

VTN Reporting - One Shot Report (Optional VTN) 47

VTN Reporting - Periodic Report (Optional VTN) 48

VTN Reporting - Multiple Report Request Payloads (Optional VTN) 49

VTN Reporting - Cancel Report (Optional VTN) 50

VTN Reporting - Piggy Back Request (Optional VTN) 51

VEN Reporting - History Usage Report 52

VEN Reporting - History Usage Report Subset 53

VEN Reporting - Telemetry Usage, One Shot 54

VEN Reporting - Telemetry Usage, Periodic 55

VEN Reporting - Metadata Report Request 56

VEN Reporting - Same Payload, Multiple Requests and Updates 57

VEN Reporting - Negative Test Scenarios 58

EiEvent Service Test Scenarios 59

VTN Event - Empty 59

VTN Event - Electricity Price Event 60

VTN Event - Load Dispatch Event 61

Trang 7

VTN Event - x Created Events, Mixed Simple/Complex 64

VEN Event - Use of oadrRequestEvent 65

VEN Event – Multiple Signals in an Event 66

VEN CreateOpt Qualification 67

VTN Ignore oadrCreatedEvent 68

VTN Event -Negative Test, Custom Event Signal 69

VTN/VEN Event -Negative Test, Case Sensitivity 70

Adjacent Event Execution 71

EiEvent Ported A Profile Test Cases 72

VEN Push/Pull Test Scenarios 72

VEN Push/Pull Negative Test Scenarios 96

VTN Push/Pull Positive Test Scenarios 104

VTN Push/Pull Negative Test Scenarios 120

General Test Scenarios 125

Transport and Security Test Overview 125

Security - Cipher and X.509 Cert support - RSA 126

Security - Cipher and X.509 Expired Certificate 127

Security - Cipher and X.509 Cert support - ECC 128

VEN - Empty oadrPoll 129

VEN – Event followed by oadrPoll 130

VEN - De-Queue using oadrPoll 131

VTN -Concurrent A and B operation 132

Omitted vs Empty Element Equivalent 133

Appendix A - Test Scenario User Prompts 134

Ported A Test Case Prompts 138

Appendix B - Test Scenario Events 143

Appendix C - Test Scenario Opt Schedules 144

Appendix D - Test Scenario Reports 145

Report Metadata 145

Report Requests 146

Report Updates 148

Appendix E - Conformance Rules Checks 149

7

Trang 8

Appendix F – Conformance Rule Validation Mapping 155

Appendix G - Use Case Scenarios 162

Diagram: S0-000 VEN Registration - Query 162

Diagram: S0-001 VEN Registration - Bootstrap Sequence 163

Diagram: S0-002 VEN Registration - Cancel Registration 164

Diagram: S0-003 VTN Registration - Cancel Registration 165

Diagram: S0-004 VTN Registration - Request Re-Registration 166

Diagram: S0-100 VTN Event - Empty 167

Diagram: S0-101 VTN Event - Send Event 168

Diagram: S0-102 VTN Event - Two Dist Event Sequences 169

Diagram: S0-103 VTN Event - x Created Events 170

Diagram: S0-104 VEN Event - Use of oadrRequestEvent 171

Diagram: S0-105 VEN CreateOpt Qualification 172

Diagram: S0-106 VTN Ignore Created Event 173

Diagram: S0-200 VEN Opt - New Opt Schedule 174

Diagram: S0-201 VEN Opt - Cancel Opt Schedule 175

Diagram: S0-202 VEN Opt - Mutliple Opt Schedules 176

Diagram: S0-203 VEN Opt - Mutliple Opt Scheduled with Cancel 177

Diagram: S0-300 VEN Reporting - One Shot Report 178

Diagram: S0-301 VEN Reporting - Periodic Report 179

Diagram: S0-302 VEN Reporting - Multiple Report Request Payloads 180

Diagram: S0-303 VEN Reporting - Cancel Report 181

Diagram: S0-304 VEN Reporting - piggy back Cancellation 182

Diagram: S0-305 VEN Reporting - piggy back request 183

Diagram: S0-306 VTN Reporting - One Shot Report 184

Diagram: S0-307 VTN Reporting - Periodic Report 185

Diagram: S0-308 VTN Reporting - Multiple Report Request Payloads 186

Diagram: S0-309 VTN Reporting - Cancel Report 187

Diagram: S0-310 VTN Reporting - piggy back Cancellation 188

Diagram: S0-311 VTN Reporting - piggy back request 189

Trang 9

Diagram: S0-400 VEN - Empty oadrPoll 191

Diagram: S1-500 VEN Registration - Query 192

Diagram: S1-501 VEN Registration - Bootstrap Sequence 193

Diagram: S1-502 VEN Registration - Cancel Registration 194

Diagram: S1-503 VTN Registration - Cancel Registration 195

Diagram: S1-504 VTN Registration - Request Re-Registration 196

Diagram: S1-600 VTN Event - Empty 197

Diagram: S1-601 VTN Event - Send Event 198

Diagram: S1-602 VTN Event - Two Dist Event Sequences 199

Diagram: S1-603 VTN Event - x Created Events 200

Diagram: S1-604 VEN Event - Use of oadrRequestEvent 201

Diagram: S1-605 VEN CreateOpt Qualification 202

Diagram: S1-606 VTN Ignore Created Event 203

Diagram: S1-700 VEN Opt - New Opt Schedule 204

Diagram: S1-701 VEN Opt - Cancel Opt Schedule 205

Diagram: S1-702 VEN Opt - Mutliple Opt Schedules 206

Diagram: S1-703 VEN Opt - Mutliple Opt Scheduled with Cancel 207

Diagram: S1-800 VEN Reporting - One Shot Report 208

Diagram: S1-801 VEN Reporting - Periodic Report 209

Diagram: S1-802 VEN Reporting - Multiple Report Request Payloads 210

Diagram: S1-803 VEN Reporting - Cancel Report 211

Diagram: S1-804 VEN Reporrting - piggy back Cancellation 212

Diagram: S1-805 VEN Reporting - piggy back request 213

Diagram: S1-806 VTN Reporting -One Shot Report 214

Diagram: S1-807 VTN Reporting - Periodic Report 215

Diagram: S1-808 VTN Reporting - Multiple Report Request Payloads 216

Diagram: S1-809 VTN Reporting - Cancel Report 217

Diagram: S1-810 VTN Reporting - piggy back Cancellation 218

Diagram: S1-811 VTN Reporting - piggy back request 219

Diagram: S1-812 VTN Reporting - Metadata Report 220

Diagram: S1-813 VEN Reporting - Metadata Report 221

9

Trang 10

This purpose of this document is to define a set of test cases that systems implementing the OpenADR 2.0b profile must successfully pass in order to obtain certification.

Terms and Acronyms

The following terms and abbreviations are used in this document:

Term Description

DUT Device Under Test – This is used to refer to the software

implementation of OpenADR and any supporting hardware necessary to execute the software DUT_VEN Device Under Test Virtual End Node (VEN)

DUT_VTN Device Under Test Virtual Top Node (VTN)

TH Test Harness - A software implementation of OpenADR

playing the role of a VEN or VTN for the purpose of running test cases

TH_VEN Test harness playing the role of the VEN

TH_VTN Test harness playing the role of the VTN

Scope

OpenADR 2.0b is an application layer message exchange specification used for two-way communication

of Demand Response (DR) , price, and Distributed Energy Resource (DER) signals between the electricity service provider and its customers OpenADR 2.0b is defined in the following documents:

 The OpenADR 2.0b Profile Specification, Revision 1.1

 The OpenADR 2.0b schema

The conformance requirements that are not defined by the schema are contained in a set of

conformance rules in the OpenADR 2.0b Profile Specification These rules are referenced in both the OpenADR 2.0 Protocol Implementation Conformance Statement (PICS) and this document Appendix F provides a mapping between each of these conformance rules and the test cases defined in the test specification The other narrative sections of the OpenADR B Profile specification are considered

informational and the Test Harness assumes that all the testable requirements have been captured in

Trang 11

The scope of the test cases will focus on validating the successful exchange of messages between VTNs and VENs, validating that any exceptions are handled gracefully and without service interruptions and that security mechanism are properly implemented Validating the functional behavior of the DUT in response to payload content is out of scope for the certification tests For instance, the test cases will test that a VEN acknowledges the receipt of a DR Event, but it will not validate that the VEN actually sheds load as a result of this message.

For product certification, the certification test suite will be will be run over top of the specific transport protocols supported by the DUT If multiple transports are supported by the DUT, the full conformance test will be run over one transport and a representative set of test cases will be run over the remaining transports The representative set shall consist of one randomly selected test case from each service plus the one security test case Conformance testing of the supported transports is out of scope for the certification tests However, limited testing will be done to validate OpenADR specific requirements for implementation of the transports, such as the inclusion of specific http headers.

The certification test suite will be run utilizing each of the security mechanisms supported by the DUT Conformance testing of the security mechanisms is out of scope for the certification tests However, limited testing will be done to validate OpenADR specific configuration requirements for

implementation of the security mechanisms, such as use of x509 certificates, TLS versions, and cipher suites.

Exception handling tests will be limited to a small set of representative real-world scenarios The intent

of these test cases will not be to exhaustively test all possible exception scenarios, but to validate that the DUT has sound exception handing capabilities For instance, a test might validate that the DUT triggers an error if it is sent a payload with an unknown VEN or VTN ID.

Boundary and limit testing will be limited and will utilize values that are deemed to be practical from a test execution perspective and typical from an implementation perspective, as specific implementation limits have not been defined by the OpenADR 2.0b Profile Specification For instance, testing that includes a responseDescription string would use, at most, a few hundred characters, not tens of

11

Trang 12

 The message interaction patterns are as expected This includes…

o Correct response payloads For instance, receiving an oadrCreatedEvent payload in response to an oadrPoll would not be correct.

o Appropriate request payloads For instance a request containing oadrCreatedOpt payload would not be correct.

 That payloads contain well-formed XML

 That payloads conform to the OpenADR 2.0b schema

 That the specific conformance rules defined in the OpenADR Profile Specification are followed.

 That the intent of the test case is met The test case may expect the VEN to send an optType of

“optOut” and if this is not received, the test case will fail.

Note that some of the OpenADR conformance rules can be validated by simply analyzing the payloads of all message exchanges that occur during execution of the certification test suites These rules are summarized in Appendix F Other conformance rules will require specific test scenarios to validate.

A few test cases will require the test engineer executing the test to observe the behavior of the device under test in order to determine the pass fail status.

Test Case Organization

The test suite is organized as follows:

Trang 13

X = A letter designating the service or special test type

 E= EiEvent Service Tests

 P=EiOpt Service Tests

 R=EiReport Service Tests

 N=EiRegisterParty Tests

 G=General Tests (includes non-service specific test such as security)

Y = A digit indicating the type of implementation:

TH_WWW = The "TH" indicates Test Harness and the "WWW" indicates the role, either "VEN" or

"VTN" This helps the user keep track of the role that the test harness is playing in the exchange _1 = An optional postfix to indicate that the test case should be started before the device under test.

So, an example test case would be R1_1223_TH_VEN_1 This would be an EiReport Pull Test case where the test case is playing role of the VEN and this test case should be started first Test case definitions reference sequence diagrams in Appendix G These sequence diagrams will follow a similar syntax staring with an S, followed by a push/pull indicator, a dash, then a 3 digit number Example: S0-123

13

Trang 14

B Profile Test Case Template

Each test case definition results in set two sets of mirrored test routines; one set each for push and pull exchange models, and within each set a VEN and VTN test routine The result is that each test case definition will result in 4 separate test routines, one each for the following entities: VEN Push, VTN Push, VEN Pull, VTN Pull.

Test cases can be run against their counterpart DUT or against each other in a self test scenario Each test case is based on a use case scenario defined by a sequence diagram in Appendix G Unless

otherwise noted, the test case will use the exact sequence of payload exchanges defined in the use case sequence diagram The use cases also provide guidance as to the location in the sequence for user prompts.

Test Case Name

Objective A sentence indicating the goal of the test case with

respect to the DUT VTN

A sentence indicating the goal of the test case withrespect to the DUT VEN

Prerequisites Any prerequisite settings or conditions Any prerequisite settings or conditions

Scenario Details Payload-specific data for the test scenario and any

necessary prompts

Payload-specific data for the test scenario and any necessary prompts

Expected Results The expected behavior of the DUT VTN The expected behavior of the DUT VEN

Note that test case definitions will reference a set of standardized resources defined in the appendices

as follows:

 Appendix A - Standard prompts referenced in the test cases as Prompt_xxx

 Appendix B - Standard events referenced in the test cases as Event_xxx

 Appendix C - Standard opt schedules referenced in the test cases as OptSchedule_xxx

 Appendix D - Standard reports referenced as Metadata_xxx, Request_xxx, Update_xxx

Trang 15

Ported A Profile Test Cases

Test cases ported from the A Profile Test Harness will retain the same test case definition format as used

in the A profile test specification as well as the same numbering, with the addition of an "A_" prefix These test cases are functionally identical to the A profile tests with the exception that the pull tests use oadrPoll rather than oadrRequestEvent Also note that while new B profile tests can be run against their matching counterparts (VEN or VTN) , the A profile test cases use a self test routine which will be stored

in a package named "selftest_a_ported" Test case definition will use the following format:

Test Case Name (Ported A Test)

TH_VTNPrerequisites - DUT_VEN has no pending or active events

TH_VTN: oadrDistributeEvent – Dispatch Event{DUT_VEN: Transport Acknowledgement }DUT_VEN: oadrCreatedEvent

TH_VTN: oadrResponse , successExpected

oadrDistributeEvent:

 Mandatory elements unless otherwise specified in the test description

 marketContext as configured in the test system

 oadrResponseRequired set to always

 eiTarget with no sub-elements

 dtstart set to currentTime +2 minutes

 x-eiNotification set to a duration of 1 minute

 signalID set to a placeholder value.

 Overall duration for CPP Event set to 3 minutes

 Interval durations for CPP Event set to 1 minute

 Interval signalPayload set to a repeating sequence of 1, 2, 3, 2, 1

 eiResponse element with mandatory elements included if responding to an oadrPoll

 requestID = unique value

15

Trang 16

 Three types of oadrDistributeEvent(s) will be used in these test scenarios:

o Critical Peak Pricing (CPP) – 3 intervals

o 24_Prices – 24 one hour intervals, notification 2 hours

o Dispatch – 1 interval with a duration of 0 and notification time of 0

oadrCreatedEvent:

 Mandatory elements only unless otherwise specified in the test description

 Respond with a oadrCreatedEvent to all oadrDistributeEvent payloads unless the test case description states differently with regards to oadrResponseRequired

 requestID set as an empty element

 Mandatory elements only

Trang 17

DUT Configuration Requirements

In order for many test cases to run successfully, the DUT must be capable of being configured to initiate action or respond to requests For instance, there should be a way to cause a DUT_VEN to respond to an event with an optOut The following is a list of required configurable device behavior for DUTs being submitted for certification

The most effective way to implement this requirement is to provide the certification test vendor a way

to automatically trigger the behavior requested by the prompts in each test case The prompts are listed

in Appendix A and each prompt has a Unique identification, either an ID number or a test case and prompt sequence string This Identification information is displayed when the prompt is presented to the user This identification in the prompt can be used in a GUI provided by the DUT vendor to trigger a specific set of actions necessary to proceed with the test case.

 Async optOut with oadrCreatedEvent

 Kill Events(reset local state)

 Specify subset of resourceIDs in original event

 Kill Opt schedules (reset local state)

 Define opt schedule (optType (in or out) , vavailablity schedule(s) , dtstart, duration (allow zero) , send with same optID

o Report Service If report requestor functionality offered)

 Request Metadata report from VTN - one shot, periodic

 Force piggyback report cancellation in oadrUpdatedReport response payload (Only if VTN reports other than metadata report are supported)

 Force piggyback report request in oadrRegisteredReport response

 Kill periodic reports(reset local state)

 Report to follow flag

 Define Resources that are included in opt schedule and report metadata

 ResourceIDs, marketContext, DeviceClass

 Assign pre-allocated venID (if any)

 Support 10 minute buffer and 1 minute samples for telemetry reports

17

Trang 18

 Support 5 minute samples and 60 minute buffer for history usage reports

 Registration State

 Periodic Reports pending

 Pending and active events, current event state

 Simple, Electricity Price, Load Dispatch((powerReal)

o dtstart, duration, Interval(s) duration, Interval(s) value, priority, randomization

 Cancel Registration (send payload)

 Kill Registration(reset local state)

o Report Service

 Telemetry Status, Telemetry Usage , metadata

 one shot, periodic (granularity, reportBackDuration, dtstart, duration)

 Report to follow flag

 Kill periodic reports(reset local state)

 If supported, force piggyback report cancellation in oadrUpdateReport response

 if supported, force piggyback report request in oadrRegisteredReport response

 Poll at interval requested in oadrCreatedPartyRegistration

 Define Resources that are sent out with events

 ResourceIDs, marketContext, DeviceClass

 Assumed Defaults - oadrResponseRequired = always

Non-Registration test cases assume that the VEN has successfully registered with the VTN prior to running the test case Registration test cases will prompt the user for the expected state prior to

executing the test case

Vendors should configure their devices with test certificates from the OpenADR/NetworkFX portal ( http://www.networkfx.net/testcerts )

VEN's being submitted for certification must have host authentication of the X.509 client certificate CN field disabled in order to avoid complex reconfiguration of the test harness and Openfire server.

Trang 19

DUT Implementation Limits

The OpenADR 2.0b specification does not define implementation limits The following values are used in various test cases, and implementations being submitted for certification must support the following capabilities in order to successfully execute the certification test suite Note that these limits do not imply minimum market needs for “b” profile implementations.

 responseDescription, vtnComment – 250 characters

 venID, vtnID, requestID, uid, resourceID, eventID – 50 characters

 SignalName, MarketContext – 100 characters

 Maximum size of oadrDistributeEvent: A payload with 4 events each with 3 intervals per signal and a payload with 1 event containing 24 intervals

 Number-of instances of resourceID in eiTarget – 4

 Number of eventResponses in oadrResponse - 4

 Telemetry Reports, Assume support for a 1 minute sample rate, 10 minutes buffer storage

 History Report, optional but if supported, assume support for 10 minute sample rate, 1 hour of buffer

Items with Limited Testing

Appendix F and related footnotes provide a detailed summary of the how each conformance rule is tested by the Test Harness and any limitations associated with testing a particular conformance rule The Transport and Security Test Overview section of this document provides a detailed overview of how security requirements are tested and any limitations associated with that testing Finally, the items below capture additional requirements that have limited testing.

 As VTNs are only required to support metadata reports and it is unlikely they will support other reports, testing is limited to requesting metadata reports A consequence of this is that

oadrUpdateReport/oadUpdatedReport payload exchanges will not be tested in the VTN-to-VEN direction.

 The Table below shows the possible combinations of dtstart and duration in oadrUpdateReport payloads The scenarios that are grayed out are not tested.

19

Point Data

Overall dtstart

Overall Duration

Interval dtstart

Interval Duration

Number of Intervals

Trang 20

 Not all possible permutations of event status, modification number, response required, previous acknowledgement, and VEN response are tested Refer the VEN and VTN event state transition tests in the EiEvent section of this document

Common Pass and Fail Verdicts

In order to avoid repeating common expected results throughout test case definitions, unless stated otherwise in the test case definition, the following criteria will be used to determine if a test case passes

or fails:

 Common Pass Verdict

o Test case scenario runs to completion

AND

o All exchanged payloads validate against the OpenADR 2.0b schema

AND

o All OpenADR 2.0a conformance rule validation checks in Appendix X pass

 Common fail verdicts

o Test case scenario fails to run to completion because of a timeout, transport error, or other problem

Trang 21

Test Case Scenarios

EiRegisterParty Service Test Scenarios

VEN Registration - Query

Test Case Root ID

Use Case Scenario

VEN oadrQueryPartyRegistration payload elements

should be appropriate for transport and exchange model

set in properties for TH_VEN

-Prompt_002 - Before the test begins

Respond with template that includes all optional and mandatorypayload elements including one mock service specific

characteristic and one mock registration extension

-Prompt_001 - As shown in sequence diagram

Expected

Results

1) Common Pass/Fail verdicts

2) Received oadrCreatedPartyregistration payload from

VTN

3) Confirm venID and registrationID elements are not

returned by the DUT VTN

1) Common Pass/Fail verdicts2) VEN sent oadrCreatePartyRegistration Payload

21

Trang 22

VEN Registration - Query While Registered

VEN sends oadrQueryRegistration payload

After query has completed

Respond with template that includes ONLY required payload elements for oadrCreatedPartyRegistration

-Prompt_001 - As shown in sequence diagram

-Prompt_034 - pull only, Y/N Prompt (Pull Only)

-Prompt_033 - Y/N prompt

Expected

Results

2) Received oadrCreatedPartyregistration payload from

VTN

3) Verify venID and registrationID elements match

registration if returned by the VTN Note that these

elements are optional in the payload

1) Common Pass/Fail verdicts2) VEN sent oadrQueryRegistration Payload

Trang 23

VEN Registration - Bootstrap Sequence

Test Case Root ID Use Case Scenario

Objective Does the VTN DUT respond to a device registration and

then send its report metadata?

Can the VEN DUT initiate device registration and send its report metadata?

VEN oadrCreatePartyRegistration payload elements

Report registration should include Metadata_001 and

should do a oadrRequestEvent

-Prompt_003

Report registration should include Metadata_003

Expected

Results

2) VTN DUT provided it report metatdata

1) Common Pass/Fail verdicts2) VEN DUT initiated device registration and provided its report metadata, as well as requested its events via oadrRequestEvent following the exchange of metadata reports

23

Trang 24

VEN Registration - Pre-allocated VEN ID

Objective Does the VTN DUT respond to a device registration with

a preallocated venID and then send its report metadata?

Can the VEN DUT initiate device registration with a preallocated venID and send its report metadata?

2) VTN DUT provided it report metadata

3) Confirm that preallocated venID is returned in VTN

response

1) Common Pass/Fail verdicts2) VEN DUT initiated device registration and provided its report metadata

Trang 25

VEN Registration - Cancel Registration

-Prompt_005 - end of test

-Prompt_004

Expected

Results

2) VTN DUT acknowledges cancellation

1) Common Pass/Fail verdicts2) VEN DUT initiates registration cancellation

25

Trang 26

VTN Registration - Cancel Registration

Expected

Results

2) VTN DUT initiates registration cancellation

1) Common Pass/Fail verdicts2) VEN DUT acknowledges cancellation

Trang 27

VTN Registration - Request Re-Registration

-for TH_Push test, invert the presence of a trailing slash

on the oadrCreatePartyRegistration transportAddress

-Prompt_008

Expected

Results

2) VTN DUT initiates re-registration request

1) Common Pass/Fail verdicts2) VEN DUT initiates a re-registration after receiving the re-registration request

27

Trang 28

VEN Registration - New Registration while Registered

Objective Does the VTN DUT respond to a NEW device registration

while already registered and then send its report

-Do NOT include the venID or registrationID in the

oadrCreatePartyRegistrationRequest

2) Successful registration completed

3) VTN DUT provided its report metadata

1) Common Pass/Fail verdicts2) VEN DUT initiated device registration and provided its report metadata

Trang 29

VEN Registration - Re-Registration

Test Case Root ID Use Case ScenarioPull Test Case N1_0065_ S0-001 VEN Registration - Bootstrap Sequence - Skip the

metadata report exchange

Objective Does the VTN DUT respond to a device re-registration

when the VEN sends an oadrCreatePartyRegistration

with an existing registrationID? Note that with

reregistration the exchange of report metadata is not

-INCLUDE the venID or registrationID in the

oadrCreatePartyRegistrationRequest

-send oadrCreatePartyRegistration

Although not part of the formal test case, the DUT may

send an oadrRegisterReport and the test harness default

handlers will need to deal with this

-Prompt_049 - Have VEN initiate reregistration-Confirm that oadrCreatePartyRegistrationRequest includes venID and registrationID

-Return same venID and registrationID in oadrCreatedPartyRegistration

Although not part of the formal test case, the DUT may send an oadrRegisterReport and the test harness default handlers will need to deal with this

Expected

Results

2) Successful registration completed

3) Confirm that the VTN returns the same registrationID

and venID as these values should not change

1) Common Pass/Fail verdicts2) Successful registration completed

29

Trang 30

VTN_VEN Registration - Negative Test Scenarios

Objective Will the VTN handle the following error scenarios:

While in a registered state:

-Will the VTN reject a cancelRegistration with an

incorrect registrationID with a 452 error?

-Will the VTN reject an oadrCancelRegistration with an

incorrect venID with a 452 error?

Will the VEN handle the following error scenarios:While in a registered state:

-Will the VEN reject a cancelRegistration with an incorrect registrationID with a 452 error?

-Attempt a cancellation with invalid registrationID

-Attempt a cancellation with invalid venID

-Cancel Registration

If self test:

- Respond to TH_VEN payloads with expected errors and

responses

Prompt_050 - Y/N device or self test?

If real device

-Prompt_006 - Before the test begins -Attempt a cancellation with invalid registrationID -Cancel Registration

2) VTN DUT initiates registration cancellation

3) Invalid registrationID is in

oadrCancelPartyRegistration

1) Common Pass/Fail verdicts2) VEN DUT acknowledges cancellation3) Invalid registrationID are responded to with oadrCanceledPartyRegistration with a 452 error

NOTE: As both sides of this test case can be initiators of error conditions, one side must claim to be testing a real device and the other side claim to be doing a self test.

Trang 31

VEN Registration - Payload While Unregistered

Test Case Root ID

Use Case Scenario

Objective Does the VTN DUT ignore attempts to communicate if

not registered other than the registration query or

device registration payloads?

Does the VEN DUT ignore attempts to communicate if not registered?

Prompt_061- Y/N Initiate Payload?

If Yes

-If Push attempt to send OptSchedule_001

-If Pull attempt oadrPoll

If No

- If Pull, exit test case with a N/A status and the Message

"A pull VTN cannot initiate payloads, so this self test is

not relevant."

-Start VEN Server, do not respond to respond to

payloads Exit after timeout with an N/A status and a log/

console message that "Selftest placeholder "

Prompt_061- Y/N Initiate Payload?

If Yes

If Pull, exit test case with a N/A status and the log/console message "A pull VTN cannot initiate payloads, so this test is not relevant."

-If Push, attempt to send oadrRegisterReport

If No Start VTN Server, do not respond to respond to payloads Exit after timeout with an N/A status and a log/console message that

"Selftest placeholder "

Expected

Results

1) DUT should either not respond or returns 463 error in

an appropriate payload Test Harness should also accept

any error code in the appropriate plyload that starts with

“4”

1) DUT should either not respond or returns 463 error in an appropriate payload Test Harness should also accept any error code in the appropriate payload that starts with “4”

31

Trang 32

EiOpt Service Test Scenarios

All of the tests in this section assume that the DUT and Test Harness are in a registered state.

VEN Opt - New Opt Schedule

Objective Does the VTN DUT accept an opt schedule from

the VEN?

Can the VEN DUT send an opt schedule?

Trang 33

VEN Opt - Open Ended Opt Schedule

Objective Does the VTN DUT accept an opt schedule from

the VEN?

Can the VEN DUT send an opt schedule?

Scenario

Details

Include marketContext and eiTarget in payload

OptSchedule_004 - duration of zero

Prompt_037 - at start of test

Trang 34

VEN Opt - Cancel Opt Schedule

Objective Does the VTN DUT accept an opt schedule

cancellation from the VEN?

Can the VEN DUT send an opt schedule cancellation?

Scenario

Details

Exclude marketContext and ieTarget subelements from payload

OptSchedule_001 - optIn schedule

Prompt_015 - after schedule is sent

Trang 35

VEN Opt - Multiple Opt Schedules, Targeting

Objective Does the VTN DUT accept a revised opt schedule

from the VEN?

Can the VEN DUT send a revised opt schedule?

Scenario

Details

Add marketContext, eiTarget, and oadrDeviceClass:endDeviceAsset targeting information to opeSchedule_001 payload (may need to add to properties for DeviceClass)

OptSchedule_001 - First Schedule (optIn)

OptSchedule_002 - Second Schedule (optOut)

Prompt_016 - after first schedule is sent

35

Trang 36

VEN Opt - Multiple Opt Scheduled with Cancel

Objective Determine if the VEN generates an error if two

opt schedules use the same optID and then are cancelled

Determine if the DUT VEN can generate opt schedule, then cancel one

Scenario

Details

Use the same optID for both schedules

OptSchedule_001 - First Schedule (optIn)

OptSchedule_003 - Second Schedule (optIn)

Prompt_018 - after first schedule is sent

Prompt_015 - after second schedule is sent

Expected

Results

1) Common Pass/Fail verdicts2) VTN DUT accepts revised opt schedules and cancellation

1) Common Pass/Fail verdicts2) VEN DUT sends opt schedules and cancellation

Trang 37

VEN Opt - Negative Test Scenario

Objective Does the VTN DUT generate an error if an

attempt to cancel an opt schedule with an invalid optID?

This is a placeholder test case to provide a self test for the TH_VEN side of the transaction

Scenario

Details OptSchedule_001 - optIn schedule

Cancellation with invalid optID

Prompt_036- Y/N device or self test?

If self test

-Respond to oadrCreateOpt with oadrCreatedOpt -Respond to oadrCancelOpt with 452 error in oadrCancelled Opt

If real device

-Post to console and test report the following:

"Placeholder test case for self test, skipping execution for real device"

-Show a NA test result

Expected

Results

1) Common Pass/Fail verdicts2) VTN DUT accepts opt schedule cancellation3) VEN DUT rejects cancellation attempt with a 452 error

37

Trang 38

EiReport Service Test Scenarios

All of the tests in this section assume that the DUT and Test Harness are in a registered state.

VEN Reporting - One Shot Report

Objective Confirm report request, and report delivery is

successful with the VEN as the report producer

Telemetry status one-shot report

Confirm request, and report delivery is successful with the VEN as the report producer Telemetry status one-shot report

Scenario

Details

-Prompt_019 - Set up report request-Send Update_001 after receiving report request

-Request report: Request_001

Trang 39

VEN Reporting - Periodic Report

Objective Confirm report request, and periodic report delivery

is successful with the VEN as the report producer

Telemetry status periodic report

Confirm report request, and periodic report delivery is successful with the VEN as the report producer Telemetry status periodic report

Scenario

Details

-Prompt_020 - Set up report request-Send Update_001 after receiving report request, then again after the

reportBackDuration interval

Request report: Request_002

-Exit after 2nd update report payload is received

Trang 40

VEN Reporting - Delayed Periodic Report, loadControlState

Objective Confirm report request, and periodic report delivery

is successful with the VEN as the report producer

Telemetry status periodic report Delayed dtstart, complete loadControlStatus payload Include optional ID attribute in signedObject element

Confirm report request, and periodic report delivery is successful with the VEN as the report producer Telemetry status periodic report Delayed dtstart, complete loadControlStatus payload

Định dạng
Số trang	221
Dung lượng	0,97 MB