OpenADR 2.0a Certification Test Specification

Introduction This purpose of this document is to define a set of test cases that systems implementing the OpenADR 2.0a profile must successfully pass in order to obtain certification. Scope OpenADR 2.0a is an application layer message exchange specification used for twoway communication of Demand Response (DR), price, and Distributed Energy Resource (DER) signals between the electricity service provider and their customers. OpenADR 2.0a is defined in the following documents: • The OpenADR 2.0a Profile Specification, Revision 1.1 • The OpenADR 2.0a schema The testable requirements are defined as a set of conformance rules in the OpenADR 2.0a Profile Specification. These rules are referenced in both OpenADR 2.0 PICS and this document. Appendix B provides a mapping between each of these conformance rules and the test cases defined in the test specification. The scope of the test cases will focus on validating the successful exchange of messages between VTNs and VENs, or VTNVEN pairs, validating that any exceptions are handled gracefully and without service interruptions, and that security mechanism are properly implemented. Validating the functional behavior of the DUT in response to payload content is out of scope for the certification tests. For instance, the test cases will test that a VEN acknowledges the receipt of a DR Event but it will not validate that the VEN actually sheds load as a result of this message. For product certification, the certification test suite will be will be run over top of the specific transport protocols supported by the DUT. If multiple transports are supported by the DUT, the full conformance test will be run over one transport and a representative set of test cases will be run over the remaining transports (See Appendix C). Conformance testing of the supported transports is out of scope for the certification tests. However, limited testing will be done to validate OpenADR specific requirements for implementation of the transports, such as the inclusion of specific http headers. ...

Trang 1

OpenADR 2.0a

Certification Test Specification

Version 1.1.0

Trang 2

0.51  Added visual checks by test engineer for event execution.

 Edits based on feedback from Rolf

01/19/12 – JZ

0.6  Add negative test cases 01/24/12 – JZ

0.70  Misc updates, input from Rish

 Added balance of negative test cases

 Added transport test cases

01/31/12 – JZ

0.90  Updated to reflect conformance rule changes at test event

 Renumbered test cases

 Added security tests

 General Clean Up

02/06/12 - JZ

0.92  Update several test cases, fix minor errors

 Add prompts to test case definitions

0.97  Removed conformance rule test and just provided reference to the

OpenADR Profile Spec

03/09/12 – JZ0.98  Numerous changes related to the 03/23/12 changes to conformance rules

and schema

 Completed Appendix B, test case to conformane rule mapping

 Added Appendix D with a test case to event state mapping

0.98.4  Deleted test E2_0660

 Changes related to new conformance rules 65 and 66

 Modified test cases E2/E3 – 0685

 Added test cases E2/E3- 0432 and E0/E1 – 0267/268

04/16/12 – JZ

Trang 3

1.0.2  Minor Edits 08/09/12 - JZ

1.0.5  E3_0655, T9_1200 - Prompt has been modified

 E2_0530/E3_0530 - Prompt has been modified

 Scenario for test case E2_0527 has been modified

 Removed the following test cases

E0_0030, E1_0030, E0_0050, E1_0050, E0_0084, E1_0084, E0_0088, E1_0088, E0_0295, E0_0350, E1_0350, E0_0382, E1_0382, E0_0386, E1_0386, E2_0464, E3_0464, E2_0466, E3_0466, E2_0476, E3_0476, E2_0478, E3_0478

 Removed the following security tests

S0_1310, S0_1320, S0_1330, S0_1340, S0_1350, S0_1360, S0_1370, S1_1410, S1_1420, S1_1430, S1_1440, S2_1510, S2_1520, S2_1530, S2_1540, S2_1550, S2_1560, S2_1570, S3_1610, S3_1620, S3_1630, S3_1640, S3_1650, S3_1660

2/12/14 – EP, BD

1.0.6  Removed all references to startBefore

 Mantis 139: Added Adjacent Event Execution - E0_0292/E1_0292

 Mantis 164: Delete Test Case E0_0325/E1_0325 - MarketContext can have

a wild card, so can't do negative test

 Mantis 166: Modified conformance rule check to verify that "Z" is at the

end of every dateTime string

 Mantis 173: Conformance rule check modified to validate ordering of

intervals in reports

 Mantis 186: Added test cases to verify expired certificates: cases S1_1405

and S3_1605

 Revised tests E1_0285, E0_0285, E2_0510, E3_0510 and deleted tests

E2_0520, E3_0520, E2_530, and E3_0530 as the result of changes to rule regarding overlapping events

Trang 4

Table of Contents

Revisions: 2

Introduction 5

Terms and Acronyms 5

Scope 5

Test Methodology 6

Test Organization 7

DUT Configuration Requirements 8

VTN Test Interface 9

DUT Implementation Limits 9

Common Pass and Fail Verdicts 9

Items with Limited Testing 10

Application Layer Test Cases 11

oadrDistributeEvent Operational Sequence 11

VEN Push/Pull Positive Test Scenarios 12

VEN Push/Pull Negative Test Scenarios 40

VTN Push/Pull Positive Test Scenarios 50

VTN Push/Pull Negative Test Scenarios 68

Transport Test Cases 73

Simple Http 73

Security Test Cases 75

VEN Push Security Test Scenarios 76

VEN Pull Security Test Scenarios 77

VTN Push Security Test Scenarios 79

VTN Pull Security Test Scenarios 80

Appendix A – Conformance Rules Payload Validation 82

Appendix B – Conformance Rules to Test Case Mapping 83

Appendix C – Additional Transport Test Case List 87

Push VEN 87

Pull VEN 87

Push VTN 88

Pull VTN 88

Appendix D– Event State Sequences 89

Trang 6

This purpose of this document is to define a set of test cases that systems implementing the OpenADR 2.0a profile must successfully pass in order to obtain certification

Terms and Acronyms

The following terms and abbreviations are used in this document:

DUT Device Under Test – This is used to refer to the software

implementation of OpenADR and any supporting hardware necessary to execute the softwareDUT_VEN Device Under Test VEN

DUT_VTN Device Under Test VTN

TH Test Harness - A software implementation of OpenADR

playing the role of a VEN or VTN for the purpose of running test cases

TH_VEN Test harness playing the role of the VEN

TH_VTN Test harness playing the role of the VTN

VTN Virtual Top Node

Scope

OpenADR 2.0a is an application layer message exchange specification used for two-way communication

of Demand Response (DR), price, and Distributed Energy Resource (DER) signals between the electricity service provider and their customers OpenADR 2.0a is defined in the following documents:

 The OpenADR 2.0a Profile Specification, Revision 1.1

 The OpenADR 2.0a schema

The testable requirements are defined as a set of conformance rules in the OpenADR 2.0a Profile Specification These rules are referenced in both OpenADR 2.0 PICS and this document Appendix B provides a mapping between each of these conformance rules and the test cases defined in the test specification

The scope of the test cases will focus on validating the successful exchange of messages between VTNs and VENs, or VTN/VEN pairs, validating that any exceptions are handled gracefully and without service interruptions, and that security mechanism are properly implemented Validating the functional behavior of the DUT in response to payload content is out of scope for the certification tests For instance, the test cases will test that a VEN acknowledges the receipt of a DR Event but it will not validate that the VEN actually sheds load as a result of this message

Trang 7

For product certification, the certification test suite will be will be run over top of the specific transport protocols supported by the DUT If multiple transports are supported by the DUT, the full conformance test will be run over one transport and a representative set of test cases will be run over the remaining transports (See Appendix C) Conformance testing of the supported transports is out of scope for the certification tests However, limited testing will be done to validate OpenADR specific requirements for implementation of the transports, such as the inclusion of specific http headers.

The certification test suite will be run utilizing each of the security mechanisms supported by the DUT Conformance testing of the security mechanisms is out of scope for the certification tests However, limited testing will be done to validate OpenADR specific configuration requirements for

implementation of the security mechanisms, such as use of x509 certificates, TLS versions, and cipher suites

Exception handling tests will be limited to a small set of representative real world scenarios The intent

of these test cases will not be to exhaustively test all possible exception scenarios, but to validate that the DUT has sound exception handing capabilities For instance, a test might validate that the DUT trigger an error if it is sent a payload with an unknown VEN or VTN ID

Boundary and limit testing will utilize values that are deemed to be practical from a test execution perspective and typical from an implementation perspective, as specific implementation limits have not been defined by the OpenADR 2.0a Profile Specification For instance, testing that includes a

responseDescription string would use as most a few hundred characters, not 10’s of thousands

Optional elements will be testing will be tested as follows: Most payloads will include only mandatory elements only Where practical all optional elements will be used in a single payload Otherwise, the entire set of optional payload elements will be exercised cumulatively across the set of test cases for a given payload Optional elements do not need to be included in the payload, but if they are, the VEN or VTN receiving the payload must understand and act upon those optional elements

Test Methodology

The test harness will play the role of the opposite party (VEN or VTN) in interactions with the DUT Each test case will define a set of perquisites, a test scenario consisting of a sequence of VEN/VTN message exchanges, and an expected result Execution of that test scenario will result in payload exchanges between the DUT and the Test Harness to be analyzed The analysis will consist of the following:

 The message interaction patterns are as expected This includes…

o Correct response payloads For instance, receiving a oadrCreatedEvent payload in response to a oadrRequestEvent would not be correct

o Appropriate request payloads For instance a request containing oadrResponse payload would not be correct

 That payloads contain well formed XML

 That payloads conform to the alliance OpenADR 2.0a schema

 That the specific conformance rules defined in the OpenADR Profile Specification are followed For instance, an OpenADR 2.0a conformance rule states that the payload element signalType must contain the string “simple” This is not validated by the schema, so it is done as a separate conformance rule analysis step

Trang 8

 That the intent of the test case is met The test case may expect the VEN to send an optType of

“optOut” and if this is not received, the test case will fail

Note that some of the OpenADR conformance rules can be validated by simply analyzing the payloads ofall message exchanges that occur during execution of the certification test suites These rules are summarized in Appendix A Other conformance rules will require specific test scenarios to validate

A few test cases will require the test engineer executing the test to observe the behavior of the device under test in order to determine the pass fail status

Test Organization

Each test case will be classified into one or more groups as follows:

 VEN_Push –Tests where the DUT target is a VEN in a push message exchange pattern

 VTN_Push - Tests where the DUT target is a VTN in a push message exchange pattern

 VEN_Pull – Tests where the DUT target is a VTN in a pull message exchange pattern

 VTN_Pull – Tests where the DUT target is a VTN in a pull message exchange pattern

 Positive - This will be the bulk of the tests on successful message exchange A small subset of

the positive tests will be tests focused on determining if the DUT can handle values at the boundary/limits of what is typical from an implementation perspective

 Negative - Tests focused on the devices ability to gracefully handle unexpected message

exchange scenarios and payload contents

 Transport - Tests for OpenADR 2.0a specific transport requirements

 Security - Tests for OpenADR 2.0a specific security requirements

Test case numbering uses the following pattern: XY_ZZZZ

X = A letter designating the service or special test type

Test cases definitions within this document will be organized by operational sequences and further subdivided by role and exchange pattern (VEN/VTN, Push/Pull) Each test case will be defined in a table

as shown below The narrative text on the right describes each row in the table Optional test cases will

be indicated by the string “(Opt)” in the title

Trang 10

Test ID - name, Sequence (Opt)

Prerequisites Any prerequisite actions required prior to the execution of the test

scenario This may include configuring the DUT VTN with specific events, or instructions on the specific payload values to be used in payloads sent by the test harness Prerequisite pending or active events in the VTN or VEN will be programmatically generated by the test case unless indicated otherwise in the test case

Scenario The specific message exchange scenario required Note that conditional

steps based on whether the DUT supports a push or pull model are enclosed in brackets {}

Expected Results In addition to the common pass verdicts, the specific expected behavior

as the result of the test scenarios execution

{Notes} Optional support required to run test case, objectives of test case, and

other notes to clarify the implementation or execution of the test case

Test Engineer observations Instruction to the test engineer to observe the DUT to confirm the

status of the event execution

DUT Configuration Requirements

In order for many test cases to run successfully, the DUT must be capable of being configured to initiate action or respond to requests For instance, there should be a way to cause a DUT_VEN to respond to anevent with an optOut The following is a list of required configurable device behavior for DUTs being submitted for certification

 VEN implementations submitted for certification testing must provide some visual means for a test engineer to determine that an event is currently active

 Some backchannel way of setting the marketContext URI so that the DUT and test harness can interact with respect to the same marketContext

 A way to seed a VTN with events and to modify and/or cancel the events

 A way to opt out of an event with a VEN

 A way to trigger requests including oadrRequestEvent, and send oadrDistributeEvent

 A way to initialize the DUT such that all events are cleared from the data store

 A way to configure or determine the assigned VEN or VTN ID

 If supported, to configure additional identification values used by the VEN and VTN

 A way to set the polling frequency on the VEN

 A way to set responseRequired for event configuration on VTN

Trang 11

VTN Test Interface

Testing of a VTN requires various event scenarios to be configured on the VTN In order to provide a simple and easy to use interface for the test house that will be doing the certification, the vendor submitting the device for certification must provide the following interface:

 A drop down menu listing an option for each point in the test case execution where a VTN control panel configuration or action is requested via a prompt

 The naming convention for the drop down menu should be the test case name followed by an index indicating the prompt sequence within the test case, such as E3_0420_FirstPrompt for thefirst prompt in the test case requiring operator intervention

 Submitting or clicking on the menu option should do one of the following:

o The first request for operator intervention should clear the data store of any pending or active events and then add the requested event(s)

o Subsequent prompts to modify, cancel, or add additional events should be executed as instructed when the menu option is selected

o Prompts in push scenarios to resend the events in the existing data store should do just that

 Events should be grouped by common push or pull functionality and sorted alphabetically in the menu

DUT Implementation Limits

The implementation being submitted for certification must support the following minimum capabilities

in order to successfully execute the certification test suite Note that these limits do not imply minimummarket needs for an “a” profile implementations

 responseDescription, vtnComment – 250 characters

 venID, vtnID, requestID, uid, groupID, partyID, resourceID, eventID – 50 characters

 SignalName, MarketContext – 100 characters

 Maximum size of oadrDistributeEvent: A payload with 4 events each with 3 intervals per signal and a payload with 1 event containing 24 intervals

 Number of instances of resourceID in eiTarget – 4

 Number of eventResponses in oadrResponse - 4

Common Pass and Fail Verdicts

In order to avoid repeating common expected results throughout test case definitions, unless stated otherwise in the test case definition, the following criteria will be used to determine if a test case passes

or fails:

 Common Pass Verdict

o Test case scenario runs to completion

Trang 12

 Common fail verdicts

o Test case scenario fails to run to completion because of a timeout, transport error, or other problem

Items with Limited Testing

The following items are tested, but with a limited scope:

 As there is no programmatic way to determine when or if the DUT_VEN is executing an event, testing of tolerance, eiNotification, x-eiRampUp, x-eiRecovery, and dtstart is limited to sending values to the device and determining that no errors occur and the device opts into events as expected A limited set of test cases will construct scenarios where the test engineer executing the test case can observe a visual indicator on the VEN to determine when an event is active, which will allow some validation of correct behavior with respect to dtstart, duration, and tolerance

 Only “level” and “price” are used as values for signalType

 EiTarget in oadrDirstributeEvent is tested with all matching sub-elements and all non-matching sub-elements Although conformance rules state that sub-elements should be OR’d together to determine a match, the rules also state that the VEN’s behavior with respect to eiTarget is implementation dependent This creates a somewhat ambiguous situation with respect to a partial match and therefore is not tested

 Testing of multiple VTNs interacting with a single VEN will only be tested in the push direction from a single common IP address, providing that the DUT VEN supports configuration of multipleVTN relationships

 Conformance Rules 16, 31, 32, and 33 have limited or no test coverage as documented in Appendix B

 Not all possible permutations of event status, modification number, response required, previousacknowledgement, and VEN response are tested as documented in Appendix D

 Test case E0_0185 attempts to push two separate events to the VEN before the VEN responds with an oadCreatedEvent to the first event Due to limitations in the Test Harness framework, the test imposes a 5 second delay between the first event push and the second push

Trang 13

Application Layer Test Cases

Each table below represents the definition of a single test case Tests are organized by operational sequence Most test case definition tables map to two test case numbers, one for push and one for pull Braces {} are used to indicate conditional portions of the scenario based pull support or conditional responses from the DUT

oadrDistributeEvent Operational Sequence

The test cases in this section all utilize the following operational sequence:

PayloadVEN (Pull only) {oadrRequestEvent }

VEN (push only) {Transport Acknowledgement }

VEN (async) oadrCreatedEvent

Three types of oadrDistributeEvent(s) will be used in these test scenarios:

 Critical Peak Pricing (CPP) – 3 intervals

 24_Prices – 24 one hour intervals, notification 2 hours

 Dispatch – 1 interval with a duration of 0 and notification time of 0

Unless otherwise specified, the following defaults will be used for payloads generated by the test system:

oadrDistributeEvent:

 Mandatory elements unless otherwise specified in the test description

 marketContext as configured in the test system

 oadrResponseRequired set to always

 eiTarget with no subelements

 dtstart set to currentTime +2 minutes

 x-eiNotification set to a duration of 1 minute

 signalType for CPP and Dispatch set to level

 signalType for 24_Prices set to price

 signalID set to a placeholder value

 Overall duration for CPP set to 3 minutes

 Interval durations for CPP set to 1 minute

 Interval signalPayload set to a repeating sequence of 1, 2, 3, 2, 1

 eiResponse element with mandatory elements included if responding to a oadrRequestEvent

 requestID = unique value

oadrCreatedEvent:

 Mandatory elements only unless otherwise specified in the test description

Trang 14

 Respond with a oadrCreatedEvent to all oadrDistributeEvent payloads unless the test case description states differently with regards to oadrResponseRequired

oadrRequestEvent:

 Mandatory elements only unless otherwise specified in the test description

oadrRequestEvent, oadrResponse:

 requestID set as an empty element

 Mandatory elements only

VEN Push/Pull Positive Test Scenarios

The VEN Push test cases are functionally identical to the VEN Pull tests cases in a pull scenario the exchange sequence is initiated by the DUT_VEN using an oadrRequestEvent

Each of the test cases below show a conditional oadrRequestEvent as the first operation in the sequencewhich applies to the pull scenario only and a conditional transport acknowledgement which applies to the push scenario only

Trang 15

E0_0020 – No Events, VEN Push oadrDistributeEvent (Opt)

E1_0020 – No Events, VEN Pull oadrDistributeEvent

Prerequisites

Scenario {DUT_VEN: oadrRequestEvent }

TH_VTN: oadrDistributeEvent {DUT_VEN: Transport Acknowledgement}

Expected Results 1)Common Pass/Fail verdicts

2)No oadrCreatedEvent returnedNotes -No events returned to the VEN in the oadrDistibuteEvent payload

Trang 16

E0_0040 –Send 24_Prices Event, VEN Push oadrDistributeEvent

E1_0040 –Send 24_Prices Event, VEN Pull oadrDistributeEvent

Prerequisites - DUT_VEN has no pending or active events

TH_VTN: oadrDistributeEvent – 24 Prices Event, “always”

{DUT_VEN: Transport Acknowledgement}

DUT_VEN: oadrCreatedEvent TH_VTN: oadrResponse , success

Note: set the responseRequired element in the oadrDistributeEvent to “always” for this test case Set signalType to “prices”

2)Received oadrCreatedEvent with optType equal to optInNotes -A basic 24 hour pricing event transaction

Trang 17

E0_0060 –Send Event with Optional Elements, VEN Push oadrDistributeEvent

E1_0060 –Send Event with Optional Elements, VEN Pull oadrDistributeEvent

TH_VTN: oadrDistributeEvent – CPP Event{DUT_VEN: Transport Acknowledgement}

Note: TH_VTN initialized with one new CPP Event containing the following optional elements:

 responseDescription = “success” (pull only)

2) Received oadrCreatedEvent with optType equal to optInNotes -Include some optional elements in the oadrDistributeEvent payload

Trang 18

E0_0070 –Send Event with EiTarget subElements, VEN Pull oadrDistributeEvent

E1_0070 –Send Event with EiTarget subElements, VEN Pull oadrDistributeEvent

Scenario Prompt: “If supported, the VEN and test harness properties should be configured

with matching target elements including groupId, resourceID, venID, and partyID Ifnot supported by the DUT, leave test harness properties for these elements undefined.”

{DUT_VEN: oadrRequestEvent } TH_VTN: oadrDistributeEvent {DUT_VEN: Transport Acknowledgement}

DUT_VEN: oadrCreatedEvent(s) TH_VTN: oadrResponse , success

Note: TH_VTN initialized with one new CPP Event containing the following optional eiTarget subelements If the DUT_VEN could not be configured to specify a value for one of these elements, include an empty element

 groupID

 resourceID

 venID

 partyID

2) Received oadrCreatedEvent with optType equal to optIn

Notes -Matching eiTarget subelements sent as part of payload to VEN

-Even though response to eiTarget is implementation dependent, the VEN should respond to the event as there is a matching condition and oadrResponseRequired

is set to always

Trang 19

Event State Sequence Tests, VEN Push oadrDistributeEvent

Event State Sequence Tests, VEN Pull oadrDistributeEvent

Note that E0 tests a push and E1 tests are Pull

Test ID EventStatus Mod Number ResponseReq Previously Ack'd

Prerequisites - Tests designated with Mod=1 must have a new event sent to the DUT_VEN with

an acknowledgement before initiating the primary test sequence on that event-Tests designated as previously ack’d = Yes ,must send the designated test sequence as both a prerquisite to get the acknowledgement, then again as the primary sequence

Sequence Prerquisite if required…

{DUT_VEN: oadrRequestEvent } TH_VTN: oadrDistributeEvent – CPP Event{DUT_VEN: Transport Acknowledgement }DUT_VEN: oadrCreatedEvent(s)

TH_VTN: oadrResponse , success

5 second delayPrimary Test Sequence…

{DUT_VEN: oadrRequestEvent } TH_VTN: oadrDistributeEvent - same CPP Event{DUT_VEN: Transport Acknowledgement }

Trang 20

Notes:

1)Use CPP event templateS 2)Pending events should be set 5 minutes in the future 3) Pending event modifications should be to set to alter the dtstart time to one minute further in the future.

4)Active events should be set 1 minute in the past 5)Active event modifications should extend the overall duration and last interval duration by one minute

2) For events with responseRequired set to never, the ven should not respond 3)For events with responseRequired set to always, the DUT should do an optIn with correct modification number returned in eventResponse

Notes -Iterate through a number of events involving different EventStatus,

modificationNumber, responseRequired, and previously acknowledged states

Trang 21

E0_0110 –Send new event with modificationNumber >0 , VEN Push oadrDistributeEvent

E1_0110 –Send new event with modificationNumber >0 , VEN Pull oadrDistributeEvent

Prerequisites - DUT_VEN has no active events

TH_VTN: oadrDistributeEvent – Mod = 3{DUT_VEN: Transport Acknowledgement}

Note: TH_VTN to send CPP event with the modification number set to 3

2) Received oadrCreatedEvent with optType equal to optIn3) As the DUT_VEN has not seen the eventID before, it should treat it as a new event even though the modification number is greater than 0

Notes Send a new event whose modification number is greater than 0 simulating a

scenario where the VEN may have missed the initial new event transmission

Trang 22

E0_0130 –Send test event , VEN Push oadrDistributeEvent

E1_0130 –Send test event , VEN Pull oadrDistributeEvent

Scenario Prompt: “Make sure that the test harness properties have been configured with

an implementation specific testEvent string to send as part of the payload for this test case.”

{DUT_VEN: oadrRequestEvent } TH_VTN: oadrDistributeEvent – Test Event{DUT_VEN: Transport Acknowledgement}

Note: TH_VTN sends a new CPP event and the testEvent element set to a value other than “false”, pulled from the test harness properties

2) Received oadrCreatedEvent with optType equal to optIn3) The assumption is that the event will execute normally, however the VEN will not initiate DR actions based in the signaling (validating this is out of scope)Notes -Send a test event and verify the message exchange proceeds as if a real event is

occurring

Trang 23

E0_0180 –New, modified, Cancelled in one payload , VEN Push oadrDistributeEvent

E1_0180 –New, modified, Cancelled in one payload , VEN Pull oadrDistributeEvent

TH_VTN: oadrDistributeEvent – two new events{DUT_VEN: Transport Acknowledgement}

DUT_VEN: oadrCreatedEvent(s) TH_VTN: oadrResponse , success {DUT_VEN: oadrCreatedEvent(s) }{ TH_VTN: oadrResponse , success }{DUT_VEN: oadrCreatedEvent(s) }{ TH_VTN: oadrResponse , success }Delay for 5 seconds – push only{DUT_VEN: oadrRequestEvent } TH_VTN: oadrDistributeEvent – new, cancelled, modified{DUT_VEN: Transport Acknowledgement}

DUT_VEN: oadrCreatedEvent(s) TH_VTN: oadrResponse , success {DUT_VEN: oadrCreatedEvent(s) }{ TH_VTN: oadrResponse , success }{DUT_VEN: oadrCreatedEvent(s) }{ TH_VTN: oadrResponse , success }

Note: The TH_VTN sends two new events Then a second oadrDistribute event is sent containing one new event, a modification of a previous event, and a

cancellation of a previous event Configuration of events as follows:

- DUT_VEN has two CPP pending events with dtstart at current time + 5 minutes and current time + 10 minutes.

-TH_VTN initialized to send 3 events in a single payload as follows:

 A new CPP event with dtstart at current time + 15 minutes

 A modified version of pending DUT_VEN event Extend dtstart by one minute

 A cancellation of the second pending DUT_VEN event

2) Received oadrCreatedEvent with optType equal to optIn for the eventID’s related to the new and modified event, and optType equal to optIn or optOut for the cancelled event

3) From one to three oadrCreated events may be returned containing the optType values for the 3 events

Notes -A more complex scenario with a single oadrDistributeEvent payload containing

new, modified, and cancelled events

Trang 24

E0_0185 –oadrCreatedEvent Response, VEN Push oadrDistributeEvent

Scenario TH_VTN: oadrDistributeEvent – one new event

DUT_VEN: Transport Acknowledgement

TH_VTN: oadrDistributeEvent – old event and one new eventDUT_VEN: Transport Acknowledgement

DUT_VEN: oadrCreatedEvent – Response to 1st payload TH_VTN: oadrResponse , success

DUT_VEN: oadrCreatedEvent(s) - Response to 2nd payload TH_VTN: oadrResponse , success

{DUT_VEN: oadrCreatedEvent(s) }{ TH_VTN: oadrResponse , success }

Note: The TH_VTN sends two oadrDisributeEvent payloads without waiting for the asyc oadrCreatedEvent from the 1 st payload The VEN should respond to each payload independently.

Set up the first event as a CPP pending event with dtstart at current time + 5 minutes and duration of 5 minutes

Set up the second payload with a new event with dtstart at current time + 20 minutes and duration of 5 minutes The second payload must also contain the event from the 1 st payload

2) Received oadrCreatedEvent with optType equal to optIn for the eventID from the first payload

2) Received oadrCreatedEvent with optType equal to optIn for the eventID from the second payload

3) From one to three oadrCreated events may be returned containing the optType values for the 3 events

Notes -Send two consecutive oadrDistribute events with no intervening delay in order to

see if the VEN responds to both payloads independently

Trang 25

E0_0190 – Static requestID, VEN Push oadrDistributeEvent

E1_0190 – Static requestID, VEN Pull oadrDistributeEvent

DUT_VEN: oadrCreatedEvent(s) TH_VTN: oadrResponse , success Delay 5 seconds – Push Only{DUT_VEN: oadrRequestEvent } TH_VTN: oadrDistributeEvent – Same request ID{DUT_VEN: Transport Acknowledgement}

DUT_VEN: oadrCreatedEvent(s) TH_VTN: oadrResponse , success Note: The TH_VTN sends a new CPP Event, follow by a second CPP event that has the same requestID as the previous oadrDistributeEvent message

2)Received oadrCreatedEvent with optType equal to optIn in both instancesNotes -Confirm that the VEN does not require the requestID contained in the

oadrDistributeEvent payload to be unique

Trang 26

E0_0200 – Limit Values, VEN Push oadrDistributeEvent

E1_0200 – Limit Values, VEN Pull oadrDistributeEvent

dtstart values set respectively to current time plus 5 minutes, 10 minutes, 15 minutes, and 20 minutes

TH_VTN: oadrDistributeEvent – four events, limits{DUT_VEN: Transport Acknowledgement}

DUT_VEN: oadrCreatedEvent TH_VTN: oadrResponse , success {DUT_VEN: oadrCreatedEvent(s) }{ TH_VTN: oadrResponse , success }{DUT_VEN: oadrCreatedEvent(s) }{ TH_VTN: oadrResponse , success }{DUT_VEN: oadrCreatedEvent(s) }{ TH_VTN: oadrResponse , success }

Note-TH_VTN sends a payload with 4 new CPP events (3 intervals each) containing the following values:

 responseDescription, vtnComment – 250 characters

 vtnID, requestID, groupID, partyID, resourceID, eventID, eventID – 50 characters

 SignalName, MarketContext – 100 characters

 In eiTarget 4 instances of groupID, PartyID, and resourceID, and one matching VEN ID

2)Received oadrCreatedEvent with optType equal to optIn Notes -Confirm the implementation limits from the Certification test do not cause any

issues with the DUT_VEN

Trang 27

E0_0210 –Observation 1: Simple Event, VEN Push oadrDistributeEvent

E1_0210 – Observation 1, Simple Event, VEN Pull oadrDistributeEvent

Prompt: “Test execution has paused to allow for manual intervention The test

engineer should observe the active event indication on the VEN The event should become active in approximately 2 minutes and stay active for 3 minutes

Prompt: Please answer Yes if the active event indicator showed VEN starting after

approximately 2 minutes and lasting approximately 3 minutes

Note: TH_VTN initialized with new CPP Event with the following settings:

 dtstart set to currentTime +2 minutes

 Overall duration for CPP set to 3 minutes

 Two Intervals with a duration of 1 minute and 2 minutes

2)Received oadrCreatedEvent with optType equal to optIn3)Visual observations as noted

Notes -A basic CPP event transaction with visual observation of active state

Test Engineer

observations

Test Engineer should observe visual indicator on VEN, verifying the following:

 Event started at the current time + 2 minutes

 That the event active indicator on the DUT_VEN stays on for 3 minutes once the events starts, and then turns off

Trang 28

E0_0220 –Observation 2: modify pending event, VEN Push oadrDistributeEvent

E1_0220 – Observation 2: modify pending event, VEN Pull oadrDistributeEvent

DUT_VEN: oadrCreatedEvent TH_VTN: oadrResponse , success Delay for 5 seconds = push only{DUT_VEN: oadrRequestEvent } TH_VTN: oadrDistributeEvent – Modified Event{DUT_VEN: Transport Acknowledgement}

engineer should observe the active event indication on the VEN The event should become active in approximately 1 minute and stay active for 3 minutes

Prompt: Please answer Yes if the active event indicator showed VEN starting after

approximately 1 minute and lasting approximately 3 minutes

Note: The first oadrDistributeEvent CPP event sent by the TH_VTN should be configured as follows”

 Two Intervals with a duration of 1 minute and 2 minutes The second oadrDistributeEvent should contain a modified version of the previous events, configured as follows:

 Event is modified so that the start time is current time + 1 minute on TH_VTN and sent to the VEN

2)Received oadrCreatedEvent with optType equal to optIn3)Visual observations as noted below

Notes -A basic CPP event that is extended with pending to have a different dtstart time

Test Engineer

observations

 Event started approximately 1 minute after the modified event was received by the VEN

 That the event active indicator on the DUT_VEN stays on for 3 minutes once the events starts, and then turns off

Trang 29

E0_0230 –Observation 3: modify active event, VEN Push oadrDistributeEvent

E1_0230 – Observation 3: modify active event, VEN Pull oadrDistributeEvent

Scenario

{DUT_VEN: oadrRequestEvent}

TH_VTN: oadrDistributeEvent{DUT_VEN: Transport Acknowledgement}

Prompt: “Test execution has paused to allow for manual intervention Please

observe the active event indicator on the VEN When it indicates an event is active,click resume on this prompt.”

{DUT_VEN: oadrRequestEvent } TH_VTN: oadrDistributeEvent – Modified Event{DUT_VEN: Transport Acknowledgement}

Prompt: “The active event indicator should go off in approximately 5 minutes

Please answer Yes to confirm that the active event indicator turned off as expected.”

 Note: TH_VTN sends CPP event to DUT_VEN with the following characteristics:dtstart set to currentTime +1 minutes

 Two Intervals with a duration of 28 minutes and 2 minutes

 After event goes active the TH_VTN sends as modified event as follows :Event is modified so that the first interval duration is changed from 28 minutes to 3 minutes (with overall duration changes to 5 minutes) on the TH_VTN

Notes -A basic CPP event that is whose duration is shortened while the event is active

Test Engineer

observations

 That the event active indicator on the DUT_VEN stays on for approximately

3 minutes once the modified event is received and then turns off

Trang 30

E0_0240 – Observation 4: cancel event, VEN Push oadrDistributeEvent

E1_0240 – Observation 4, cancel event, VEN Pull oadrDistributeEvent



observe the active event indicator on the VEN When it indicates an event is active(about 1 minutes), press resume on this prompt.”

TH_VTN: oadrDistributeEvent - Cancelled{DUT_VEN: Transport Acknowledgement}

Prompt: “Please answer Yes to confirm that the event active indicator went off

within 1 minute of pressing resume on the previous prompt

Note: TH_VTN sends CPP event to DUT_VEN with the following characteristics:

 dtstart set to currentTime + 1 minutes

 Two Intervals with a duration of 15minute and 15 minutes The second oadrDisributeEvent send by the TH_VTN cancelled the previously sent event.

Notes -A basic CPP event that is cancelled part way through execution

Test Engineer

observations

 That the event active indicator on the DUT_VEN turns off shortly after the receipt of the cancellation

Trang 31

E0_0250 –Observation 5: Dispatch Event, VEN Push oadrDistributeEvent

E1_0250 – Observation 5, Dispatch Event, VEN Pull oadrDistributeEvent

TH_VTN: oadrDistributeEvent{DUT_VEN: Transport Acknowledgement}

observe the active event indicator on the VEN When it indicates an event is active (about 2 minutes), press resume on this prompt.”

TH_VTN: oadrDistributeEvent – Same event with cancellation{DUT_VEN:

Transport Acknowledgement}

Prompt: “Please answer Yes to confirm that the event active indicator went off

within 1 minute of pressing resume on the previous prompt

Note: TH_VTN first sends ad new Dispatch Event with the following settings:

 dtstart set to currentTime +2 minutes

 Overall duration for CPP set to 0 minutes

 One Intervals with a duration of 0 minute

 Interval signalPayload set to 2

The TH_VTN then sends the same event as in the previous step with the eventStatus set to cancelled

2)Received oadrCreatedEvent with optType equal to optIn3)Visual observations as noted above

Notes -A basic Dispatch event transaction Confirm that event is active until cancelled

Test Engineer

observations

 That the event active indicator stayed on until the event was cancelled

Trang 32

E0_0260 –Observation 6: Randomization VEN Push oadrDistributeEvent

E1_0260 – Observation 6, Randomization, VEN Pull oadrDistributeEvent

Scenario Prompt: “Test execution has paused to allow for manual intervention This test

observes a randomized start time from between 1 and 4 minutes from the time you press resume on this prompt” Please record the time starting from when clickresume on this prompt until you see the active event indicator indicate an event has started, and confirm the event lasts 3 minutes Run this test several time until you observer a noticeable difference in start time between test execution

Prompt: “Please answer Yes to confirm you have run this test case several times

and observed a difference in the start time between test runs and that the event itself lasted 3 minutes, otherwise answer No

Note: TH_VTN sends a new CPP Event with the following settings:

 start after set to 3 minutes

Notes -A basic CPP event transaction with tolerance values to randomize the event start

Trang 33

E0_0262 –Observation 10: Modified Event Randomization VEN Push oadrDistributeEvent

E1_0262 – Observation 10, Modified Event Randomization, VEN Pull oadrDistributeEvent

observes a randomized start time from between 1 and 4 minutes from the time you press resume on this prompt” Please record the time starting from when clickresume on this prompt until you see the active event indicator indicate an event has started Run this test several times until you observe a noticeable difference in start time between test execution iterations.”

TH_VTN: oadrDistributeEvent – With tolerance element, dtstart + 30 minutes{DUT_VEN: Transport Acknowledgement}

DUT_VEN: oadrCreatedEvent TH_VTN: oadrResponse , success{DUT_VEN: oadrRequestEvent}

TH_VTN: oadrDistributeEvent – With tolerance element, dtstart + 1 minute{DUT_VEN: Transport Acknowledgement}

and observed a difference in the start time between test runs, otherwise answer No

Note: TH_VTN sendsa new CPP Event with the following settings:

 dtstart set to currentTime + 30 minutes

On the second oadrDisributeEvent payload, modify the event such that the dtstart is equal to current time plus 1 minute

Notes -A basic CPP modified event transaction with tolerance values to randomize the

event start time

Test Engineer

observations

The test engineer should run the test case several times until a noticeable difference is detected between the time the test case execution completes and when the active event indicator on the VEN indicates that an event is occurring

Trang 34

E0_0267 –Observation 8: Explicit Cancel Randomization VEN Push oadrDistributeEvent

E1_0267 – Observation 8, Explicit Cancel Randomization, VEN Pull oadrDistributeEvent

observes the end time randomization of a cancelled event Please press Resume to send an event to the VEN.”

TH_VTN: oadrDistributeEvent – With tolerance element{DUT_VEN: Transport Acknowledgement}

Prompt: “Test execution has paused to allow for manual intervention When the

active event indicator shows that the event is active, click Resume on this prompt Record the time that it takes between pressing Resume on this prompt and the active event indicator showing the event has been cancelled (should be between 0 and 2 minutes) Run this test several times until you observe a noticeable

difference in the cancellation end time between test execution iterations.”

TH_VTN: oadrDistributeEvent – With cancel{DUT_VEN: Transport Acknowledgement}

and observed a difference in the end time, otherwise answer No

 dtstart set to currentTime plus 1 minute

 Two Intervals with a duration of 3 minutes and 2 minutes The second oardDistributeEvent Payload explicitly cancels the event.

time Test objective is to cancel the event and observe that the cancellation is randomized

Test Engineer

observations

The test engineer should run the test case several times until a noticeable difference is detected between the time the event is cancelled and when the activeevent indicator indicates the event has been stopped

Trang 35

E0_0268 –Observation 9: Implicit Cancel Randomization VEN Push oadrDistributeEvent

E1_0268 – Observation 9, implicit Cancel Randomization, VEN Pull oadrDistributeEvent

observes the end time randomization of an implicitly cancelled event Please press Resume to send an event to the VEN.”

TH_VTN: oadrDistributeEvent – With tolerance element{DUT_VEN: Transport Acknowledgement}

Prompt: “Test execution has paused to allow for manual intervention When the

active event indicator shows that the event is active, click Resume on this prompt Record the time that it takes between pressing Resume on this prompt and the active event indicator showing the event has been cancelled (should be between 0 and 2 minutes) Run this test several times until you observer a noticeable

difference is the cancellation end time between test execution iterations.”

TH_VTN: oadrDistributeEvent – Empty{DUT_VEN: Transport Acknowledgement}

and observed a difference in the end time, otherwise answer No

 dtstart set to currentTime plus 1 minute

 Two Intervals with a duration of 3 minutes and 2 minutes The second oardDistributeEvent Payload should be sent with no events in it to trigger an implicit cancellation of the event.

time Test objective is to cancel the event and observe that the cancellation is randomized

Test Engineer

observations

The test engineer should run the test case several times until a noticeable difference is detected between the time the event is cancelled and when the activeevent indicator indicates the event has been stopped

Trang 36

E0_0270 – Multiple Event Execution, VEN Push oadrDistributeEvent

E1_0270 – Multiple Event Execution, VEN Pull oadrDistributeEvent

TH_VTN: oadrDistributeEvent – 2 CPP Events{DUT_VEN: Transport Acknowledgement}

DUT_VEN: oadrCreatedEvent(s) TH_VTN: oadrResponse , success {DUT_VEN: oadrCreatedEvent(s) } { TH_VTN: oadrResponse , success}

engineer should observe the active event indication on the VEN Two events should execute The first should start in 1 minute and last 1 minute The second will start in 3 minutes and last 3 minutes.”

Prompt: “Please answer Yes if the active event indicator showed two events

execute the first lasting 1 minute and the second lasting 3 minutes.”

Note: TH_VTN initialized with two Events with the following settings:

 Event 1: dtstart=current time + 1 minute, duration = 1 minute

 Event 2: dtstart= current time + 3 minutes, duration =3 minutes

2)Received one or two oadrCreatedEvent messages cumulatively containing optIn for both events

Notes -Two events in a single oadrDistributeEvent payload

Trang 37

E0_0280 – Observation 7, Async Opt Out, VEN Push oadrDistributeEvent

E1_0280 – Observation 7 Async Opt Out, VEN Pull oadrDistributeEvent

engineer should wait until active event indicator shows an event is active (about 1 minute), then initiate an optOut using the VEN control panel The event should stop executing with 1 minute of the initiating the optOut

Prompt: Please answer Yes if the active event indicator showed the optOut

effectively cancelled the event from the VEN’s perspective

Note: TH_VTN initialized with new CPP Event with the following settings:

2)Received oadrCreatedEvent with optType equal to optIn3)Visual observations as noted

Notes -Observe an Async optOut initiated from the VEN control panel

Test Engineer

observations

 That the event active indicator on the DUT turns off within 1 minute of manually initiating an optOut

Trang 38

E0_0285 – Multiple Market Contexts, VEN Push oadrDistributeEvent

E1_0285 – Multiple Market Contexts, VEN Pull oadrDistributeEvent

TH_VTN: oadrDistributeEvent – 2 CPP Events{DUT_VEN: Transport Acknowledgement}

Note: TH_VTN sends two new CPP Events, each in a separate market context, but with second event in the payload having a higher priority than the first The events should be overlapping with dtstart values at current time + 5 minutes

2) The behavior of overlapping events is undefined, so the intent of this test case is to simply verify that the DUT does not crash as the result of receiving two overlapping events Acceptable results would include opting In, Opting Out, or generating a 4xx error at either the payload or event level in the VENs oadrCreatedEvent response.

Notes -Send two overlapping events in the two market contexts

Trang 39

E0_0290 – Dual Push/Pull Mode, VEN Push oadrDistributeEvent (opt)

Scenario Prompt: “A Push VEN should also be able to concurrently communicate in a pull

model If the device requires enabling of the polling, please do so now The test case will push an event to the VEN, then will poll for that event.”

TH_VTN: oadrDistributeEvent DUT_VEN: Transport AcknowledgementDUT_VEN: oadrCreatedEvent

TH_VTN: oadrResponse , success Pause 5 seconds

DUT_VEN: oadrRequestEvent } TH_VTN: oadrDistributeEvent DUT_VEN: oadrCreatedEvent TH_VTN: oadrResponse , success

Prompt: “Please disable polling for subsequent testing”

Note: Set response required to always on the poll oadrDistributeEvent TH_VTN initialized with new CPP Event with the following settings:

2)Received oadrCreatedEvent with optType equal to optIn on both events3)Visual observations as noted

Notes -Demonstrate that VEN can concurrently operating in push a push and pull model

Trang 40

E0_0292 – Adjacent Event Execution, VEN Push oadrDistributeEvent

E1_0292 – Adjacent Event Execution, VEN Pull oadrDistributeEvent

TH_VTN: oadrDistributeEvent – 2 Events{DUT_VEN: Transport Acknowledgement}

Note: TH_VTN initialized with two Events with the following settings:

Event 1:

 dtstart= CurrentTime + 1 minutes, duration = 5 minutes

 SIMPLE, signalType = level

 Interval #1, 5 minutes, signalPayload = 2Event 2

 dtstart= End time of Event 1, duration = 5 minutes

 SIMPLE, signalType = level

 Interval #1, 5 minutes, signalPayload = 2The second event's dtstart time must be exactly aligned with the effective end time

of the first event (i.e dtstart + duratrion)

2) VEN DUT opts in to both events

Notes Will the VEN DUT accept two events where the end time of the first event is

aligned with the start time of the subsequent Event?

Note this test case maps to E1_1090 and E0_6090 in the B Profile Test Suite

Định dạng
Số trang	94
Dung lượng	178,56 KB