Cisco networkers 2009 session BRKDCT 2998 cabling for next generation data center technologies DDU

The session explores server access layer network architecture deployment models for Top-of-Rack ToR and Middle of Row MoR / End-of-Row EoR, Distributed Access Fabric DAF with respect to

Cabling for Next Generation

Data Center Technologies

BRKDCT-2998

Cabling for Next Generation Data

Center Technologies

Abstract:

A d di f f ili i d i i i d h l i i li i h

An understanding of facilities and its restrictions, and the resulting implication on the

placement of switches, servers, racks/cabinets and cabling media is critical to the

success of the planning and deployment of a Greenfield data center The transition

from 1G to 10GE and the evolution toward Unified Fabric environments have close

ties with the physical critical infrastructure The next generation data center

ties with the physical critical infrastructure The next generation data center

architecture is as good as the flexibility provided by physical facilities Therefore, a

facilities architecture that enables network layer architecture flexibility will provide

customer with network deployment options at enable modularity and flexibility.

The session explores server access layer network architecture deployment models for Top-of-Rack (ToR) and Middle of Row (MoR) / End-of-Row (EoR), Distributed Access Fabric (DAF) with respect to cabling strategy to support 1G to 10GE migration, 10GE ( ) p g gy pp g , transceiver form factors, 40GE/100GE infrastructure support, horizontal and in rack

cabling Modular design methodology leveraging the POD concept for repeatable

access layer building blocks to support server access and aggregation will be explored with respect to impact on the critical facility cabling architecture.

2

Agenda

DC Facilities Top of Mind

Complexity, Cost, Power, Cooling

Standards Compliance

Reliability, Availability

Increased Efficiency, Simpler Operations

Scalability, Flexibility,

4

y, y Management, Security

Future Proofing

Modularity, Mobility

Data Center Architectures and Design

several correlated inputs

Density?

current and future needs

required to maximize

Investment in Data Center

Facilities & infrastructure

Unified Fabric?

10 Gigabit Ethernet to the Server

Impacting DC access Layer Cabling Architecture

Impacting DC access Layer Cabling Architecture

bandwidth per server

increased business agility

Unified Fabrics / UIO

Plant, 10G/40G/100G

6

High Speed Ethernet Adoption

on Servers

10GE Copper NIC Trend

10GBASE-T PHY from NIC to LOM – The Server View

ƒ Dual port NICs on

2008 1 ili 90 / 10W

2008 1st gen silicon 90nm w/ ~10W

2009 2nd gen silicon 90-65nm w/ ~6W 2010/11 3rd gen silicon 65-40nm w/ ~4W peak, ~3W Avg w/EEE (802.3az)

2012 - 4th gen silicon 40nm w/ ~3W peak, <1W Avg w/EEE

ƒ LOM removes the cost barrier to adopt 10G on servers

ƒ Server vendors require LOM to be backward compatible, hence LOMs

should support:

interoperate with 100/1000/10000 switches support RJ45 cabling infrastructure

ƒ PHY of choice for 10G LOM

10G Sever Media Option ?

• Fiber, Copper (CX1) , 6A in rack

10G Sever Media Option ?

• Fiber, Copper (CX1) , 6A in rack

8

Fiber, Copper (CX1) , 6A in rack

• Fiber, CX1, 10GBASE-T (2011) Fiber, Copper (CX1) , 6A in rack

• Fiber, CX1, 10GBASE-T (2011)

UIO w/ Fiber Channel Over

Ethernet (FCoE)

Packet into a Ethernet Frame

Lossless Packet Performance

cost Ethernet electronics

up to the SAN switch

FCoE Server Adapter Card FCoE Gateway Line Card

10 Gigabit Ethernet for

Power (each side) Distance

~50% power savings with EEE

© 2009 Cisco Systems, Inc All rights reserved Cisco Public

Cat6 Cat6a/7 Cat6a/7

RJ45 10GBASE-T

*** As of 2008; expected to decrease over time

* Terminated cable ** Draft 3.0, not final

DC Server Access Layer

Interconnect Concerns

ƒ Balancing cabling requirements with Balancing cabling requirements with

Power, Cooling, Loading, Modularity, Row / Rack level mobility y y

Migration to Unified Fabric g

General Considerations for DC Cabling

ƒ Fast deployment and accurate

moves adds and changes

ƒ Fast deployment and accurate

moves adds and changes

ƒ High quality, reliability and scalability

ƒ High quality, reliability and scalability

moves, adds, and changes

ƒ Standards based open

systems

moves, adds, and changes

ƒ Standards based open

systems

scalability

ƒ Redundancy and path diversity

ƒ High capacity and density

scalability

ƒ Redundancy and path diversity

ƒ High capacity and density

ƒ High performance and high

bandwidth with growth factors

incorporated

ƒ High performance and high

bandwidth with growth factors

incorporated

High capacity and density

ƒ Efficient allocation of space

ƒ Proper racking, enclosures,

High capacity and density

ƒ Efficient allocation of space

ƒ Proper racking, enclosures,

ƒ Support for 10G or higher

speed technologies

ƒ Support for storage devices (ie

ƒ Support for 10G or higher

speed technologies

ƒ Support for storage devices (ie

pathways and access flooring

ƒ Incorporation of data center security and monitoring

pathways and access flooring

ƒ Incorporation of data center security and monitoring

ƒ Support for storage devices (ie

fiber channel, SCSI or NAS)

ƒ Support for convergence with

ƒ Support for storage devices (ie

fiber channel, SCSI or NAS)

ƒ Support for convergence with

growth factors incorporated

ƒ Support for UIO / UF

growth factors incorporated

ƒ Support for UIO / UF

ƒ Initial investment protection

ƒ Initial investment protection

Media Options for 1GE – High Speed

Ethernet Migration

10 Gigabit Transmissions

ƒ Different Standards

10GBase-T (IEEE 802.3an) 10GBase-CX4 (IEEE 802.3ak) 10GB R (IEEE 802 3 ) 10GBase-R (IEEE 802.3xx) LRM (802.3aq)

LR, ER, SR (802.3ae) SFF 8431 (SFP+ Fiber & cu)

ƒ Applications

Server Interconnects Aggregation of Network Links

14

Switch to Switch Links Storage Area Networks (SAN)

10GBase T

ƒ 100m on Class EA/Category 6 augmented 100m on Class EA/Category 6 augmented

copper cabling

ƒ Cat 7 insertion loss characteristics Cat 7 insertion loss characteristics

Twisted Pair Cabling For 10GBASE-T

(IEEE 802.3an)

(IEEE 802.3an)

16

10GBASE-T: Alien X-Talk (AXT)

ƒ Main Electrical Parameter Limiting 10GBASE-T ; Every

Cable vendor has solutions to mitigate g

ƒ Alien crosstalk is defined as “Unwanted signal coupling

from one balanced twisted-pair component, channel, or

permanent link to another” Unwanted coupling of signals

between adjacent signals

ƒ Cannot be cancelled

ƒ High crosstalk levels can compromise the operation of the

10GBASE-T application by significantly reducing expected

signal-to-noise (SNR) margins, thus potentially causing

re-transmissions or even auto-negotiation of the switch

to a lower Ethernet speed

ƒ Category 6/class E UTP g y

No added protection from AXT

ƒ Category 6A/class EA UTP

Greatly increased outside jacket wall thickness (increased cable diameter up to 9 0 mm/0 354 in ) provides some degree of

Can be prevented or mitigated by:

space (Cat6a solution)

hi ld (C 6/C 6 /C

diameter up to 9.0 mm/0.354 in.) provides some degree of protection from AXT but is very much dependent of Installation and maintenance practices (excessive pathway fill and over-cinched tie wraps would compromise AXT performance).

shield (Cat6/Ca6a/Cat7 shielded solutions)

Cable Sizing At-a-Glance

18

Shielded (F/UTP) vs Unshielded

(6a U/UTP)

ƒ Alien XT reduction w/ shielding

ƒ Diameter 0.3in/7.6mm

ƒ Up to* 40% cable conduit fill-ratio

ƒ Alien XT reduction w/ spacing

ƒ Diameter 0.354in/9mm (worst case)

ƒ Up to 40% cable conduit fill-ratio

higher than U/UTP

ƒ Cabling choice in Cisco’s new

Richardson Data Center

ƒ Stiffer/less flexible cable than F/UTP

*Assumption of 40% conduit fill ratio and various conduit sizes (Source Tyco)

10G Copper Infiniband - 10GBase-CX4

10G Copper on Twin Axial copper

10G Copper on Twin Axial copper

20

10G SPF+ Cu

to 7 meters

1GE-10GE Transceiver Perfomance

Mid to End of

22

of Rack

<100 M

of Rack

<100 M

In Rack X-rack

<10M

In Rack X-rack

<10M

Across Aisles

<300 M

Across Aisles

<300 M

Across Sites

<10 KM

Across Sites

<10 KM

10GE (IEE 802.3ae) Optical Transmission

100Mb/s OM1 OM1 OM1

1,000Mb/s OM1 OM2 OS1

OM1 is equivalent to standard 62.5/125µm MM fiber OM2 is equivalent to standard 50/125µm fiber

10Gb/s OM3* OS1 OS1

OM3 is laser enhanced 50/125µm fiber – 10gig OS1 is equivalent to SM 8/125µm fiber.

* This refers to 10GBASE-SR, for LX4 & LRM you get 300 and 220m respectively irrespective of fiber type

Not all laser optimized 10Gig

fiber cable is the same.

Parameters Affecting Optical Channel

Performance

ƒ Transmitter:

O ti l M d l ti A lit d (OMA) Optical Modulation Amplitude (OMA) Center wavelength

Spectral width Jitter

ƒ Receiver:

Sensitivity Relative Intensity Noise (RIN) Jitter

ƒ Fiber:

Inter-modal dispersion Chromatic dispersion

Cost Effective 10G Server Connectivity

ƒ Not a standard.

ƒ 0.1W Power

26

Introducing Energy Efficient Ethernet

IEEE 802.3az

ƒ EEE is a method to facilitate transition to and from lower power consumption in response to

changes in network demand g

ƒ In the process of being defined for these copper PHYs

100BASE-TX (Full Duplex) 1000BASE-T (Full Duplex) 10GBASE T

10GBASE-KR 10GBASE-KX4

ƒ Uses Low Power Idle (LPI) to save energy

Concept: transmit data as fast as possible, return to Low-Power Idle Saves energy by cycling between Active and Low Power Idle

Power reduced by turning off unused circuits during LPI Energy use scales with bandwidth utilization

ƒ EEE is an Energy Star requirement for PCs in 2010

ƒ EEE uses autonegotiation to notify partner of EEE capabilities.

ƒ EEE uses LLDP to notify link partner of parameter changes y p p g

(e.g control policy: energy savings over performance mode or vice versa)

as video-on-demand

ƒ Defined Channel Reach: SMF: 10 km (40G/100G), 40 km (100G), OM3: 100 m (40G/100G),

Twinax: 10m , Backplane: 1m.

28

High Speed Ethernet Standard Interfaces

40G/100G Form Factor Landscape

CFP

<2X XENPAK Size

QSFP

Small Footprint Power: ~24W ( 40G Target:8W)

MPO and Duplex Supported

Max Density: 4 – 6 /linecard

Power: ~3.5W Needs Fiber Array Cable (MPO or MPC)

30

40GBASE-SR4 (100M on OM3)

40GBASE-CR4 (10M Twinax)

100G BASE-CR10 (10M Twinax)

40G/100G (IEEE 802.3ba) MM

Cable Connector

connectors on both ends

20 fibers for 100G

100G (IEE 802.3ba) SMF Using WDM

EML EML

100G (IEE 802.3ba) SMF Using WDM

Serializer

EML EML EML

4x1 WDM MUX

Driver Driver Driver

25G 25G

TEC

10x10G

LC Connector SMF A

SOA

P A

PIN PIN

TIA

WDM 4:10

Optional for 40km

SMF Pre Amp

PIN PIN

TIA TIA

DeMUX Deserializer

40Gb/s = 4 x 10Gb/s

32

Data Center Access Cabling Models

Modular Design, PODs, EoR, ToR, MoR, DAF

34

Data Center Trends and Modular

DC Design…

ƒ Move toward Pod based architectures that allow for scalability and easy of duplication leading to higher density but smaller lengths

easy of duplication leading to higher density but smaller lengths

within pods - Defines a discrete amount of physical infrastructure

Racks + Power Distribution + CRACs

ƒ Increased cable media options to meet the needs of

new architectures

ƒ “Pay-as-you-grow” modularity - Predictable, Scalable & Flexible

ƒ Tiered DC architectures (Access, Core, Aggregation) - with mixed

fabrics and protocols(Ethernet, Fibre Chan & FCoE)

ƒ Further drive to “Green” data centers

ƒ Further drive to Green data centers

Lower power cabling solutions (CX1 and Fiber) Drive towards pre-terminated cabling solutions

ƒ Requests for 40/100 Gbits to meet backbone applications of 10 GE

Mapping Modular Design to the

Physical Infrastructure

Physical Infrastructure

SAN Fabric

Storage Arrays

LAN Fabric Core

POD: Modular Repeatable Compute Environment w/ Predictable Scalability &

Core

Aggregation

SAN Edge B SAN Edge A

blade1 blade6

slot 1 slot 6

blade1 blade7

slot 1 slot 7

blade1 blade6

slot 1 slot 7

blade1 blade6

slot 1 slot 6

blade1 blade7

slot 1 slot 7

blade1 blade6

slot 1 slot 7

blade1 blade6

slot 1 slot 6

blade1 blade7

slot 1 slot 7

blade1 blade6

slot 1 slot 7

blade1 blade6

slot 1 slot 6

blade1 blade7

slot 1 slot 7

blade1 blade6

Access

Virtual Access

General Purpose POD

™ Typical enterprise application

environment

ƒ Classic client server applications

Unified Compute Pod

™ Collection of Unified Compute Arrays

ƒ IO Consolidation

W kl d M bilit

ƒ Classic client server applications

ƒ Multi-tier Applications: web, app, DB

ƒ Low to High Density Compute Environments

ƒ Include stateful services

ƒ Workload Mobility

ƒ Application Flexibility

ƒ Virtualization

36

The POD Concept: applies to distinct application environments and through a modular

approach to building the physical, network and compute infrastructure in a predictable

and repeatable manner It allows organizations to plan the rollout of distinct compute

environment as needed in a shared physical data center using a pay as you go model

The POD Concept: applies to distinct application environments and through a modular

approach to building the physical, network and compute infrastructure in a predictable

and repeatable manner It allows organizations to plan the rollout of distinct compute

environment as needed in a shared physical data center using a pay as you go model

The Pod Concept

A Modular Approach to DC Build-Outs

™ Build as you grow model

™ CAPEX and OPEX advantages

™ Predictable Infrastructure Characteristics

™ Predictable Network Scalability

™ Deterministic Functions

™ Repeatable

ƒ Pod

™ Eases adoption cycles

™ Adaptable to Brownfield and Greenfield

Physical Infrastructure and Network Topology

Mapping the Physical to the Logical

Zone

DC

COLD AISLE

HOT AISLE POD

DAF Rack

ToR POD UCS Blade Rack

POD POD

38

EoR Access

POD

DAF UCS POD

slot 1 slot 6

blade1 blade7

slot 1 slot 7

blade1 blade6

slot 1 slot 6

blade1 blade7

slot 1 slot 6

blade1 blade7

slot 1 slot 7

blade1 blade6

slot 1 slot 6

blade1 blade7

slot 1 slot 7

blade1 blade6

slot 1 slot 6

blade1 blade7

slot 1 slot 7

blade1 blade6

slot 1 slot 7

blade1 blade6

slot 1 slot 6

blade1 blade7

slot 1 slot 7

blade1 blade6

Access Layer Network Model

End of Row Top of Rack and Blade Switches

What it used to be…

GE Access

Access Model

Port Density

What Cisco has done…

Nexus 2K+Nexus 5k Future 7K & 6K

Unified Compute System

Network Equipment Distribution

Top of Rack

ToR

ƒ Used in conjunction with dense access racks

ƒ Used in conjunction with dense access racks

(1U servers)

ƒ Dense server counts

ƒ Copper cabling typically stays within the rack

ƒ Typically one access or two switch per rack

Patch panel server

X-connect

Patch panel

Top of Rack Top of Rack

Typically one access or two switch per rack

ƒ Top Rack switch may be outside the rack

Cabling:

ƒ Within rack: Copper for server to access switch

Network Aggregation Point

A - B

server

Network Aggregation Point

A - B

server

ƒ Copper in rack options CX1, Cat 5/6/6a

ƒ Outside rack (uplink):

ƒ Fiber (GE or 10GE): requires aggregation

model (MoR / EoR)

ƒ East-West traffic to aggregation more common

Subnets and VLANS:

ƒ One or many subnets per access switch

ƒ Subnets tent to be small

A - B

Network Access Point

C - D

Network Equipment Distribution

End of Row or Middle of Row

End of Row

End of Row

ƒ Poses challenges on highly dense server farms

Distance from farthest rack to access point Row length may not lend itself well to switch port density

Common Characteristics Network

Patch panel X-connect

Network

Patch panel X-connect

ƒ Typically used for modular access

ƒ Multiple Ge or 10Ge NICs

ƒ Cabling is done at DC build-out

ƒ Model evolving from EoR to MoR

Lower cabling distances (lower cost)

ƒ # servers dependent RU & Power Rack

ƒ Subnets and VLANs: one or many per switch Subnets tend

to be medium and large; Low STP logical port counts

Copper Middle of Row

Middle of Row (half row)

Access

ƒ Used to EoR cable distance challenges

ƒ Copper from servers to access switches

Patch panel server

X-connect Patch panel

Aggregation

ƒ Fiber may be used to aggregate ToR

ƒ It addresses aggregation requirements for ToR

access environments

Network Access Point

A - B

server

Network Access Point

C - D

server