Advanced Computer Architecture - Lecture 42: Networks and clusters

Advanced Computer Architecture - Lecture 42: Networks and clusters. This lecture will cover the following: multistage interconnect network; switch topologies with centralized switch topology, distributed switch topology; cluster; tree network topology; hypercube network topology;...

CS 704

Advanced Computer Architecture

Lecture 42

Networks and Clusters

(Networks Topology and Internetworking Cont’d)

Prof Dr M Ashraf Chughtai

Today’s Topics

Recap:

Switch Topologies Cont’d

 Centralized Switch Topology

Cluster

Summary

Recap: Lecture 41

Last time we discussed:

 The formation of generic interconnection

networks and their categorization;

 The networks communication model,

performance, media, software, protocols,

subnet and networks topologies

Here, we noticed that a generic interconnection network comprises:

The interconnections are classified based

on the number of processors or nodes and the distance between them as:

– Local Area Network-LAN

– Wide Area Network-WAN

– System Area Network-SAN

Recap: Lecture 41

The interconnect communication model

shows that two machines are connected via

two unidirectional wires with a FIFO (queue) at the end to hold the data

The communication software separates the

header and trailer from the message and

identifies the request, reply, their

acknowledgments and error checking codes

The communication protocols suggest the

sequence of steps to reliable communication

Recap: Lecture 41

The network performance that defines the

latency of the message as the sum of the:

Sender overhead, time to flight, receiver

overhead and the ratio of the message size to the bandwidth

We also discussed the properties and

performance of interconnect network media or link – the unshielded twisted pair (UTP), coaxial cable and fiber optics

However, the interconnect sharing media are challenging as it requires coordination and …

Recap: Lecture 41

… arbitration when more than one computer needs the same media simultaneously

Alternative to sharing media is to use a switch

to provide a dedicated line to all destinations

in order; and facilitates point-to-point

communication much faster than the shared media

A switch provides unidirectional

inter-connection of input to any one of multiple

output terminals

The Crossbar switch is typical example of

non-blocking switch; an is employed in the centralized switching topology

Last time we discussed the crossbar topology in detail and noticed that a crossbar uses n 2

switches to interconnect n processors in a

network

Recap: Lecture 41

Here the routing, to establish interconnection between two node at a time, depends on the addressing style

i.e., source-based routing where message

specifies the path to the destination or

destination-based routing where the message simply contains the destination address and a program running in the switch selects the port

to take for a given destination

Multistage Interconnect Network

Today, continuing our discussion on the

centralized switching topologies , we will

discuss an intermediate class of network

interconnect which lies between crossbar and bus-based networks

This interconnect topology is referred to as the

Multistage network topology

A centralized multistage network, shown here,

is built from number of large switch boxes ,

placed at multiple stages to interconnect all of the nodes

Multistage Interconnection Topology

Each stage contains number of

small crossbar switches and

allows the straight or cross

connections through the

switch, as shown

Multistage Interconnection Topology

The number of stages are related to the

number of nodes and the size of the crossbar switch

Consequently, its performance and cost are

more scalable than bus-based networks

The number of identical stages (N s ) in the

network having n nodes and switches of size m

x m, in each stage, is given as:

Ns = log m n

Multistage Interconnection Topology

And, the number of switches per stage is n/m

Thus, the total number of switches used in multistage network of n nodes is n/m log m n

i.e., its cost is

O(n log n) as compared O(n 2 ) for crossbar

To understand the design and working of

multistage networks, let us consider Omega Network, depicted here, as a typical

implementation of multistage network

010 011

100 101

110 111

000 001

010 011

100 101

110 111

number of identical stages [log 2 8] = 3

And, switches per stage [n/m] = 8/2 =4

Omega Topology: Multistage Interconnect

 let us see how the switches at each stage

operate to establish connection

 Note that for the 8-nodes Omega Network the node address is of 3 bits, which is equal to

number of stages of the switch

000 001 010 011 100 101 110 111

000 001 010 011 100 101 110 111

Omega Network: Example

Here, the 3-bit code a 2 a 1 a 0 represents 3 stages

of the network, as stage S 2 S 1 S 0 , from left to

right

To find the connection pattern XOR the source and destination, e.g.,

 Src (010)  dest (110) then XOR results

 100  Cross (S2) Straight (S1) Straight (S0)

The switch connections are shown Green

Circles

Omega Network: Example

Thus, the generalized rule to find the switch connection can be summarized as

 For the stage i

IF the source and destination differ in ith bit

THEN connection Cross the switch in

the i th stage”

ELSE Connection is Straight in the i th stage”

Characteristics of Omega

There exist an single path from source to

destination, thus contrary to the non-blocking crossbar network, the omega network is

blocking network

This is shown here as:

- the path 010  110 (red) and

- the path 110  100 (blue)

have blockage as the S2 for 110 has to wait till

010 has passed otherwise it results in

collision

Omega Network Characteristics … Cont’d

However, in order to minimize collisions and to improve fault tolerance to achieve high

reliability and dependability extra pathways

can be added

000 001 010 011 100 101 110 111

000 001 010 011 100 101 110 111

000 001 010 011 100 101 110 111

000 001 010 011 100 101 110 111

Distributed Switch Networks

So far we have been discussing the Centralized switching topologies

The distributed switching network is one where the switches are distributed throughout the

network and they allow interconnection of one node to:

 either all the nodes

 or to a limited number of nodes A

B

C

Distributed Switch Networks Cont’d

A network where each

node interconnects all

nodes of the network is

called, Fully connected

network

B

D

There exist different interconnects for

distributed switch networks

Before discussing these interconnects, let us understand the parameters of interconnect

performance measure

Interconnect Performance Measure

Criteria

Latency: Number of Links and must be small

Bandwidth: The number of messages or the

length of massages; it should be large

Node Degree : Number of links connected to a node

Diameter: Maximum distance between any two processors, i.e., the number of nodes between source and destination; this is in deed the

measure of maximum latency

Performance Measure Criteria … Cont’d

Bisect: The imaginary line that divides the

interconnect into roughly two equal parts, each having half the nodes

Bisection Bandwidth: Sum of the bandwidth of lines crossing the imaginary bisection line

It measures the volume of communication

allowed between any two halves of network

with equal number of nodes

Parameters of Interconnect Performance

Measure

Distributed Switch Topologies

Based on the concept of distributed-switch interconnects, there exist numerous

Linear Array / Ring

The simplest possible, low cost distributed

switch network topology, is a linear array and

ring network

As shown here, the Linear Array networks is one where a small switch is placed at every node (processor)

Linear Array / Ring

The switch at the i th node connects the i th node

to the:

 (i-1) th node except for i=1, and

 (i+1) th node except for i =n

In the linear array, as the i th node is connected

to (i-1) th and (i+1) th node, therefore

the message will have to hop along

intermediate node until it arrives at the final

destination at (i ± m) where m>1

Linear Array / Ring

For example, where the message is to pass from 1 st node is to the 4 th node, it hops the 2 nd

and 3 rd nodes

The ring network is established by

establishing an interconnect between the 1 st

and the n th nodes in the linear array network

Ring /Token Ring

Like linear array, in Ring network some

massages hop along the intermediate nodes until they reach destination

However, it allows many transfers

simultaneously; the 1 st node can send to the

2 nd at the same time as the 3 rd can send to 4 th

and so on.

A variation called Token Ring is used in the

Ring Network, to simplify the arbitration in the ring topology

Ring Network

Here, a single slot (token) goes around the ring

to determine which node is allowed to send the message – a node can send a message if it

gets a token

The common performance of an n - node linear array and ring network are as follows:

 Cost: Cheap as the cost is O(n)

 Bandwidth: Overall bandwidth is high

 Latency: High as it is of O(N)

Performance: Array verses Ring

Fully Connected

B

D

Fully Connected: Performance Metrics

B

D

2D Mesh and 2D Torus

Two dimensional Mesh or Grid is an example of asymmetric network topology and uses

bisection bandwidth as the performance metric

Here, the nodes (processors) are arranged in a array structure forming 2D Grid or Mesh

An example 3x4 mesh

structure is shown here P P P P

2D Mesh and 2D Torus

A switch is associated to each (processor)

node [shown as blue circle]

Each switch has one port for the processor

and four ports to interconnect the processor to the four nearest-neighbor nodes , i.e., the nodes

to the left - right and up - down position

This structure is sometimes also referred to as

NEWS communication pattern , representing

North, East, West and South communication

2D Mesh and 2D Torus

Note that here the switches associated with the

top/bottom rows or left/right columns don’t

connect among themselves, thus have unused ports

Connecting the unused

ports of switches of the

top/bottom rows and the

left/right columns forms 2D

Torus , using wraparound

links, as shown here

2D Mesh / Torus: Performance Metrics

The performance metrics of n-node 2D Mesh / Torus are as follows

Tree Network Topology

Another example of distributed switch network

is the Tree Topology

Here, the switches associated with each node have the number of ports equal to the number

of braches of the tree plus one for the

processor

A Binary Tree structure

shown here has two

branches of the root node

and branch nodes

Tree Network: Performance Metrics

The performance metrics of N-nodes Tree Network are as follows

Cost: It is cheap as cost

is O(N) Degree: Number of

branches 1, 2, 3 … Latency:O(log deg N)

Diameter: 2log deg N

Bisection Width: 1

Tree Network: Bottlenecks

The root node and the branch nodes of the leaf-nodes are the bottleneck

For example, leaf-nodes

1, 2 of the branch nodes 9

and 3,4 of branch node

10, may be

interconnect-ed simultaneously, but

the leaf-nodes 1,3 and 2,4

cannot, as the there may

be collision at branch

nodes 13, 9 and 10 1 2 3 4 5 6 7 8

14 13

15

Fat Tree Network

To avoid root being the bottleneck, multiple

paths are provided between any two nodes, as shown here

This structure is called the Fat Trees

Fat Tree Network

Here, the black dots show the

processor-memory nodes connected through the

multiple stages of 2x2 crossbar, 4x (2+2)

crossbar and 8 x (4+4) crossbars switches and

so on

This 3D switching increase the bandwidth via extra links at each level over the simple tree

In CM-5 the concept of Fat Tree is used as

Centralized switching Network

Hypercube Network Topology

Another example of distributed switch network

is the Hypercube topology which is also called

binary n-cubes, as it has 2 nodes of n-cubes

It is an n-dimensional interconnect for 2 n nodes

As can be seen from the figure here, that for 16 nodes; the hyper cube is a 4D structure as

N=16 = 2 4 therefore n=4

Hypercube Network Topology

It requires n ports per switch plus one for the processor this have n nearest neighbors nodes Thus, it minimizes hops and have latency of

O(log 2 N); the other performance metrics are:

Hypercube Network Topology

Note that the bisection bandwidth is good but it

is difficult to layout in 3D space

Hypercube has been popular in early message passing machines, e.g., Intel iPSC, NCUBE etc

K-ary n-cube Network Topology

Rather than having just 2 nodes of n-cubes in the binary hypercube, the generalization of

hypercube is to interconnect k nodes of

n-cubes in a string

The total number of nodes: N= k n

A 64 node, where 64 = 4 3 [4 ary 3 cube)

structure is shown here

This structure allows for wider channel but

requires more hops

K-ary n-cube Network Topology

64 = 4 3 [4-ary 3-cube) (3 cube is a 16 nodes binary hypercube)

Comparing Network Topologies

The relative cost and performance of

topologies discussed, based on the bisection bandwidth and number of links for 64 nodes network is given in the table here

Evaluation Bus Ring 2D Torus Fully

Comparing Network Topologies

Here, bus is used as the standard reference at unit cost, all transfers are done by taking the time units equal to the number of messages

Where as the fully connected network has all nodes at equal distance therefore the number

of links and ports per switch are maximum and all transfers are done in parallel taking only

unit time

The nodes for ring topology are differing

distances

Comparing Network Topologies

Here, bus is used as the standard reference at unit cost, all transfers are done by taking the time units equal to the number of messages

Where as the fully connected network has all nodes at equal distance therefore the number

of links and ports per switch are maximum and all transfers are done in parallel taking only

unit time

The nodes for ring topology are differing

distances

Internetworking deals with the communication

of computers on independent and incompatible networks reliably and efficiently

The software standards are the basic enabling technologies of internetworking

(Transmission Control Protocol/Internet

Protocol) TCP/IT is the most popular

internetworking standard

The detailed discussion on Internetworking is beyond the scope of this course

Internetworking deals with the communication

of computers on independent and incompatible networks reliably and efficiently

The software standards are the basic enabling technologies of internetworking

(Transmission Control Protocol/Internet

Protocol) TCP/IT is the most popular

internetworking standard

The detailed discussion on Internetworking is beyond the scope of this course

Thanks

and Allah Hafiz

Today, we discussed an intermediate class of network interconnect which lies between

crossbar and bus-based networks, referred to

as the Multistage Switch network topology

A multistage centralized switch is built from

number of large switch boxes , placed at

number of stages to interconnect all of the

nodes

Here, The number of identical stages (N s ) in the network having n nodes and switch es in each stage are of MAC/VU-Advanced size m x m is as given as:

Computer Architecture Lecture 42 Networks and Clusters (2) 56