Advanced Computer Architecture - Lecture 34: Multiprocessors

Advanced Computer Architecture - Lecture 34: Multiprocessors. This lecture will cover the following: shared memory architectures; parallel processing; parallel processing architectures; symmetric shared memory; distributed shared memory; performance of parallel architectures;...

CS 704

Advanced Computer Architecture

Lecture 34

Multiprocessors

(Shared Memory Architectures)

Prof Dr M Ashraf Chughtai

Today’s Topics

Recap:

Parallel Processing

Parallel Processing Architectures

Performance of Parallel Architectures

Summary

So far our focus have been to study the

performance of a single instruction stream computers; and methodologies to enhance the performance of such machines

We studied how

– the Instruction Level Parallelism is exploited

among the instructions of a stream; and

– the control, data and memory dependencies

are resolved

These characteristics are realized through:

– Pipelining the datapath

– Superscalar Architecture

– Very Long Instruction Word (VLIW)

Architecture

– Out-of-Order execution

Parallel Processing and Parallel

Architecture

However, further improvements in the

performance may be achieved by exploiting parallelism among multiple instruction

streams, which uses:

– Multithreading, i.e., number of instruction

streams running on one CPU

– Multiprocessing, i.e., streams running on

multiple CPUs where each CPU can itself

be multithreaded

Parallel Computers Performance

Amdahl’s Law

Furthermore, while evaluating the

performance enhancement due to parallel processing two important challenges are to

be taken into consideration

– Limited parallelism available in program – High cost of communication

These limitations make it difficult to

achieve good speedup in any parallel

processor

Parallel Computers Performance

Amdahl’s Law

For example, if a portion of the program is

sequential, it limits the speedup; this can be

understood by the following example:

Example: What fraction of original computation can be sequential to achieve speedup of 80 with

Parallel Computers Performance

Here, the fraction enhanced is the fraction in

parallel, therefore speedup can be expressed as

80 = 1 / [(Fraction parallel /100 + (1-Fraction parallel )]

Simplifying the expression, we get

0.8*Fraction parallel + 80*(1-Fraction parallel ) = 1

80 - 79.2* Fraction parallel = 1 Fraction parallel = (80-1)/79.2 = 0.9975

i.e., to achieve speedup of 80 with 100 processors only 0.25% sequential allowed!

Parallel Computers Performance

processing is the communication cost that involves the latency of remote access

Now let us consider another example to

explain the impact of communication cost

on the performance of parallel computers

on 32-processors multiprocessor, with 40 nsec time to handle remote memory

reference

Parallel Computers Performance

reference hit is 2 and processor clock rate is 1GHz, and find:

How fast is the multiprocessor when there

is no communication versus 0.2% of the

instructions involve remote access?

multiprocessor with remote reference is:

Remote request rate x remote access cost

Introduction to Parallel Processing

Substituting the values we get:

0.2% x remote request cost

= [1/2] + 0.2% x (400 cycle)

And, CPI without remote reference

Hence, the multiprocessor with all local

reference is 1.3/0.5 = 2.6 times faster as

Introduction to Parallel Processing

Considering these limitations let us

explore how improvement in computer performance can be accomplished

using Parallel Processing Architecture

Parallel Architecture is a collection of processing elements that cooperate

and communicate to solve larger

problems fast

Introduction to Parallel Processing

Parallel Computers extend the

traditional computer architecture with a communication architecture to achieve synchronization between threads and consistency of data in cache

Parallel Computer Categories

In 1966, Flynn proposed simple categorization

of computers that is still valid today

This categorization forms the basis to

implement the programming and

communication models for parallel computing

Flynn looked at the parallelism in the

instruction and in the data streams called for

by the instructions and proposed the following four categories:

Parallel Computer Categories

SISD (Single Instruction Single Data)

– This category is Uniprocessor

SIMD (Single Instruction Multiple Data)

– Same instruction is executed by multiple

processors using different data streams

– Each processor has its own data memory

(i.e., multiple data memories) but there is a

single instruction memory and single control processor

Parallel Computer Categories

– Illiac-IV and CM-2 are the typical examples of

SIMD architecture, which offer:

Simple programming model Low overhead

Flexibility All custom integrated circuits

MISD (Multiple Instruction Single Data)

– Multiple processors or consecutive functional

units are working on a single data stream

– (However, no commercial multiprocessor of this

Parallel Computer Categories

MIMD (Multiple Instruction Multiple Data)

– Each processor fetches its own instructions and

operates on its own data

Origin The characteristics these machines are:

Flexibility: it can function as Single-user multiprocessor or as multi-programmed multiprocessor running many programs simultaneously

Use of off-the-shelf microprocessors

MIMD and Thread Level Parallelism

MIMD machines have multiple processors and can be used as:

– Either each processor executing different

processes in a multi-program environment

– Or multiple processors execute a single

program sharing the code and most of their address space

In the later case, where multiple processes

share code and data, such processes are

referred to as the threads

MIMD and Thread Level Parallelism

Threads may be

– either large-scale independent

processes, such as independent

programs, running in multi-programmed fashion

– Or parallel iterations of a loops having

thousands of instructions, automatically generated by a compiler

This parallelism in the threads in called

Thread Level Parallelism

MIMD Classification

Based on the memory organization and

interconnect strategy, the MIMD machines are classified as:

Centralized Shared Memory

Architecture

Distributed Memory Architecture

Centralized Shared-Memory

The Centralized Shared Memory design,

shown here, illustrates the interconnection

of main memory and I/O systems to the

processor-cache subsystems

In small-level designs, with less than a

dozens processor-cache, subsystems

share the same physical centralized

memory connected by a bus; while

In larger designs, i.e., the designs with a few …

Centralized Share-Memory

Architecture

Centralized Shared-Memory

… dozens processor- cache subsystems, the single bus is replaced with multiple

buses or even a switch are used

However, the key architectural property of the Centralized Shared Memory design is the Uniform Memory Access – UMA;

i.e., the access time to all memory from all the processors is same

the single

Centralized Shared-Memory

Furthermore, the single main memory has a symmetric relationship to all the processors

These multiprocessors, therefore are

referred to as the Symmetric (Shared

Memory) Multi-Processors (SMP)

This style of architecture is also sometimes called the Uniform Memory Access (UMA)

as it offers uniform access time to all the

memory from all the processors

Decentralized or Distributed Memory

The decentralized or distributed memory

design style of multiprocessor architecture

is shown here

It consists of number of individual nodes

containing a processors, some memory and I/O and an interface to an interconnection

network that connects all the nodes

The individual nodes contain a small

number of processors which may be ……

Decentralized Memory Architecture

Decentralized or Distributed Memory

interconnected by a small bus or a different interconnection technology

Furthermore, it is a cost effective way to

scale the memory bandwidth if most of the accesses are to the local memory in the

node

Thus, the distributed memory provides

more memory bandwidth and lower memory latency

Decentralized or Distributed Memory

This makes the design more attractive for small number of processors

The disadvantage of the distributed

Parallel Architecture Issues

While studying parallel architecture we will be

considering the following fundamental issues that characterize parallel machines:

– How large is a collection of processor?

– How powerful are processing elements?

– How do they cooperate and communicate? – How are data transmitted ?

– What type of interconnection?

– What are HW and SW primitives for

programmer? and

Issues of Parallel Machines

These issues can be classified as:

1) Naming

2) Synchronization

3) Latency and Bandwidth

Fundamental Issue #1: Naming

Naming deals with:

– how to solve large problem fast?

– what data is shared?

– how it is addressed?

– what operations can access data? – how processes refer to each other?

Fundamental Issue #1: Naming

The segmented shared address space

locations are named uniformly for all

processes of the parallel program as:

<process number, address>

Choice of naming affects:

– code produced by a compiler as for

message passing via load, the compiler just remembers address or keep track of processor number and local virtual

address

MAC/VU-Advanced

Computer Architecture Lec 34 Multiprocessor (1) 32

Fundamental Issue #1: Naming

– replication of data, because in case of

cache memory hierarchy the replication

and consistency through load or via SW is affected by naming

– Global Physical and Virtual address

space, as naming determines if the

address space of each process can be

configured to contain all shared data of

the parallel program

Issue #2: Synchronization

In parallel machines to achieve

synchronization between two processes,

the processes must coordinate:

– Message passing implicitly coordinates

with transmission or arrival of data

– Shared addresses explicitly coordinate

through additional operations, e.g., write a flag, awaken a thread, interrupt a

processor

Issue #3: Latency and Bandwidth

Bandwidth

– Need high bandwidth in parallel

communication; however, bandwidth

cannot be scaled, but stays close to the requirements

– Match limits in network, memory, and

processor

– Overhead to communicate is a problem in

many machines

Issue #3: Latency and Bandwidth

Latency

– Affects performance, since

processor may have to wait

– Affects ease of programming, since

requires more thought to overlap

communication and computation

Issue #3: Latency and Bandwidth

Latency Hiding

– As the latency increases the

programming system burden,

therefore mechanisms are found to help hide latency

Framework for Parallel processing

The framework for parallel architecture

is defined as a two layer representation

These layers define Programming and Communication Models

These models present sharing of

address space and message passing

in parallel architecture

Framework for Parallel processing

The shared address space model at:

– The communication layer, defines the

communication via memory to handle load, store, and etc.; and at

– The programming layer, it defines

handling several processors operating

on several data sets simultaneously;

and to exchange information globally and simultaneously

Framework for Parallel processing

Message passing model at the:

– communication layer defines sending

and receiving messages and library

calls; and at the

– programming layer provides a

multiprogramming model to conduct lots of jobs without I/O

communication simultaneously

Shared Address Space Architecture

(for Decentralized Memory Architecture)

Shared Address space is referred to as the Distributed Shared Memory – DSM

where, each processor can name every physical location in the machine; and

each process can name all data it

shares with other processes

Data transfer takes place via load and store

Shared Address Space Architecture

(for Decentralized Memory Architecture)

Data size is defined as: byte, word, or

Multiple private address spaces offer

message passing multicomputer via

separate address space

Shared Address Space Architecture

Programming Model

Virtual address spaces for a collect-ion

of ‘n+1’ processes communicating via

shared addresses

Machine physical address

space

Shared Address Space Architecture

Programming Model

The programming model defines how

to share code, private stack, some

shared heap, some private heap

Here, Process is defined as virtual

address space plus one or more

threads of control

Shared Address Space Architecture

Programming Model Multiple processes can overlap, but ALL

threads share a process address space, i.e., portions of address spaces of processes

are shared; and

writes to shared address space by one

thread are visible to reads of all threads in other processes as well

There exist number of shared address

space architectures

Shared Address Space Architecture

Programming Model

The most popular architectures are:

– Main Frame Computers

– Minicomputers – Symmetric Multi

Processors (SMP)

– Dance Hall

– Distributed Memory – Non-Uniform

Multiprocessor Architecture (NUMA)

1: Main Frame Architecture –

Shared Address Space Architecture

1: Main Frame Architecture Cont’d

The main frame architecture was motivated

by multiprogramming

As shown here, it extends crossbar for

processor interface to memory modules and I/O

Initially this architecture was limited by

processor cost; but, later by the cost of

crossbar

IBM S/390 (now z-Server) is typical example

of cross-bar architecture

2: Minicomputers (SMP) Architecture

The minicomputer architecture was

also motivated by multiprogramming

and multi-transaction processing

As shown here, as all the components are on shared bus and all memory

locations have equal access time so

this architecture is referred to as the

Symmetric Multi Processors (SMPs)

2: Minicomputers (SMP) Architecture

2: Minicomputers (SMP) Architecture

However, the bus is bandwidth bottleneck

as sharing is limited by Bandwidth when we add processors, I/O

Furthermore, caching is key to the

coherence problem – we will talk about this later

The typical example of SMP architecture is Intl Pentium Pro Quad

Intel Pentium Pro Quad

Here, all the coherence and multi-processing is glued in processor module

glued in processor module

It is highly integrated and have low latency and bandwidth

3: Dance Hall Architecture

All processors are on one side of the network and all memories on the other side

3: Dance Hall Architecture

As we have notice that in the cross-bar

architecture the major cost is of

interconnect and in SMPs bus bandwidth is the bottleneck

This architecture offers solution to both the problems through its scalable interconnect network where the bandwidth is scalable

However, the interconnect network has

larger access latency; and

caching is key to the coherence problem

4: Distribute Memory Architecture

It is a large scale multiprocessor

architecture where Memory is distributed with Non-Uniform Access Time

This architecture is therefore referred to as Non Uniform Memory Access (NUMA)

4: Distribute Memory Architecture

Cray T3E is a typical example of NUMA

architecture

4: Distribute Memory Architecture

Cray T3E is a typical example of NUMA

architecture which scales up to 1024

processors with 480 MB/sec links

Here, the non-local references are accessed using communication requests generated

automatically by the memory controller in the external I/Os

Here no hardware coherence mechanism is employed rather directory based cache-

coherence protocols are used – We will

discuss this in detail later