A realtime software solution for resynchronizing filtered MPEG2

MPEG2 sys-temlayerstream,especiallyTransportLayerStream,isadopted widely for video distribution applications like digital TV broadcast, since it provides many realtime functionalities su

A Realtime Software Solution for Resynchronizing Filtered

[Extended Abstract]

Bin Yu, Klara Nahrstedt Department of Computer Science University of Illinois at Urbana-Champaign

binyu,klara@cs.uiuc.edu

ABSTRACT

Withtheincreasing demandandpopularity ofmultimedia

streamingapplicationso erthecurrentInternet,

manipulat-ing MPEG streamsina realtime software manner is

gain-ing more and more importance In this work, we studied

theresynchronization problem thatarises when a gatewa

changes the datacontent carried inan MPEG2Transport

stream Inshort,thedistancebetweenoriginaltimestamps

is changed non-uniformly, and decoders willfail to

recon-structtheencodingclockfromtheresultingstream.We

pro-poseacheapsoftwarerealtimeapproachtosolvethis

prob-lem Experimental results from a realtime HDTV stream

ltershowsthatourapproachiscorrect andeÆcient

1 INTRODUCTION

Withtheincreasing demandandpopularity ofmultimedia

streamingapplicationsandtheever-growing scaleand

het-erogeneity ofthe Internet, adaptivecontentdelivery,

espe-ciallywith intermediaryproxysupporthas becomean

ex-tensivelystudiedarea ExamplesincludeProxyNet[1],IBM

Transcodingproxy[2],UC-BerkeleyTranSend[3]and

Con-tentServiceNetwork[4] One typicalservice theseproxies

mayprovideisto"lter"themediastream,thatis,to

trans-formitby changingthecontent oraddingnewvaluestoit

sothattheresultingstreamisbettermatchedwithresource

a ailabilityand userpreference Example lter operations

includeFrameSize Reduction, LowPass Filtering[5],and

InformationEmbedding[6]

Atthesametime,MPEG2[7]hasbecomethemostaccepted

standardforvideostorage andtransmission MPEG2

sys-temlayerstream,especiallyTransportLayerStream,isadopted

widely for video distribution applications like digital TV

broadcast, since it provides many realtime functionalities

suchasclocksynchronizationbetweenencoderanddecoder,

decoding/presentationtimestampingandmultiplexingof

mul-tiple streams with dierent time base However, when it

comes to HDTV [8, 9], its huge amount of data volumn

literallydisablesanysoftware encoding/decodingapproach

yet,asfarasweknow,almostallhardwareencoder/decoder

boards from the industry only dealwith Transport Layer

MPEG2format

Therefore,despitetheimportanceofMPEG2stream

lter-ing, notmucheort hasbeendone inthis area,and many problemsremaintobesolved.Inthispaper,wediscussone majorproblem that hasto be solvedbefore allfurther at-tempts canbe made To be more specic, we have found thateditingorlteringoftheHDTV(HighDenitionTV) streamwillcausetheaccessunitstodriftfromtheiroriginal timepointwithinthetimelinecarriedbythestream,which willhinder,orevendisablethedecoderfromdecoding and presentingthevideocontentinatimelymanner Ourgoalis

toidentifythecauseoftheproblemforMPEG2Transport streamandgiveaneÆcientrealtimesoftwaresolution

This paper is organized as follows: In section 2, we will introduce how the synchronization between MPEG encoderanddecoderworksandtheproblemthatarisesafter the ltering operations Oursolution is then discussedin detailinsection3and experimentresults follow insection

4 Finallyweconcludethispaperinsection5

2 THE SYNCHRONIZATION PROBLEM

Figure 1(on the next page)shows the MPEG2 Transport stream'smanagementtomaintainsynchronizationbetween the sender, which encodes the stream, and the receiver, whichdecodesit Astheelementarystreamscarryingvideo and audio content are packetized, their target Decoding Time Stamp (DTS) and Presentation Time Stamp (PTS) are determinedbased on the current sender clock and in-sertedintothepacketheaders Forvideostreams,theaccess unitisaframe,andbothDTSand PTSaregivenonlyfor the rstbitafterthe picture headerofeveryframe, which arelaterusedbythedecodertocontrolthespeedatwhich

itstartstododecodingandpresentation Forexample,ifat time5:0sanencodedframecomestothemultiplexingstage, and the encoder believesthat the decoder should beginto decodethatframe0:5safteritreceivesitandoutputthe de-codedframe0:1sthereafter,thentheDTSshouldbesetto 5:5sandthePTS5:6s.Afterthat,asallofthesepacketized elementarystreampacketsarefurthermultiplexedtogether, the nalstreamis time-stampedwith ProgramClock Ref-erence(PCR), which is given by periodically sampling the senderclock Thisresulttransportlayerstreamisthensent

o erthe network to the receiver, or stored in storage de-vicesforthedecoderto readinthefuture As longas the delay the wholestreamexperiences remainsconstant from the receiver's point ofview, the receivershould be ableto

streamwasencoded Theaccuracyandstabilityofthis

re-covered clock is veryimportant, since thedecoderwilltry

tomatchthePTSand DTSagainst thisclock toguide its

decodinganddisplayingactivities

Figure 2: MPEG2TransportStreamSyntax

Knowingthegeneralideaintiming,wenowintroducehow

theTransportLayersyntaxworksasshowninFigure2.All

substreams (video, audio, data and timestamps) are

seg-mentedintosmallpacketsofconstant size(188bytes),and

the Packet ID(PID) eld inthe 4-byteheader of a packet

tells which substream that packet belongs to The PCR

packets occur at constant intervals, and they form a

run-ning timelinealong which allotherpackets are positioned

at the right time point Data packets arrive and are read

into thedecoder buerat constant rate,and thisrate can

be calculatedby dividing the numberof bits between any

2consecutive PCRpacketsbythe timedierence between

theirtimestamps Inotherwords,ifthenumberofpackets

betweenany2PCRpacketsremainsconstant,thenthe

dif-ferencebetweentheirtimestampsshouldalsobeconstant

Inanideal state, packetsare read into thedecoderat the

constantbitrate,andwheneveranewPCRpacketarrives,

itstimestampshouldmatchexactlywiththereceiverclock,

constructedthe sameclockasthe encoder However, since PCRpacketsmayhaveexperiencedjitterinnetwork trans-missionorstoragedeviceaccessingbeforetheyarriveatthe receiver,wecannotsimplyset thereceiver'slocalclock to

bethesameasthetimestampcarriedbythenextincoming PCR nomatter when it comes To smoothout the jitter andmaintainastableclockwithalimitedbuersizeatthe receiver,generallythereceiverwillresorttosomesmoothing techniquelikethePhase-Locked-Loop(PLL)[10]togenerate

astableclockfromthejitteredPCRpackets.PLLisa feed-back loop that uses an externalsignal(the incomingPCR packetsinourcase)totunealocalsignalsource(generated

by a local oscillator in our case) to generate a relatively more stable result signal(the receiver'sreconstructed local clock inourcase) Solong asthe timing relationbetween PCRpacketsiscorrect,thejittercanbesmoothedoutwith PLL

After the brief introduction on the usage and importance

of the PCRpackets, nowwe are ready to discusshow the ltering operation may aect their validity and accuracy First,evenforthesametypeoflteringoperation,e.g.,low passltering, fordierent frames,the timerequired todo thecalculationandprocessingcanbequitedierent Since the ltering istransparent to thedecoder, itseems to the decoderthatthejitterthestreamhasexperiencedislarger Thisisnotabigproblem,sincethroughlongerbueringat the lter and the receiver and the use of jitter smoothing mechanisms,this stronger jitter willnot greatlyaect the decoding process The second problem, however, is more intractable As we said abo e, the packets forany access unitshould be positioned within the streamand so arrive

at the receiverat itssupposed time point forthe decoder

framebecomessmallerorlarger Ittakeslessormore

pack-etstocarry,andsoitsfollowingframesaredraggedearlier

orpushed lateralong thetimeline Insuchcircumstances,

ifwekeepboththetimestampandthespacingofthePCR

packetsunchanged,thenthereceiver'sclockcanstillbe

cor-rectlyreconstructed,butthearrivingtimeofeachframewill

beskewedalongthetimeline Forexample,ifthestreamis

lowpassltered,theneveryframebecomesshorter,andso

following frames are dragged forward to pack up the

va-cancyspared out Fromthedecoder'spointof view,more

and more future frames beginto come earlier and earlier,

and tobuer themuntil their stamped timefordecoding,

thebuerwillbeo wedinthelongrunnomatterhow

large itis The fundamental problem isthat after the

l-tering,the actualbitratebecomeslowerorhigher, butthe

dataisstillreadinbythedecoderattheoriginalrate Soif

thenewrateislower,moreandmorefuturedataisreadin

bythedecoder,causingthedecoderbuertoo w

even-tually;ontheotherhand,ifthenewrateishigher, thenat

somepoint inthefuture, the datawillbereadinafter its

decodingtimehasalreadypassed

3 OUR SOLUTION

One immediate thought would be to dothe same kind of

clockreconstructionasthedecoderdoesatthelter,andso

re-generatingthePCRpacketsto thechanges

How-ever,weknowthatthesmoothingmechanismslikePLLare

implementedinhardwarecircuitscontainingavoltage

con-trolledoscillatorthatgenerateshighfrequencysignalstobe

tunedwiththeincomingPCRtimestamps Butthisisnot

easy,ifnotimpossible,tobedoneinsoftware Weimagine

thataworkingsoftwareversionofthisschemerequiresome

specialrealtime supportfromtheoperating systemkernel,

whichwehavenotfullyexploredyet Furthermore,a

soft-ware lter is amuchmore convenient formforservice

dis-tributionand proxypropagationo erawideareanetwork,

suchasthe Internet, because theonlyresource requiredis

CPUandmemoryandnotanyspecialpurposehardware

The key idea of our solution comes from the observation

thatif we donottake anatomicviewof eachframe,then

theDTSandPTSareonlyassociatedwiththebeginningbit

ofeachframe Consequently,solongaswecanxthatpoint

tothecorrect positiononthetimeline,thedecodershould

be working ne even if the remaining bits of that frame

followingthestartingpointstretchesshorterorlonger

3.1 Simple Solution: Padding

Following thediscussion abo e, we have designed asimple

solutionthat worksforbitratereducing operations Wedo

not change the timestamp and the position of any PCR

packet along the timelinewithin the stream, and we also

preservethepositionoftheframeheaderandsothatofthe

beginningbitofevery frame Whatis changedhere isthe

sizeofeachframe intermsofnumberof bits, andwe just

pack thelteredbits ofapicture closelyfollowing the

pic-tureheader.Sinceeachframetakeslesspacketstocarry,yet

theframeheadersare stillpositionedattheiroriginaltime

points, we can imagine that there would be some \white

space" leftbetweenthe lastbit of oneframe and therst

bitofthe headerof thenext frame Actually thecapacity

spacewithemptypacketslikeNULLpackets

Thissolutionisverysimpletounderstand, implement and

itpreservesthetimingsynchronization, sinceweonlyneed

to pack the ltered bits of each frame continuously after thepicture headerandtheninsertNULL packetsuntilthe headerofthenext frame However,itinevitablyhasmany drawbacks First, it canonlyhandlebitratereduction op-erations We only try to x the header of each frame to itsoriginalposition,whichmeansthechangedframeshould not occupy a space larger than the distance between the currentframeheaderandthenext Thispropertydoesnot always hold, since some ltering operations like informa-tionembeddingandwatermarkingmayincreasethebitrate Secondly,the savedbitsare paddedwithNULLpacketsto maintaintheoriginalconstantbitrateandthestartingpoint

ofeachframe,andthisironicallyrunscountertoourinitial goalofbitratereduction Theresultingstreamcontainsthe same numberofpackets asthe originalone Theonly dif-ference isthat the numberof bits representing eachframe has been shrunk, yet this saving is spent immediately by paddingNULLpacketsattheend ofeachframe

Herewewanttomentionthattheredoesexistanother ap-proachtobypassthesecond problem Upto now we have been using alter modelthat is transparent to the client player, whichconnes usstrictly to the MPEG2 standard syntax However, ifsome ofthe lteringintelligenceis ex-portedtotheendhosts,thensomesavingcanbeexpected For example, instead of inserting NULL packets, we may compress them by insert only a special packet saying the next N packets should be NULL packets, and on seeing this packet, a smallstubat the end host (right before the clientplayer)takestheresponsibilityofreplacingthispacket withthe supposedamountofpadding packets (Note that thispaddingisimportanttomaintaincorrecttiming, espe-ciallyif theclientis usingsomestandardhardware decod-ing board.) This wa , thebandwidth is indeed saved,but

at theprice of relyingonextra non-standardprotocol Of course,thiswillintroduceallproblemsassociatedwith non-standardizedsolutions, suchasdiÆculty insoftware main-tenanceandupgrading Therefore,weonlyconsiderthisas

asecondchoice,andnotasafeasiblesolution

Figure3: Example: 2/3 shrinking

3.2 Enhanced Solution: Time-Invariant Bi-trate Scaling

To ultimately solve the synchronization problem, a more complexalgorithmhas beendesigned Thekey insight be-hinditisthatwecanchangethebitratetoanotherconstant valuewhilepreservingthePCRtimestampsbychangingthe

stant number This wa, we can scale the PCR packets

distanceand achieveaconstant yetdierent bitrate,asif

the time line is scaled looser or tighter to carry more or

less packets All non-videostream packets canbe simply

mappedtothescaledpositionthatcorrespondstothesame

timepoint onthe scaled timeline as on the original

time-line Incasethat noexact mappingis a ailable, we could

simplyusethenearesttimepointonthenewtimeline

with-outintroducinganyseriousproblem Forvideostream,the

samekindof pictureheaderxing andframedatapacking

areconductedasintherstsolutionbutinascaledwa

Anexample of shrinkingthe stream to its2/3 bandwidth

isgiveninFigure3 Allnon-videopacketsand video

pack-etsthatcontainpictureheaders aremappedtotheir

corre-spondingpositiononthenewtimeline,andsotheirdistance

isalsocutto2/3oftheoriginal Thevideopacketsare

l-teredandpackedcloselyandasearlyaspossiblewithinthe

new stream following the header Intuitively, the ltered

non-videopacketsandpictureheaderpackets

Thisalgorithmis alsoverysimpletoimplement Foreach non-videopacket, itsdistance(in numberofpackets) from thelastPCRpacketismultipliedbyascalingfactors,and theresultisusedtosetthedistancebetweenthispacketand thelastPCRpacketintheoutputstream.Forvideoframes, theheadercontainingDTSandPTSisscaledandpositioned

inthesamewa ,andtheremainingbitsarecloselyappended

totheheaderintheresultstream

Nowtheonlyproblemishowtodetermines Ifweshrinkthe timelinetoomuchandforsomeframesthebitratereducing operation doesnothave asignicant eect,thenagain we willnothaveenoughspaceto squeezeinthis frame,which will push the beginning bit of the next frame behind on thetimeline On theotherhand,ifweshrinkthe timeline toolittleorexpand( s>1)ittoomuch,thenmore space willbe paddedusingNULLpacketstopreserveimportant

one optimal scale factor sopt that can balance these two

strengthsifitfullstheconditionthat

 thelteredframedatawillalwaysbesqueezedintothe

scaledstream;

 the number ofNULL packets forpadding purposeis

minimum

However, this optimal scale factor is hard to estimate in

advance since for dierent operations and dierent video

clips of dierent scenes, the eect of the ltering on the

bitratecangreatlyvary

Therefore,inourcurrent implementation,wesimplyusea

slightlyexaggeratedscalefactorbasedontheoperationtype

andparameters For example, forlowpasslteringwitha

thresholdof5,ascalingfactorof0.9willworkalmostforall

streams Evenifwemeet aframethat still occupiesmore

than0.9numberofpacketsaftertheltering,onlythenext

fewframes maybe slightlyaected Since a

smaller-than-a erageframeisexpected tofollowshortly,this localskew

canbe absorbed by the decoder easilyand doesnot have

anychaineect

Ournext stepwillbelookinginto how to\learn"this

op-timalscalefactorbyanalyzing history bitratechange ofa

streamandadjustthisfactorsonthe Itisstillnotclear

howadecoder, especiallyhardwaredecoding board,would

reactiftheincomingstreamchangesfromoneconstant

bi-tratetoanother,anditisalsoanopenquestionhowquickly

itwouldadapttothenewrate

4 EXPERIMENTAL RESULT

Figure4shows theeect ofthe timelinescaling approach

EachpointonthexaxisrepresentsanoccurrenceofaPCR

packet, and the y axis shows in 3 colorshow many video

packets,NULLpacketsorpacketsforotherdatastreamare

inbetween each2PCRpackets We canseethat the

dis-tributionofthe3areasiskeptalmostconstantforthe

orig-inal frame exceptfor moreNULL packets at the end ofa

frame However,withoutscaling,thenumberofvideo

pack-etsvaries acrossdierentPCRintervalsand alot ofextra

spaceispaddedwithNULLpacketsas shownintheupper

rightsub-gure Ontheotherhand,ifwedoscalingwitha

scalingfactorof80%,thenthepaddingoccursmostlyonly

at the end of framesand the streamcontains mostlyonly

usefuldata

One thingwe need to point out here is that the skew of

access units along the timelinestillexist withthis scaling

approach What happens is that after the ltering

opera-tion,eachframeshrinkstoasizeapproximately80% ofits

originalsize Ifwemask out allother packets,wecan see

thatinthevideostream,framesare packedcloselyone

af-teranother If oneframe takesmorespace thanitsshare,

thenthe next frame may be pushed behind itstimepoint,

yetthis skew willbe compensated later by another frame

withalargershrink eect As wesaid before,thiskind of

smalljitteraroundtheexacttimepointonthescaled

time-lineisacceptable,anditisthechangeinthebitrateatwhich

thedecoderreadsinthedatathatfundamentallymakesour

5 CONCLUSION AND FUTURE WORK

In this paper, we have presented our experience in imple-mentingarealtimesoftwareltersystemforMPEG2 Trans-portstream.Thesynchronizationproblemisintroducedand oursolutionsandexperimentalresultsaregiven

Ourworkisarsteortinpromotingrealtimesoftware l-teringofMPEG2streams,andcanbebenecialtomany re-altimeapplicationsthatworkwithMPEG2systemstreams likeHDTVbroadcast

However,manyproblemsremaintobesolved,suchas real-timeadaptation ofthescalefactors Theywillbefurther studiedinourfuture work

6 ACKNOWLEDGMENT

ThisworkwassupportedbytheNASAgrantundercontract number NASA NAG 2-1406, National ScienceFoundation under contract numberNSFCCR-9988199 and NSF CCR

0086094, NSFEIA 99-72884 EQ, and NSFEIA 98-70736 Anyopinions,ndings,andconclusionsorrecommendations expressedinthismaterialarethoseoftheauthorsanddonot necessarily theviews of theNational Science Foun-dationorNASA

7 REFERENCES

[1] ProxiNet.http://www.proxinet.com

[2] J.Smith,R.Mohan,andC.Li\Scalablemultimedia deliveryforpervasivecomputing",ACM Multime-dia, 1999

[3] A.Fox,S.D.Gribble,Y.Chawathe,andE.A.Brewer

\Adaptingtonetworkandclientvariationusingactive proxies: lessonsandperspectives",IEEEPersonal Communication,Vol.5,No.4,pp.10C19,August 1998

[4] W.Y.Ma,B.ShenandJ.Brassil\ContentServices Network: TheArchitectureandProtocols",

Proc edingsoftheSixthInternationalWorkshopon WebCachingandContentDistribution,June2001

[5] NicholasJYeadon,PhDThesis.LancasterUniversity, Lancaster,May,1996 \QualityofServiceFiltersfor MultimediaCommunications",

http://www.comp.lancs.ac.uk/computing/users/njy/thesis/

[6] BinYu,KlaraNahrstedt.\RealtimeInformation EmbeddingofHDTVStream",Tob submittedto ICME2002

[7] ISO/IECInternationalStandard13818-1/13818-2,

\Genericcodingofmovingpicturesandassociated audioinformation",November1994

[8] IntroductiontoHDTV,http://www.ee.washington.edu /conselec/CE/kuhn/hdtv/95x5.htm

[9] HDTV-over-IPhasarrived! http://www.2netfx.com/

[10] C.E.Holborow\SimulationofPhase-LockedLoopfor processingjitteredPCRs,"ISO/IEC

JTC1/SC29/WG11,MPEG94/071,March1994

However,manyproblemsremaintobesolved,suchas real-timeadaptation ofthescalefactors... class="text_page_counter">Trang 4

stant number This wa, we can scale the PCR packets

distanceand achieveaconstant yetdierent bitrate,asif... paddedusingNULLpacketstopreserveimportant

one optimal scale factor sopt that can balance these