Playing with tension a computational mode of improvisational accompaniment by secondary rhythmic performer in carnatic music

... development and validation of a tension model that, assuming restricted sowkhyam, is able to generate alternate variations of secondary accompaniment that are as valid as the original accompaniment. .. the database Since each rhythm in the database is distinctly characterized by a single set of accompaniment values, there is always only one accompaniment available for any given musical scenario... multiple valid accompaniments by modeling the constraints of accompaniment playing, is the problem of interest in this thesis Computational creativity is an emerging field of research in artificial intelligence,

PLAYING WITH TENSION

PRASHANTH THATTAI RAVIKUMAR

NATIONAL UNIVERSITY OF SINGAPORE

2015

PLAYING WITH TENSION

GENERATING MULTIPLE VALID ACCOMPANIMENTS FOR THE

SAME LEAD PERFORMANCE

PRASHANTH THATTAI RAVIKUMAR

B.Tech, National Institute of Technology, Trichy,

2012

A THESIS SUBMITTED FOR THE DEGREE OF MASTER OF ARTS COMMUNICATIONS AND NEW MEDIA NATIONAL UNIVERSITY OF SINGAPORE

2015

Foremost, I would like to express my sincere gratitude to my supervisorsProf Lonce Wyse and Prof Kevin McGee for their continuous support,patience, motivation, enthusiasm and immense knowledge in guiding me

to learn and do research "To define is to limit" – I cannot quantify theknowledge that I have learned from them in the past two years Theirconstant guidance, support and dedication has been a immense inspirationfor me to finish this dissertation

Besides my supervisors, I would like to thank Dr Srikumar KaraikudiSubramanian, who has been a friend, a mentor and a person to look upto

I will long cherish the memorable coffee-chats that have lead to so manynew insights about the thesis, music and varied things in life

I thank my fellow lab mates from the Partner Technologies group, Dr.Alex Mitchell, Teong Leong, Chris, Jing, Evelyn, and Kakit, for their stim-ulating discussions every week Our weekly meetings used to be a ton of fun

in terms of discussing and learning diverse perspectives of doing research

I thank the faculty, the staff and the graduate students of the nications and New Media department for supporting and housing me as agraduate student for the last two years

Commu-I thank the musicians, Dr Ghatam Karthik, Mr Trichur Narendran,

Mr Arun Kumar, Mr Sumesh Narayan, Mr Sriram, Mr Hari, Mr.Shrikanth, Mr Santosh and all others who have imparted their musicalknowledge to help my understanding of the genre

This thesis could not have progressed as much as it has, if not forthe musical insights and inspirations that I drew from our group musicjamming sessions I take this moment to thank to my close friends andmusic collaborators - Vinod, Vishnu, Lakshmi Narasimhan, Prasanna andArun – who have enhanced my musical growth and helped me achieve theinsights that I have in this thesis

I thank my close friends Shyam and Kameshwari who have been a stant source of support during the tough times I thank my friend Akshayfor the intellectually stimulating conversations I also thank him for histimely help during the thesis revisions I thank Spatika Narayanan for herhelp in proof-reading the document

con-Last but not the least, I would also like to thank my family

March 20, 2015

Name : Prashanth Thattai Ravikumar

Supervisor(s) : Associate Professor Kevin McGee, Associate

Pro-fessor Lonce Wyse

Generating multiple valid accompanimentsfor the same lead performance

Abstract

One area of research interest in computational creativity is the opment of interactive music systems that are able to perform variant, validaccompaniment for the same lead performance

devel-Although previous work has tried to solve the problem of ing multiple valid accompaniments for the same lead input, success hasbeen limited Broadly, retrieval-based music systems use static databasesand produce accompaniment that is too repetitive; generation-based mu-sic systems that use hand-coded grammars are less repetitive, but have

generat-a more limited rgenerat-ange of pre-defined generat-accompgenerat-animent options; generat-and fingenerat-ally,transformation-based music systems produce accompaniment choices whichare predictably valid for only a few cases

This work goes beyond the existing work by proposing a model ofchoice generation and selection that generates multiple valid accompani-ment choices given the same input The model is applied to generate sec-ondary percussive accompaniment to an lead percussionist in a Carnaticimprovisational ensemble

The central insight – the main original contribution – is that the eration of valid alternate variations of secondary accompaniment can beaccomplished by formally representing the relationship between lead andaccompaniment in terms of musical tension By formalizing tension rangesfor acceptable accompaniment, an algorithmic system is able to generate al-ternate accompaniment choices that are acceptable in terms of a restrictednotion of sowkhyam (roughly, musical consonance) In the context of thisthesis, restricted sowkhyam refers to the sowkhyam of accompaniment con-

gen-sidered independent of the secondary performer (and his creativity).The research proceeded in three stages First, Carnatic music perfor-mances were analyzed in order to model the performance structures and im-provisation rules that provide the freedom and constraints in secondary per-cussion playing Second, based on the resulting tension model, a softwaresynthesis system was implemented that can take a transcribed selection

of a Carnatic musical performance and algorithmically generate new formances, each with different secondary percussion accompaniment thatmeet the criteria of restricted sowkhyam Third, a study was conductedwith six expert participants to evaluate the results of the synthesis.The main contribution of this thesis is the development and validation

per-of a tension model that, assuming restricted sowkhyam, is able to generatealternate variations of secondary accompaniment that are as valid as theoriginal accompaniment

accompaniment

1.1 Structure of this document 2

2 Related work 5 2.1 Retrieval-based music systems 5

2.1.1 Retrieval from a database 6

2.1.2 Retrieval using dynamic learning models 6

2.1.3 Generation-based music systems 8

2.2 Hand-coded grammars 8

2.2.1 Online learning of grammars 8

2.3 Transformation-based music systems 9

2.3.1 Transformation function is pre-given 9

2.3.2 User selects the transformation function 10

3 Research problem 13 3.1 Summary of the related work 13

3.2 Proposed solution 15

4 Method 17 4.1 Analysis of the Carnatic musical performances 17

4.2 Model development 17

4.3 Evaluating the tension model 18

4.4 System development 18

5 Background: Carnatic quartet performance 19 5.1 Overview 19

5.2 Musical structures 20

5.3 Choices in different styles of accompaniment playing 21

5.4 Musical actions in the improvisation 22

5.4.1 Major variations 22

5.4.2 Minor variations 24

6 System: design criteria & constraints 27

6.1 Research/Implementation model 27

6.2 Lead percussionist: improvisation and variation 28

6.3 Secondary percussionist: accompaniment and variation 29

7 Possible approaches 31 7.1 The Direct Mapping model 32

7.2 The Horizontal Continuity model 33

8 The tension model 35 8.1 Tension model applied to secondary playing 35

8.2 Tension model applied to generate multiple accompaniments 36 9 Tension synthesis protocol 37 9.1 Choose Carnatic performance recording 38

9.2 Choose a sixteen bar sample of performance recording 39

9.3 Transcribe the sixteen-bar selection 39

9.3.1 Transcribing double hits 40

9.3.2 Transcribing hit loudness 40

9.3.3 Transcribing rhythmic repetition of bars 40

9.4 Compute tension scores for each hit 42

9.5 Compute tension scores for each beat 42

9.6 Compute tension range for each bar 43

9.7 Generate all viable accompaniment sequences 46

9.7.1 Enumerate all unique triplet values for each beat 47

9.7.2 Collect all viable 8-beat (1-bar) sequences 47

9.7.3 Collect secondary sequences that meet tension con-straints 48

9.8 Construct secondary transcription for entire piece 50

9.9 Synthesize performance 51

10 Tension synthesis: practical details 53 10.1 Separating tracks from original recording 53

10.2 Storing the transcript 54

10.3 Sequencing audio from a transcript 54

10.4 Creating a new recording 54

11 Study protocol 55 11.1 Participants 56

11.2 Materials 57

11.2.1 Documents 57

11.2.2 Equipment 58

11.2.3 Recordings (original) 58

11.2.4 Recordings (with new accompaniment) 59

11.3 Study Disclaimer 63

11.4 Study Session Protocol 66

11.4.1 Gather demographic information 67

11.4.2 Explain evaluation criteria 67

11.4.3 Sequencing the recordings 68

11.4.4 Evaluate recordings 69

11.5 Evaluation 71

12 Study results 73 12.1 RQ1: does system produce acceptable accompaniment 74

12.1.1 Recording 1 75

12.1.2 Recording 2 75

12.1.3 Recording 3 76

12.2 RQ2: are accompaniments inside the range better? 76

12.2.1 Recording 1 77

12.2.2 Recording 2 77

12.2.3 Recording 3 78

12.3 RQ3: do ratings decrease as a function of distance 78

12.3.1 Recording 1 79

12.3.2 Recording 2 79

12.3.3 Recording 3 80

12.4 Summary 80

13 Potential objections 81 14 Discussion 85 14.1 Algorithmic limitations 85

14.2 Transcription limitations 86

14.3 System limitations 86

15 Future work 89 Appendices 93 A Key Terms 95 A.1 Terms: tension model 95

A.2 Terms: Carnatic music 96

B Enumerating the accompaniment sequences 99

C.1 Diction 101

C.2 Loudness 102

C.3 Note duration 104

D Transcription: internal representation 105 D.1 Transcription: internal representation 109

E Results 111 E.1 Complete results for recordings 111

E.2 Complete results for variants 113

F Study documents 115 F.1 Session checklist 116

F.2 Demographic questionnaire 117

F.3 Participant variant sequence 118

F.4 Evaluation sheet 119

F.5 Participant observation form 120

F.6 Participant definition sheet 121

List of Tables

9.1 Rhythmic repetition of bars 41

9.2 Tension scores for each hit 42

9.3 Tension scores for each beat 43

9.4 Computing TZP and tension range for a bar 44

9.5 Computing TZP and tension range for a bar 44

9.6 Computing TZP and tension range for a bar 45

9.7 Lookup table for 2-beats 47

9.8 Possible 2-beat diction combinations 47

9.9 Possible 2-beat diction combinations 48

9.10 Valid 3-beat diction combination 48

9.11 Two bars (average tension scores) 49

9.12 Two bars of valid sequences 49

9.13 Rhythmic repetition of bars, with accompaniment 50

11.1 Participant data 57

11.2 Two bars of valid sequences 60

11.3 Two bars of valid sequences 61

11.4 Variants by distance value 62

11.5 Distance of variants used for recording 1 64

11.6 Distance of variants used for recording 2 64

11.7 Distance of variants used for recording 3 64

11.8 Recording sequences for participants 68

11.9 Variant sequences for participant 69

12.1 Average accompaniment rating per recording 74

12.2 Average rating for variants of recording 1 75

12.3 Average rating for variants of recording 2 75

12.4 Average rating for variants of recording 3 76

12.5 Accompaniment ratings for variants of recording 1 77

12.6 Accompaniment ratings for variants of recording 2 78

12.7 Accompaniment ratings for variants of recording 3 78

12.8 Accompaniment ratings for different variants 79

12.9 Accompaniment ratings for different variants 79

12.10Accompaniment ratings for different variants 80

C.1 Weights for lead strokes 101

C.2 Weights for secondary strokes 102

C.3 Perceived loudness of lead and secondary hits 103

C.4 Weights for loudness 103

C.5 Weights for note duration 104

D.1 Transcription of recording 1, bars 1-16 106

D.2 Transcription of recording 2, bars 1-16 107

D.3 Transcription of recording 3, bars 1-16 108

E.1 Accompaniment ratings for recordings 1, 2, and 3 112

E.2 Accompaniment ratings for variants 0-6 113

F.1 Recording and variant sequences 118

List of Figures

5.1 The Carnatic quartet (from left): lead percussionist, sec-ondary, vocalist, Tambura (provides the background drone),

and violinist 19

5.2 Two bars of lead and secondary playing 21

5.3 Different minor variations 24

7.1 Direct Mapping 32

7.2 Horizontal Continuity: secondary follows the lead changes 33 8.1 Tension-relaxation visualization 35

8.2 Tension between lead and secondary 36

List of Algorithms

1 Hit tension score calculation 42

2 Beat tension score calculation 43

3 Unique 1-hit and 2-hit triplets 99

4 Unique 1-beat triplets 99

5 Unique 8-beat triplets 99

Chapter 1

Introduction

This chapter introduces the research area of musical sational accompaniment systems and highlights an importantproblem in this field Improvisational accompaniment systemsdiffer from score-following, solo-trading, and tap-along systems

improvi-in that they are able to produce multiple valid musical tives for the same performance Developing musical accompa-niment systems that generate multiple valid accompaniments

alterna-by modeling the constraints of accompaniment playing, is theproblem of interest in this thesis

Computational creativity is an emerging field of research in artificialintelligence, cognitive psychology, philosophy, and the arts The goal ofcomputational creativity is to model, simulate or replicate human creativ-ity using a computer One area of research interest in computational cre-ativity is the development of improvisational music systems that are able

to perform variant, valid accompaniment for the same lead performance.Developing musical accompaniment systems that generate multiple validaccompaniments by modeling the constraints of accompaniment playing, isthe problem of interest in this thesis

Although previous work has tried to solve the problem of generatingmultiple valid accompaniments for the same lead input, success has beenlimited Broadly, retrieval-based music systems that use static databasesare produce accompaniment that is too repetitive; generation-based mu-sic systems that use hand-coded grammars are less repetitive, but have

a more limited range of pre-defined accompaniment options; and finally,transformation-based music systems produce accompaniment choices whichare predictable valid for only a few cases

This work goes beyond the existing work by proposing a model ofchoice generation and selection that generates multiple valid accompani-ment choices given the same input.

1.1 Structure of this document

The remainder of this document is structured as follows:

• Related work This chapter summarizes the previous work on visational accompaniment systems developed for generating multiplevalid accompaniments by modeling the constraints of accompanimentplaying

impro-• Research problem This chapter identifies a significant problem leftopen by previous work and presents the research focus: to develop amodel of rhythmic accompaniment for Carnatic ensemble music thatproduces multiple musically valid accompaniments, given the sameinput

• Method This chapter provides a brief overview of the method usedduring this thesis research The method included the analysis ofCarnatic music performances, development of different models of ac-companiment playing, their implementation as computer programs,and their evaluation

• Background This chapter describes the roles and activities of thelead and secondary percussionist within a Carnatic quartet perfor-mance It further describes the musical structure and provides exam-ples of different scenarios of lead and secondary percussion playing in

a performance ensemble

• System design criteria This chapter describes the narrow subset

of constraints that guided the research and development of the ondary accompaniment system The structural constraints separatethe music into improvisational cycles made of eight bars in a 4/4 timesignature The input constraints restrict the lead to minor bar varia-tions The output constraints restrict the scope of secondary accom-paniment to playing compliant accompaniment to the lead Withinthese constraints, the secondary system still has the freedom to play

sec-a vsec-ariety of vsec-alid sec-accompsec-animents in sec-a given situsec-ation

• Possible approaches This chapter describes two seemingly-reasonableapproaches – Direct Mapping and Horizontal Continuity – and showswhy they will not effectively solve the central research problem

• The tension model This chapter describes the tension model thatwas developed to address the shortcomings of the previous models.Applied to the activity of secondary accompaniment playing in aCarnatic performance, the tension model is used as a constraint sat-isfaction mechanism to generate multiple accompaniments given thesame lead.

• Tension synthesis protocol This chapter describes the main stepsinvolved in synthesizing recordings with variant valid accompaniment

• Tension synthesis: practical details This chapter describes thedifferent steps in the synthesis process in terms of the different tech-nologies used to implement them

• Study protocol This chapter describes the study conducted withmusical experts for evaluating the ability of the system to producealternate valid secondary accompaniments for a Carnatic musical per-formance

• Study results This chapter describes the main results from the userstudy and uses them to answer the research questions

• Potential objections This chapter highlights the aspects of thestudy design that could raise objections about the claims made fromthis work

• Discussion This chapter identifies the main limitations of the search reported here and discusses their impact on the findings fromthe study

re-• Future work This chapter proposes directions for future work.The next chapter reviews work on developing improvisational accompa-niment systems that generate multiple valid accompaniments by modelingthe constraints of accompaniment playing

Chapter 2

Related work

This chapter summarizes the previous work on improvisationalaccompaniment systems developed for generating multiple validaccompaniments by modeling the constraints of accompanimentplaying Previous work has developed retrieval-based musicsystems, generation-based music systems and transformation-based music systems to solve the problem Retrieval-basedmusic systems use dynamic learning models to produce differ-ent sequence continuations given the same input, but at anygiven point in the performance they produce deterministic out-put Generation-based music systems dynamically update theproduction rules of a grammar that are used to generate dif-ferent accompaniments, but at any given point in the per-formance the production rules produce deterministic output.Transformation-based music systems generate permutations of

a source rhythm representation to generate multiple niments, but the generated choices are not always musicallyvalid

accompa-Previous work that has tried to solve the research problem can be fied into retrieval-based, generation-based, and transformation-based musicsystems This chapter reviews the systems and highlights the problems theysolve

classi-2.1 Retrieval-based music systems

Retrieval-based music systems use musical parameters to retrieve the bestpossible accompaniment from a set of accompaniment patterns The focus

is on optimizing the parameters for efficient representation and real-time

retrieval There are two variations of retrieval-based music systems based

on the type of data structure used to store the accompaniment: retrievalfrom a database and retrieval using dynamic learning models

2.1.1 Retrieval from a database

The first type of retrieval-based music systems store the accompaniments in

a database which is queried to retrieve the accompaniment The niments in the database are organized by their musical features Retrievalsystems extract the necessary musical features from the input, packagethem into a data format which is suitable to query the database, and re-trieve the accompaniment The best matching accompaniment is retrievedand played

accompa-Impact is an accompaniment system that uses case-based reasoning andproduction rules to retrieve accompaniment from a database of accompani-

meta-level descriptions of musical scenarios (such as the beginning and end of abar), fills in the sections and the duration of chords, and uses the result toform a query This query is used to retrieve the best matching accompa-niment from the database The best accompaniment is selected according

to a measure of mathematical distance between the query (called targetcase) and each of the patterns in the database Given a single input, thesystem always returns one accompaniment (the best matching accompani-ment) as output Cyber-Joao is an adaptation of the Impact system thatoptimizes the number of parameters used for the retrieval (Dahia et al.,

data, and uses the ranking to determine the important musical features in

a given performance situation Each rhythm is distinctly characterized by

a single set of accompaniment values and the musical features are used toquery and retrieve the accompaniment pattern from the database Sinceeach rhythm in the database is distinctly characterized by a single set ofaccompaniment values, there is always only one accompaniment availablefor any given musical scenario

2.1.2 Retrieval using dynamic learning models

In order to overcome the limitations of statically stored accompanimentoptions, systems were developed with capabilities to model the input ratherthan statically store it

One of the earlier systems that retrieved accompaniment using Markov

perfor-mance as streams of MIDI data and builds a Markov chain representation

on the fly It traverses over the representation in order to send the output

sequence continuations of musical sequences played earlier in the mance For any given sequence of musical notes, the accompaniment isretrieved by selecting the longest sequence continuation A later version ofthe Continuator system models the trade-offs between adaptation and con-

Apart from finding a continuation sequence, the system constantly reviewsthe relationship between the retrieved accompaniment and the harmoniccontext to retrieve a new continuation in case of any mismatch Anothersystem, Omax, in addition to listening to lead, listens to its own past im-

system listens to its own outputs to bias its Markov model This results

in a variety of possible choices for future accompaniment, depending onwhether the system was listening to itself or to the lead

The second variation of the retrieval systems also use Markov models toproduce sequence continuations of accompaniment These systems modelthe music as sequence continuations, based on listening to the improviser’sinput Given a starting note or a sequence, the model is traversed to pro-duce the musical continuation As the system listens to more of the input

it changes the Markov model and the sequence continuations Thus it isable to produce multiple alternate accompaniments for different situations.Although the use of modeling approaches improves performance over thestatic database approach, at any point in the performance these systemsretrieve and play only one valid accompaniment

There is, however, one non-accompaniment system that falls broadlyinto this category, but which generates musically-valid variations that do

This “control improvisation system” generates variations of a lead melody

in jazz Specifically, given a reference melodic and harmonic sequence, thesystem builds a probabilistic model of all state transitions between thenotes of the melody The probability values assigned to the transitionsdetermine the variations of the main melody produced Assigning a highprobability to transitions of the reference melody (called direct transitions),

it produces melodic sequences similar to the reference melody Assigning

low probability to the direct transitions, it produces melodic sequences ferent from the reference line Thus, given the same harmonic progressionand a reference melodic line, the system produces variations by controlling

dif-a single pdif-ardif-ameter, the probdif-ability vdif-alue of trdif-ansitions

Although it is not an accompaniment system, the approach could ceivably be used as the basis for one, but not without significant modifica-tion This is because the generation part of the system is entirely influenced

con-by itself, con-by what it played earlier Without modification, this would sult in an odd accompaniment scenario, one in which the choices of theaccompanist are based on his own decisions rather than being based on thechanges played by an improviser And if the goal was to transform thisinto an accompaniment system, it would not be sufficient to simply modifythe system so that it listened to the lead performer; many of the challengesand limitations described in future chapters would still appear

re-2.1.3 Generation-based music systems

Generation-based music systems use musical grammars to generate paniment The grammars contain production rules that associate the char-acteristics of the input rhythm with an output accompaniment rhythm.The grammars are either hand-coded by a human expert or automaticallyinducted by listening to performances There have been several systemsdeveloped using each type of grammar

accom-2.2 Hand-coded grammars

accom-paniment systems that uses hand-coded grammars to generate ment responses They contain pre-defined sub-routines that are triggered

accompani-by specific conditions to generate the different accompaniment responses.However, the rules of these grammars are rigid and unchanging, and as aresult, these systems are limited in their ability to respond to the sameinput with alternative outputs

2.2.1 Online learning of grammars

One improvement over hand-coded grammars is the development of mars that are more flexible and learn on the fly

gram-ImprovGenerator is an example of an accompaniment system that learns

varia-tions of a base rhythm and generates production rules corresponding to thevariations The different production rules are assigned a probability valuethat changes over the course of a performance FILTER is another systemthat employs an online learning approach (Van Nort, Braasch, and Oliv-

instrument system that reacts in novel and interesting ways by recognizingthe gestures of a performer The system comes pre-loaded with 20 gesturesand the transitions between the gestures are modeled by a Markov model.Over the course of a performance, it varies the transition probabilities of thegestures to produce interesting and varied responses However, the relationbetween the gesture and the output parameters itself remains constant Inother words, it does a better job of generating different responses over thecourse of a performance, but at any given point in the performance, it will

Grammar systems that use online learning are more flexible and ate more varied responses compared to the systems developed using hand-coded grammars However, in both cases, the grammars are modeled deter-ministically and once the grammar is inducted, the same input will producethe same output

gener-2.3 Transformation-based music systems

Transformation-based music systems apply a transformation function onthe input to generate the output The transformation function is usually amathematical operation that is applied on each of the input parameters toproduce the output accompaniment values Multiple accompaniments aregenerated by permuting a representation of the input parameters Thereare two kinds of transformation systems based on how the transformationsare generated: systems where transformation function is pre-given and sys-tems where the user selects the transformation function

2.3.1 Transformation function is pre-given

In pre-given transformation systems, the transformation function computed

is given through a target accompaniment value, which is given as input tothe system The transformation function is computed as a function of the

1 One notable thing about FILTER is that it models the interplay between lower level audio features and higher level gestural parameters This will be discussed in more detail in later chapters.

target accompaniment and is applied on the input values to generate theaccompaniment.

Ambidrum is one system that uses a statistical measure of rhythmic

It measures rhythmic ambiguity using a statistical correlation between therhythmic metre and three rhythmic variables: the beat velocity, pitch,and duration The system is given the target correlation values which ituses to transform the input to the output which can be either metricallycoherent or metrically ambiguous rhythms Metrically coherent rhythmsare musically valid as accompaniment and are generated by Ambidrumsystem when its target correlation matrix (transformation function) is anidentity function When the transformation function is not an identityfunction, the rhythms generated by Ambidrum are metrically ambiguousand their musical appropriateness varies widely

Another system, Clap-along, uses values from the target accompaniment

sys-tem uses four musical features to compute the distance between the sourceand the target accompaniment and progressively modifies the source to-wards the target For each generation, the system generates 20 choices andfinds the closest rhythm to the target by computing the Euclidean distance.When the performer repeatedly claps the exact same pattern, the system isable to slowly evolve its output towards the target accompaniment How-ever, variations in the performer’s rhythms causes unstable changes in thesystem’s output, often resulting in inappropriate accompaniment

The main limitation of both these systems (and systems like these) isthat there are very few cases when the accompaniment generated by thesystems is predictably valid (musically)

2.3.2 User selects the transformation function

In order to get transformation systems to generate [musically predictable]output, systems have been created in which user preference is used to gener-ate transformation functions For example, NeatDrummer generates drum-tracks by transforming the other musical parts in the song (Hoover, Szerlip,

ac-companiment tracks are generated by giving different input tracks (likepiano, violin, and vocal) to an Artificial Neural Network, called CPPN,that generates the output rhythms The CPPN is initially trained by usingthe input from the different audio tracks In the successive generations, the

user ranks the generated tracks, which are used to generate multiple CPPNs

in each step Thus the different CPPNs generate multiple accompanimenttracks according to the user preference In the successive generations, theuser ranks the generated tracks, the properties of which are permuted togenerate multiple transformations function that generates different accom-paniment drum tracks

The problem with the approach followed by NeatDrummer is related tothe musical validity of the generated tracks Although the user’s preferencesare used to generate multiple alternate transformation functions, the useractually has minimal control over the accompaniment generation processitself Thus, the system generates musically valid accompaniment in only

a few cases

Chapter 3

Research problem

This chapter identifies a significant problem left open by ous work and presents the research focus: to develop a model ofrhythmic accompaniment for Carnatic ensemble music that pro-duces multiple musically valid accompaniments, given the sameinput Although there has been previous work on improvised ac-companiment playing systems, none of them have addressed theproblem of generating multiple accompaniment, given the sameinput This work goes beyond the existing work by proposing aformal model of choice generation that provides multiple validaccompaniment choices given the same lead input This formalmodel was used to develop a rhythmic improvisation system –specifically, a system that will provide percussive improvisa-

Carnatic performance

3.1 Summary of the related work

Although there has been previous work that has tried to solve the problem

of accompaniment playing, the problem of generating multiple ment given the same input is still largely unsolved

accompani-Retrieval-based music systems use dynamic learning models to producedifferent sequence continuations, but at any given point in the performancethey produce a single valid output Generation-based music systems dy-namically update the production rules of a grammar that are used to gen-erate different accompaniments, but at any given point in the performancethey produce a single valid output Transformation-based music systemspermute a source rhythm representation to generate multiple accompani-

ments, but the generated choices correspond to valid musical descriptions

in very few cases

Among the different systems surveyed in the related work, the FILTERsystems comes close to generating multiple accompaniment, given the sameinput It models the interplay between the low-level audio features andthe higher level gestural parameters (that have visual correspondences),

to identify the player’s intent and adapts the output in interesting ways.Though this interplay produces a variety of responses, the mapping betweenthe gesture and the associated output parameters is one-to-one Once amapping is established, the system produces the same outputs for the sameinput

Another related system is the control improvisation system that

accompaniment system but is relevant in that it models accompanimentconstraints to generate multiple variations of a given melodic line Thissystem is an enhancement of factor oracle approach used in (Assayag et

satisfy a given accompaniment specification Given a reference melodicand harmonic sequence, the system builds a probabilistic model of all statetransitions between the notes in the melodic The probability values as-signed to the transitions determine the variations of the main melody pro-duced Assigning a high probability to transitions of the reference melody(called direct transitions) produces melodic sequences similar to the refer-ence melody and assigning low probability of direct transitions producesmelodic sequences different from the reference line Thus, given the sameharmonic progression and a reference melodic line, the system producesvariations by controlling a single parameter, the probability value of tran-sitions

However, this system would not scale very well in an accompanimentscenario as the generation part of the system is purely influenced by what

it played (or listened to) earlier This results in an odd accompanimentscenario, one in which, the choices of the accompanist are based on his owndecisions rather than being based on the changes played by an improviser.This raises concerns about the validity of the accompaniment played usingsuch a system

3.2 Proposed solution

The central goal of the work reported here is to develop an algorithmthat generates valid alternate variations of secondary accompaniment forrecordings of Carnatic musical performances The central insight – themain original contribution – is that the generation of valid alternate vari-ations of secondary accompaniment can be accomplished by formally rep-resenting the relationship between lead and accompaniment in terms ofmusical “tension” By formalizing tension ranges as constraints for accept-able accompaniment, an algorithmic system is able to generate alternateaccompaniment choices that are acceptable in terms of a restricted notion

of sowkhyam (roughly, musical consonance)

In the context of this thesis restricted sowkhyam refers to the sowkhyam

of accompaniment considered independent of the secondary performer (andhis creativity) This specifically ignores influences of any particular school

of percussion playing, any particular secondary performer’s playing style,creative kanjira variations, and the tonal quality unique to any performer’sinstrument Unless otherwise noted, sowkhyam in this document refers tothe restricted sowkhyam described above

The central insight is roughly as follows For any given performance,there is a degree of sowkhyam (consonance) between the lead and sec-ondary accompaniment For the work reported here, this degree of (re-stricted) sowkhyam has been numerically formalized as the inverse rela-tionship: tension With this formalization, any synthesized accompani-ment that has equivalent or less tension (relative to the lead) is consideredequally sowkhyam as the original

The research resulted in a system that can take a transcribed selection

of a Carnatic musical performance and algorithmically generate new formances, each with different secondary percussion accompaniment thatmeet the criteria of restricted sowkhyam as well as the original secondaryaccompaniment

per-In order to evaluate the ability of the system to produce alternate validsecondary accompaniments for a Carnatic musical performance, a studywas conducted with musical experts to address three related research ques-tions:

• RQ1: Does the system produce secondary accompaniment that israted at least as high as the original accompaniment?

• RQ2: Are accompaniments inside the range better (i.e., do variantswithin the range get higher scores than variants outside the range)?

• RQ3: Do the ratings for accompaniment decrease as a function of thedistance from the tension zero point?

The remaining chapters in this thesis provide details about the synthesisprotocol for generating alternate valid accompaniments, the study protocolused to evaluate the system, the results of the study and their analysis

Chapter 4

Method

This chapter provides a brief overview of the method used ing this thesis research The method included the analysis ofCarnatic music performances, development of different models

dur-of accompaniment playing, their implementation as computerprograms, and their evaluation

4.1 Analysis of the Carnatic musical

perfor-mances

Since the rules and constraints for secondary improvisation in Carnaticensemble are not clearly specified in the literature (or in oral tradition),the first step involved the development of a method to systematically un-derstand secondary improvisation in performances Different performancerecordings were analyzed to find performance structures (e.g., bar and im-provisation cycle) and improvisation rules (e.g., forced and discretionaryplaying) that restricted/imposed constraints on the playing, but also of-fered some flexibility for improvisation The analysis of the performanceswas used to develop the different models of accompaniment playing

4.2 Model development

During the course of this thesis, different models were developed to solvethe research problem of developing systems that play multiple valid ac-companiment given the same lead input They were the Direct Mappingmodel, the Horizontal Continuity model, and the tension model The firsttwo models were limited in their ability to generate multiple accompani-ments that are musically valid In order to address these shortcomings, a

third model was developed that generates multiple valid accompanimentfor the same lead input: the tension model.

4.3 Evaluating the tension model

In order to evaluate the ability of the system to produce alternate valid ondary accompaniments for a Carnatic musical performance, a study wasconducted with musical experts to answer three related research questions:

sec-• RQ1: Does the system produce secondary accompaniment that israted at least as high as the original accompaniment?

• RQ2: Are accompaniments inside the range better (i.e., do variantswithin the range get higher scores than variants outside the range)?

• RQ3: Do the ratings for accompaniment decrease as a function of thedistance from the tension zero point?

The methodology followed to answer these research questions was togenerate accompaniment variants that were qualitatively different from anevaluation standpoint For this, six accompaniment variants of 16-bar du-ration were created with different distance values from the original Thesewere presented to musical experts who evaluated and rated them

4.4 System development

The Direct Mapping and the Horizontal Continuity accompaniment modelswere implemented as computer programs that were evaluated in restrictedreal-time performance settings For these models, the accompaniment sys-tem is a computer program that plays a melody, accepts percussive inputthrough a midi controller, and combines both of those with the secondaryaccompaniment (which is algorithmically generated) The lead’s input isused to drive the algorithmic secondary accompaniment generation, andthe combined output is played back through a speaker

The accompaniment system built using the tension model accepts cussive input for lead and secondary through the keyboard and combinesboth of those to algorithmically generate multiple accompaniments Theinputs are obtained in their respective format on the keyboard It (input)consists of sequences of diction, note duration, and loudness represented

per-in array format Usper-ing this per-input, the system algorithmically generatesthe corresponding arrays for the secondary accompaniments The tensionmodel and its generation process are described in much more detail in sec-tion 8

It further describes the musical structure and provides examples

of different scenarios of lead and secondary percussion playing

ist performs the main melody and the violinist plays the accompanimentmelody The lead percussionist improvises relative to the melody (vocal andviolin) and the secondary percussionist mostly provides accompaniment tothe lead percussionist The actions of the secondary percussionist are con-strained by what the lead percussionist plays and by what the secondarypercussionist predicts that the lead will play.

There is one main difference in the nature of playing on the lead and

by the secondary drums The lead uses both hands to simultaneouslystrike the different sides of the lead drum, called the mridangam Thesecondary uses one hand to strike the drum, while controlling the tension

of the membrane that he strikes with the other hand The secondary drum

is called the kanjira or the frame drum

In general, the secondary percussionist is trying to “follow” the lead.This means that the secondary takes cues from the lead, is guided by whatthe lead plays, is not allowed to freely improvise, and has fairly constrainedchoices in terms of accompaniment selection However, as noted earlier,the secondary also has some degree of freedom in playing Within theconstraints imposed by the lead, the secondary percussionist may:

• Proactively suggest variations for the lead to incorporate

• Improvise by making references to earlier changes played by the lead

• Improvise somewhat freely in the last bar of an improvisational cycle

• Play complementary accompaniment using off-beat strokes, rolls,changes to the accent structure of the accompaniment rhythm, etc.Note that there is also one exceptional case whereby the secondarylargely ignores the lead percussionist and instead follows the melody di-rectly This is only justified if the lead is playing the same rhythmic pat-terns without variation, but even so, such a decision is controversial andproblematic for a number of reasons The work detailed in this documentdoes not attempt to handle this case

5.2 Musical structures

Musical improvisations occur in cycles of many different lengths, such as 10,

12, 14, 32, or 64 beats This document considers improvisations that takeplace within a 32 beat (or 64 beat) improvisation cycle which is known asthe Adi talam in Carnatic music Since the structure of improvisations thathappen in a 64 and 32 beat improvisational cycle are similar, for simplicity’ssake the descriptions are provided for a 32 beat improvisational cycle Theimprovisation cycle is further divided into different bars In a 4/4 time

signature, each bar consists of four beats The numerator of 4/4 denotesthe number of beats in a bar and the denominator denotes that each beathas a duration of a quarter note Similarly, in an 8/4 time signature, eachbar contains eight beats each of quarter note duration Improvisationalcycles of 64 beats usually contain 8 bars in an 8/4 time signature Figure5.2 shows a rhythm pattern of two-bar duration, containing eight lead andsecondary hits.

Figure 5.2: Two bars of lead and secondary playing

In an improvisational cycle, typically the lead introduces a rhythmicgroove in the first two bars, plays variations of the groove in the followingbars and either syncopates or intensifies the groove in the final bar Thereare three different styles of accompaniment that the secondary can play.They are:

• Compliant accompaniment: the secondary complies with the actions

of the lead and closely matches the different changes played by thelead

• Interactive accompaniment: the secondary introduces changes in theaccompaniment by referring to the past actions of the lead

• Proactive accompaniment: the secondary plays complementary rhythmpatterns, supplements certain hits of the lead, matches the musicalstructure of the melody, etc

Each of these accompaniment scenarios impose different demands andconstraints on the secondary percussionist and result in very different kinds

of musical choices and decisions

5.3 Choices in different styles of

accompani-ment playing

In compliant accompaniment playing style, the secondary has the mostconstraints and the fewest choices in terms of accompaniment playing Ba-sically, the secondary must strictly follow the changes in the bar activityand loudness of the lead Within those constraints, the secondary is able

to make some discretionary choices about diction, note duration, and ness.

loud-In interactive accompaniment playing, the secondary has the freedom

to deviate from the lead but is constrained by the type of deviations played

by the lead Deviations played by the secondary are typically in the form

of changes in the bar activity and loudness In rare cases, the deviationsinclude major structural changes (e.g., violating bar boundaries, grouping

of hits) In this style of playing, the secondary is allowed to deviate fromthe lead but is restricted to the deviations played by the lead in earliercycles of the improvisation

In proactive accompaniment playing, the secondary percussionist hasthe maximum freedom to deviate from the lead The secondary is free tochange the bar activity, loudness and musical structure (alternate groupings

of hits, changing the lead’s groove) of the accompaniment The deviationsare constrained by their appropriateness to the melodic structure Thistype of improvised behavior is very discretionary and depends greatly onthe aesthetic sense and skill of the secondary percussionist

Transcriptions of representative examples of accompaniment playing areprovided in the appendix

5.4 Musical actions in the improvisation

This section describes the musical actions corresponding to the differentvariations in improvisations played by the lead and secondary percussion-ists There are broadly two kinds of variations that the lead and secondaryplay while improvising in performances: major variations and minor vari-ations The main distinction between the two is that major variationseither change or obstruct the flow of the rhythmic groove whereas minorvariations are considered as variations around the rhythmic groove

5.4.1 Major variations

Major variations are variations in playing that obstruct the groove Inperformance situations, these are usually played to align the rhythm struc-ture with the melodic structure The actions in major variations includeplaying rolls, interspersing pauses with rolls, changing the grouping of hits,speed-doubling and playing sequences that violate bar-boundaries Thelead introduces major variations in order to minimize the sense of repeti-tion between the different bars, and play accompaniment that better suits

the melody The secondary employs major variations in sections of active and proactive accompaniment playing Major variations in the leadmake the groove less predictable for the secondary Usually, the secondarypercussionist keeps silent when the lead plays major variations and joinsthe lead after the major variations are over The major variations are asfollows:

inter-• Rolls Rolls are rhythm patterns that are filled with double-hits orquadruple-hits They are played to break the flow of an existinggroove and start a new groove Rolls are difficult to anticipate ifthey have not been played before Usually the secondary percussion-ist stays silent when the roll is played the first time However, thesecondary plays the roll if he is able to predict the occurrence of theroll before the lead plays or he is able to predict the continuation ofthe roll after listening to a few beats

• Pauses They occur in a specific scenario where the lead leaves pauses

of unequal duration in the different bars and occasionally intersperseswith rolls between the pauses This makes predictability of the leaddifficult for the secondary percussionist In this case, the secondarypercussionist is usually silent but joins with the lead when the lead’splaying becomes predictable

• Speed doubling In speed doubling, an entire groove is played attwice its speed Speed doubling is common while playing for melodicphrases at fast tempos and is repeated for one improvisation cycle.Secondary percussionists either predict the moments of speed dou-bling in a performance or they join after the lead doubles the speed

• Violation of bar-boundaries Bar-boundaries are violated whenrhythm patterns of durations greater (or less) than one bar are repeti-tively played There are certain types of bar-boundary violations thatare more common than others The secondary percussionist initiallystays silent to predict the type of bar-boundary violation He joinsthe lead once he predicts the type of bar-boundary violation

The lead plays major variations in some parts of the concert, but theunpredictability in playing makes it difficult for the secondary percussion-ist to follow them It is much easier to follow the lead when he playspredictably There are other sections of the performance when the leadplays predictably but provides variations using minor variations Thesevariations are more predictable and easier to follow for the secondary Thisthesis addresses the accompaniment played by the secondary when the lead

plays with minor variations It will not address accompaniment issues thatarise when the lead performs major variations (and the remainder of thisdocument ignores those variations).

V ar2 − Loudness : ta #tum ta #tum ta tum tum ta

V ar3 − P itchBend : ’tum ’tum ’tum tum ta tum tum ta

V ar4 − Doubletaps : ta ta-te ta-te tum ta tum tum ta

Figure 5.3: Different minor variations

The lead percussionist also performs the same actions while playing nor variations The different minor variations with their associated actionsare:

mi-• Diction

• Intonation

• Loudness (Emphasis)

It is important to note that the pauses and double taps mentioned hereare different from pauses and speed doubling mentioned in major variations.The duration, position, and occurrence of the pauses in the minor variationsare fairly repetitive, predictable, and help to establish the groove Doubletaps, by contrast to speed doubling, are changes in note duration of one ortwo hits in a bar in which an entire groove is played at double speed.The lead and secondary percussionist follow these same actions to playminor variations, but the implementation differs slightly due to the differentways that the instruments are played The physical constraints of playingare some of the reasons for this difference In particular, the lead playersplay strokes with both hands whereas the secondary plays strokes withone hand One consequence of this, is that pitch bending, for example,

is performed very differently by the lead and by the secondary Anotherconsequence is that lead players can typically play multiple sounds at thesame time by simultaneously playing the hits on both hands, increase theintensity of the groove, and provide more variation Secondary players,however, are much more restricted in the kind of the grooves and variationsplayed