Commodity Trading Advisors: Risk, Performance Analysis, and Selection Chapter 5 pdf

Tobit regressions ofefficiency scores on equity betas, beta-squared, fund size, length of managertrack record, investment style market focus, and strategy discretionary vs.systematic are

CHAPTER 5 CTA Performance Evaluation with Data Envelopment Analysis Gwenevere Darling, Kankana Mukherjee, and Kathryn Wilkens

We apply data envelopment analysis to a performance evaluation work for CTAs The technique allows us to integrate several perform-ance measures into one efficiency score by establishing a multidimensionalefficient frontier Two dimensions of the frontier are consistent with thestandard Markowitz mean-variance framework, while additional risk andreturn dimensions include skewness and kurtosis We also illustrate amethod of analyzing determinants of efficiency scores Tobit regressions ofefficiency scores on equity betas, beta-squared, fund size, length of managertrack record, investment style (market focus), and strategy (discretionary vs.systematic) are performed for CTA returns over two time frames represent-ing different market environments We find that the efficiency scores arenegatively related to beta-squared in both time periods Results also indi-cate that emerging CTAs (those with shorter manager track records) tend tohave better efficiency scores as defined by the DEA model used in our study.This relationship is strongest during the period from 1998 to 2000, but notstatistically significant during the period from 2000 to 2002 For both timeperiods, fund size is not related to efficiency scores

frame-INTRODUCTION

Industry performance reports for commodity trading advisors (CTAs)present multiple performance measures such as return, standard deviation,drawdowns, betas, and alphas Investors and fund managers recognize theimportance of considering a multitude of performance measures to analyzefund risk from various perspectives It is particularly important for thegrowing alternative investment class of managed futures, which have dif-

79

ferent risk/return profiles from those of traditional mutual funds as well asthose of many hedge fund strategies For all asset classes, however, the aca-demic literature has done little to offer a comprehensive framework thatincorporates multiple risk measures in an integrated fashion (Arnott2003) Too often, studies focus on single measure of risks, arguing for onerelative to another

“Managed futures” are a subset of hedge funds that uses futures tracts as one among several types of trading instruments (including swapsand interbank foreign exchange markets) and for which futures are ameans, rather than an end, with which to implement their strategy Thename wrongly suggests that futures are the dog rather than the tail Man-aged futures encompass the broad set of individual commodity tradingadvisors (CTAs) CTAs are also unfortunately named because, on balance,most of their trading is in the financial markets, not the commodity mar-kets Like any other class of alternative investments, managers are repre-sented by a variety of styles and substyles For example, there are systematicand discretionary CTAs, CTAs who exclusively try to capture trends, thosewho identify countertrend opportunities, and those who combine the twoapproaches.1

con-In this study we look at the performance of CTAs based on multiple criteriausing data envelopment analysis (DEA) DEA establishes a multidimensionalefficient frontier and assigns each CTA an efficiency score whereby 1 (or 100percent) indicates perfect efficiency and scores lower than 1 represent rela-tively less efficient CTAs based on the performance criteria chosen

The criteria we choose as bases for performance evaluation are monthlyreturns, kurtosis, minimum return, skewness, standard deviation of returns,and percentage of negative monthly returns Although there are many otherpossibly appropriate criteria, those not included here are likely either to beredundant with variables included or to not make sense in an optimizationframework Criteria that make sense in this framework are those that aredesirable to maximize or minimize across various market conditions Thisaspect leads us to reject equity betas as a criterion in the DEA model, forexample, because CTAs may desire a higher beta in up-market environ-ments but negative betas in down-market periods

In addition to applying the DEA methodology to evaluate CTA formance, we explore the relationship between the efficiency scores andfund size, investment style and strategy, length of the manager’s track

short-term, and medium-term traders and those who combine time frames.

record, and measures of the covariance of CTA returns with equity marketreturns We ask:

■ Do emerging hedge fund managers2really do better than larger, lished managers?

estab-■ Is there a relationship between efficiency scores and equity markets,and if so, does the market environment impact the relationship?

■ Do strategies (systematic, discretionary, trend-based) or styles fied, financial, currency, etc.) matter in different market environments?

(diversi-We analyze monthly CTA returns in two different market ments: over 24 months beginning in 1998, when equity market returns arepredominantly positive, and over 24 months beginning in 2000, when theyare more often negative We find that emerging managers perform betterthan well-established managers in the sense that funds with shorter trackrecords have a greater efficiency score Fund size and manager tenure areweakly positively correlated In contrast with the conventional wisdom,however, larger funds have better efficiency scores These results providesome insight into capacity issues concerning optimal fund size The fundsize and manager tenure coefficients are, however, statistically significantonly during the first (1998–2000) time period, indicating that capacityissues may be less important during flat equity markets

environ-For both time periods, squared equity beta is inversely related to theefficiency scores and the coefficient is highly significant This result appears

to be influenced by the risk-minimizing design of our DEA model The styledummy variable (diversified versus nondiversified) was not a significant fac-tor impacting efficiency scores The systematic strategy variable was signif-icant, but only during the second (2000–2002) down-market period Weconsider these results as preliminary because several issues may be affectingtheir significance Notably, when our sample size is broken down by invest-ment style and strategy, the number of CTAs representing each group is verysmall Nevertheless, we believe that the approach is a promising avenue forfurther research

The next section of this chapter provides a background discussion on ious risk measures and performance evaluation issues The variables chosen

var-as inputs to the DEA model and the regression model are then discussed inthe context of prior research, and the data are described The variable descrip-

CTA Performance Evaluation with Data Envelopment Analysis 81

2 We consider managers with short track records to be emerging CTAs This gory is distinctly different from managers who invest in emerging markets.

cate-tion is followed by an explanacate-tion of the DEA methodology and Tobit sions used to explore determinants of the efficiency scores obtained from theDEA model Results are presented and the final section concludes.

regres-RISK MEASURES AND PERFORMANCE EVALUATION

A multitude of investment fund performance models and metrics exist in partbecause some measures are more appropriate for certain purposes than others.For example, the Sharpe ratio is arguably more appropriate when analyzing

an entire portfolio, while the Treynor ratio is appropriate when evaluating asecurity or investment that is part of a larger portfolio.3The multitude of per-formance measures and approaches also suggests that more than one meas-ure of risk may be needed to accurately assess performance Conversely, somemeasures can be redundant For example, Daglioglu and Gupta (2003b) findthat returns of hedge fund portfolios constructed on the basis of some riskmeasures are often highly correlated, and sometimes perfectly correlated,with returns of portfolios constructed on the basis of others Burghart, Dun-can, and Liu (2003) illustrate that the theoretical distribution of drawdownscan be replicated with a high degree of accuracy given only a manager’s aver-age return, standard deviation of returns, and length of track record

In this section we begin by briefly reviewing some of the traditionalportfolio performance measures and analysis techniques We review singleparameter risk measures based on modern portfolio theory, we discussexpanded performance models that account for time-varying risk, discussconcerns over assuming mean-variance sufficiency, and consider multifactormodels of style and performance attribution This short review exposes aplethora of performance measures The question of appropriateness andredundancy is revisited in the section that describes the data used in thisstudy The current section also discusses the seemingly paradoxical issue ofusing benchmarks to evaluate absolute return strategies4 and concludeswith a discussion of potential determinants of performance

Alpha and Benchmarks

Traditional asset managers seek to outperform a benchmark, and their formance is measured relative to that benchmark in terms of an alpha

standard deviation, or total risk, is in the denominator whereas beta is the nator of the Treynor measure, and beta measures the systematic risk that will con- tribute to the risk of a well-diversified portfolio.

In contrast, relative return strategies seek only to outperform a benchmark.

While CTAs follow absolute return strategies that seek to make positivereturns in all market conditions, benchmarks now exist for CTAs and otherhedge fund strategies Before considering benchmarks for absolute returnstrategies, we first review the concepts in the context of traditional assetmanagement Jensen’s (1968) alpha is generally a capital asset pricing model(CAPM)-based performance measure of an asset’s average return in excess

of that predicted by the CAPM, given its systematic risk (beta)5 and themarket (benchmark) return Alphas also may be measured relative to addi-tional sources of risk in multi-index models

Whereas various single-index models are based on the CAPM andassume that security returns are a function of their co-movements6with themarket portfolio, multi-index (or multifactor) models assume that returnsare also a function of additional influences.7For example, Chen, Roll, andRoss (1986) develop a model where returns are a function of factors related

to cash flows and discount rates such a gross national product and tion The purposes of multi-index models are varied and, in addition toperformance attribution, include forming expectations about returns andidentifying sources of returns

infla-Sharpe (1992) decomposes stock portfolio returns into several “style”factors (more narrowly defined asset classes such as growth and incomestocks, value stocks, high-yield bonds) and shows that the portfolio’s mixaccounts for up to 98 percent of portfolio returns Similarly, Brinson,Singer, and Beebower (1991) show that rather than selectivity or markettiming abilities, it is the portfolio mix (allocation to stocks, bonds, andcash) that determines over 90 percent of portfolio returns However, Brownand Goetzmann (1995) identify a tendency for fund returns to be correlatedacross managers, suggesting performance is due to common strategies thatare not captured in style analysis

Schneeweis and Spurgin (1998) use various published indexes man Sachs Commodity Index, the Standard & Poor’s 500 stock index, the

(Gold-CTA Performance Evaluation with Data Envelopment Analysis 83

deviation of returns Tobin (1958) extended the Markowitz efficient frontier by adding the risk-free asset, resulting in the capital market line (CML) and paving the way for the development of the capital asset pricing model, developed by Sharpe (1964), Lintner (1965), and Mossin (1966) The CAPM defines systematic risk,

diver-sify away the remaining portion

portfolio, however, as noted earlier, they can attempt to describe coskewness and cokurtosis as well.

multi-index model can be an equilibrium description (Ross, 1976)

Salomon Brothers government bond index, and U.S dollar trade-weightedcurrency index, the MLM Index8) with absolute S&P 500 returns andintramonth S&P return volatility in a multifactor regression analysis todescribe the sources of return to hedge funds, managed futures, and mutualfunds The index returns employed in the regression analysis are intended

to be risk factors that explain the source of natural returns The tory variable, absolute equity returns, captures the source of return thatderives from the ability to go short or long Returns from the use of options

explana-or intramonth timing strategies are proxies fexplana-or the intramonth standarddeviation The MLM Index, an active index designed to mimic trend-following strategies, is used to capture returns from market inefficiencies inthe form of temporary trends

Seigel (2003) provides a comprehensive review of benchmarking andinvestment management Despite the fact that CTAs and many hedge fundmanagers follow absolute return strategies, various CTA benchmarks nowexist, as described by Seigel (2003)

Addressing Time-Varying Risk

Single-parameter risk measures are problematic if managers are changingfund betas over time, as they would if they were attempting to time the mar-ket For example, when equity prices are rising, the manager might increasethe fund’s beta and vice versa Although market risk can be measured if theportfolio weights are known, this information is generally not publiclyavailable and other techniques must be employed.9

index methodology that involves changing (commodity) market sides long and short

to measure economic return.

model to capture nonlinearities in beta resulting from market timing activities Kon and Jen (1978, 1979) use a switching regression technique Merton (1981) and Hen- riksson and Merton (1981) develop nonparametric and parametric option-based methods to test for directional market timing ability The nonparametric approach requires knowledge of the managers’ forecasts The more commonly employed parametric approach involves adding an extra term to the usual linear regression model and is CAPM based Ferson and Schadt (1996) note that fund betas may change in response to changes in betas of the underlying assets as well as from changing portfolio weights They modify the classic CAPM performance evaluation techniques to account for time variation in risk premiums by using a conditional CAPM framework This method removes the perverse negative performance often found in earlier tests and suggests that including information variables in perform- ance analysis is important

Mitev (1998) uses a maximum likelihood factor analysis technique toclassify CTAs according to unobservable factors Similarly, Fung and Hsieh(1997b) also use a factor-analytic approach to classify hedge funds In bothcases, the results identify general investment approaches or trading strate-gies (e.g., trend-following, spread strategies, or systems approaches) assources of returns to these alternative investment classes Factor analysisand multifactor regression analysis differ in their approach to identifyingthe factors (benchmarks) that serve as proxies for risk In multifactorregression analysis, the factors are specified in advance Factor analysis willidentify funds that have common yet unobservable factors, although thefactors can be inferred from the qualitative descriptions of the funds Whilethis may seem redundant, the clustering of funds is done independently ofthe qualitative descriptions in a formal data-driven process

The data envelopment analysis methodology used in this chapter, anddescribed in more detail in Wilkens and Zhu (2001, 2004), incorporatesmultiple criteria and “benchmarks” funds or other securities according tothese criteria This is distinctly different from multifactor analysis Herebenchmarks are not risk factors but rather are efficient securities as defined

in n dimensions where each dimension represents risk and return criteria.

Recently Gregoriou (2003) used the DEA method in the context of marking hedge funds

bench-Skewness and Kurtosis:

Questioning Mean-Variance Sufficiency

The standard CAPM framework assumes that investors are concerned withonly the mean and variance of returns Ang and Chau (1979) argue thatskewness in returns distributions should be incorporated into the perform-ance measurement process Even if the returns of the risky assets within aportfolio are normally distributed, dynamic trading strategies may producenonnormal distributions in portfolio returns Both Prakash and Bear (1986)and Stephens and Proffitt (1991) also develop higher-moment performancemeasurements

Fishburn (1977), Sortino and van der Meer (1991), Marmer and Ng(1993), Merriken (1994), Sortino and Price (1994), and others also havedeveloped measures that take into account downside risk (or semivariance)rather than the standard deviation of returns Although some differencesexist among these measures, the Sortino ratio captures their essence.Whereas the Sharpe ratio is defined as excess return10divided by standard

CTA Performance Evaluation with Data Envelopment Analysis 85

deviation, the Sortino ratio is defined as return divided by downside tion Downside deviation (DD) measures the deviations below some mini-mal accepted return (MAR) Of course, when the MAR is the averagereturn and returns are normally distributed, the Sharpe and Sortino ratioswill measure the same thing Martin and Spurgin (1998) illustrate that even

devia-if individual asset or fund returns are skewed, the skewness tends to bediversified away at the portfolio level However, they also illustrate thatmanagers may choose to follow strategies that produce skewed returns as aform of signaling their skill Note that coskewness remains irrelevant if itcan be diversified away, but skewness may have some signaling value Addi-tionally, the popularity of the related value at risk (VaR) measure11and thecommon practice of reporting drawdown12 information for various alter-native investments suggest that skewness may be important, whether interms of investor utility or skill signaling

Beta-Squared Coefficient The classic paper by Fama and MacBeth (1973),and several other early papers (e.g., Carroll and Wei 1988; Shanken 1992)empirically test a two-pass regression methodology for stock returns.Assuming a nonlinear relationship between stock returns, the tests includebeta-squared in the second-pass regression These tests find that the coeffi-cient for beta-squared is negative and statistically significant, providing evi-dence of a nonlinearity in stock returns

Schneeweis and Georgiev (2002, p 7) provide evidence that CTAs havenonlinear returns with respect to the equity market: “When S&P 500returns were ranked from low to high and divided into four thirty-threemonth sub-periods, managed futures offered the opportunity of obtainingpositive returns in months in which the S&P 500 provided negative returns

as well as in months in which the S&P 500 reported positive returns.”

We include equity beta-squared in our Tobit regressions where thedependent variable is not the expected return of the CTA, but is rather the efficiency score obtained in the DEA models Although the dependentvariable is not the same as in the earlier stock studies, we might hypothe-size that CTA efficiency scores are also negatively related to beta-squared

period and is defined as the return from a fund’s net asset value peak to trough The Calmar ratio is a similar measure that CTA investors are often interested in and is defined as the average annual return over the past three years divided by the absolute value of the maximum drawdown during that period.

We infer a direct correspondence between the efficiency score and expectedreturn The CTA returns observed by Schneeweis and Georgiev (2002),therefore, imply a positive coefficient Finally, we note that the efficiencyscores used in this study minimize variability This leads to the hypothesisthat the beta-squared coefficient is negatively correlated with the efficiencyscore, unless the enhanced return from high (absolute) betas is an offset-ting factor

Fund Size In his chapter “The Lure of the Small,” Jaeger (2003) describeshow small firms and small portfolios are desirable features of hedge funds.Small firms satisfy hedge fund managers’ entrepreneurial spirit, and smallportfolios are often necessary to enable hedge funds to implement theirstrategies, especially if they trade in markets that are sometimes illiquid.Gregoriou and Rouah (2002) find, however, that fund size does not matter

to hedge fund performance Being a subset class of hedge funds, CTAs areexamined in this chapter to see if fund size or length of manager trackrecord is related to the DEA efficiency scores

Determinants of Performance Based on the discussion above, we choose asbases for performance evaluation in a DEA model monthly returns, kurto-sis, minimum return, skewness, standard deviation of returns, and percent-age of negative monthly returns We then investigate the potential of fundsize, length of track record, strategy, and style to impact performance scores

of funds created by the DEA model

DATA DESCRIPTION

Monthly CTA return data for 216 CTAs over two periods surroundingMarch 2000 are obtained from the Center for International Securities andDerivatives Markets (CISDM) Alternative Investment Database.13The firstperiod is an up-market period for the equity market (March 31, 1998, toFebruary 28, 2000) and the second period is a down market environment(April 30, 2000, to March 31, 2002) The daily high for the S&P 500occurred in March 2000, as illustrated in Figure 5.1 The mean monthlyreturn for the S&P 500 was 1.28 percent and −1.11 percent for the first andsecond periods, respectively

CTA Performance Evaluation with Data Envelopment Analysis 87

investment styles and strategies.

Performance criteria used in the DEA model were calculated from theCTA returns for each of the two periods The DEA approach to “estimat-ing” the efficient frontier is a nonstatistical approach As a result, all devi-ations from the efficient frontier are measured as inefficiency (i.e., there is

no allowance for statistical noise) The efficiency measures obtained fromthis method are, therefore, very sensitive to the effect of outliers Hence, foreach performance criterion used in the DEA model, particular effort wasmade to detect any outliers CTAs with outliers in one subperiod weredeleted from both subperiods so as to have the same group of CTAs Ourfinal sample consisted of 157 CTAs that were used for analysis in the DEAmodel and the subsequent Tobit regression analysis Table 5.1 providesdescriptive statistics for the DEA model criteria over both periods and forthe full and final sample

Other information we use from the CISDM Alternative InvestmentDatabase includes the assets under management over time, the datesthe funds were established, and information on the investment style14

750 850 950 1,050 1,150 1,250 1,350 1,450 1,550

Date 11-Mar-98 18-May-98 24-Jul-98 30-Sep-98 7-Dec-98 16-Feb-99 23-Apr-99 30-Jun-99 7-Sep-99 11-Nov-99 20-Jan-00 28-Mar-00 5-Jun-00 10-Aug-00 17-Oct-00 22-Dec-00 5-Mar-01 10-May-01 18-Jul-01 28-Sep-01 5-Dec-01

14-Feb-02 24-Apr-02 1-Jul-02 6-Sep-02 12-Nov-02

FIGURE 5.1 S&P 500 Daily Closing Values, from 1998 to 2002

14 We follow the terminology established by Sharpe (1992) and call the market focus investment style.

(agriculture, currencies, diversified, financial, and stocks) and strategy(discretionary, systematic, and trend-based15) of the fund The diversifiedinvestment style is most common, accounting for 59 percent of the CTAs

in our final sample, as illustrated in Table 5.2 Comprising 66 percent ofour final sample, the systematic investment strategy is the most common,

as indicated in Table 5.3 Table 5.4 describes the distribution of thelength of the managers’ track record (maturity) in years, and Table 5.5 pre-sents the distribution of the average funds under management for the two periods

Table 5.6 presents correlation coefficients for the DEA model criteria

We see that in both periods, minimum return and standard deviation arehighly (negatively) correlated, as one might expect Kurtosis and skewnessare also highly (positively) correlated, but only in the first period We notethat we are therefore potentially including redundant information in themodel That is, by maximizing the minimum return, we may not necessar-ily need to minimize correlated measures such as the standard deviation.Following Daglioglu and Gupta (2003b), however, we sort the portfolios bythe various performance criteria and find that the returns to the sorted port-

TABLE 5.2 Number of CTAs, by Investment Style

CTA Performance Evaluation with Data Envelopment Analysis 91

TABLE 5.3 Number of CTAs, by Investment Strategy

After computing efficiency scores with the DEA methodology described

in the following section, determinants of the scores are explored by ing them against four additional variables: beta, beta-squared, averagefunds managed, and length of manager track record Table 5.8 presents thesummary statistics for these variables