# Latest News

## Certificate in Quantitative Investment (CQI)

Posted by on Jul 15, 2013 in News | 0 comments

Numerical Method Inc. has the vision to promote rational investment and trading. Jointly organized with top universities, we are offering a 6-months 9-courses program that teach mathematics, programming and quantitative or algorithmic trading. We invite famous and established traders from Wall Street banks and funds to share their experience. Students may choose to participate in classroom or online. More information can be found here: http://en.cqi.sg

## Solving the "Corner Solution Problem" of Portfolio Optimization

Posted by on Jun 19, 2013 in AlgoQuant, Algorithmic Trading, Investment | 0 comments

Many portfolio optimization methods (e.g., Markowitz/Modern Portfolio Theory in 1952) face the well-known predicament called the “corner portfolio problem”. When short selling is allowed, they usually give efficient allocation weighting that is highly concentrated in only a few assets in the portfolio. This means that the portfolio is not as diversified as we would like, which makes the optimized portfolio less practically useful.

In [Corvalan, 2005], the author suggests to look for instead an “almost efficient” but “more diversified” portfolio within the close neighborhood of the Mean-Variance (MV) optimal solution. The paper shows that there are many eligible portfolios around the MV optimal solution on the efficient frontier. Specificially, given the MV optimal solution, those “more diversified” portfolios can be computed by relaxing the requirements for the portfolio return $R$ and risk $\sigma$ in an additional optimization problem:

max_{w} D(w) \ \ \textup{s.t.,} \\ \begin{aligned} \sqrt{w' \Sigma w} & \le \sigma^* + \Delta \sigma \\ R^* - \Delta R & \le w'r \\ w' 1 & = 1 \\ w_i & \ge 0 \end{aligned}

where $( \sigma^* , R^* ) = ( \sqrt{w^{*'} \Sigma w^*} , w^{*'} r )$, $w^*$ is the Markowitz MV optimal weights, $\Delta \sigma, \Delta R$ are the relaxation tolerance parameters, and $D(w)$ is a diversification measure for the portfolio (for example, $\sum_i w_i, ln(w_i)$, $\prod_i w_i$). In other words, the new optimization problem looks for a portfolio with the maximal diversification around the optimal solution.

Corvalan’s approach can be extended to create an approximate, sufficiently optimal and well diversified portfolio from the optimal portfolio. The approximate portfolio keeps the constraints from the original optimization problem.

References:

## Change of Measure/Girsanov’s Theorem Explained

Posted by on May 16, 2013 in Seminars | 1 comment

Change of Measure or Girsanov’s Theorem is such an important theorem in Real Analysis or Quantitative Finance. Unfortunately, I never really understood it until much later after having left school. I blamed it to the professors and the textbook authors, of course. The textbook version usually goes like this.

Given a probability space ${\Omega,\mathcal{F},P}$, and a non-negative random variable Z satisfying $\mathbb{E}(Z) = 1$ (why 1?). We then defined a new probability measure Q by the formula, for all $A in \mathcal{F}$.

$Q(A) = \int _AZ(\omega)dP(w)$

Any random variable X, a measurable process adapted to the natural filtration of the $\mathcal{F}$, now has two expectations, one under the original probability measure P, which denoted as $\mathbb{E}_P(X)$, and the other under the new probability measure Q, denoted as $\mathbb{E}_Q(X)$. They are related to each other by the formula

$\mathbb{E}_Q(X) = \mathbb{E}_P(XZ)$

If $P(Z > 0) = 1$, then P and Q agree on the null sets. We say Z is the Radon-Nikodym derivatives of Q with respect to P, and we write $Z = \frac{dQ}{dP}$. To remove the mean, μ, of a Brownian motion, we define

$Z=exp \left ( -\mu X - \frac{1}{2} \mu^2 \right )$

Then under the probability measure Q, the random variable Y = X + μ is standard normal. In particular, $\mathbb{E}_Q(X) = 0$ (so what?).

This text made no sense to me when I first read it in school. It was very frustrated that the text was filled with unfamiliar terms like probability space and adaptation, and scary symbols like integration and $\frac{dQ}{dP}$. (I knew what $\frac{dy}{dx}$ meant when y was a function and x a variable. But what on earth were dQ over dP?)

Now after I have become a professor to teach students in finance or financial math, I would get rid of all the jargon and rigorousness. I would focus on the intuition rather than the math details (traders are not mathematicians). Here is my laymen version.

Given a probability measure P. A probability measure is just a function that assigns numbers to a random variable, e.g., 0.5 to head and 0.5 to tail for a fair coin. There could be another measure Q that assigns different numbers to the head and tail, say, 0.6 and 0.4 (an unfair coin)! Assume P and Q are equivalent, meaning that they agree on what events are possible (positive probabilities) and what events have 0 probability. Is there a relation between P and Q? It turns out to be a resounding yes!

Let’s define $Z=\frac{Q}{P}$. Z here is a function as P and Q are just functions. Z is evaluated to be 0.6/0.5 and 0.4/0.5. Then we have

$\mathbb{E}_Q(X) = \mathbb{E}_P(XZ)$

This is intuitively true when doing some symbol cancellation. Forget about the proof even though it is quite easy like 2 lines. We traders don’t care about proof. Therefore, the distribution of X under Q is (by plugging in the indicator function in the last equation):

$\mathbb{E}_Q(X \in A) = \mathbb{E}_P(I(X \in A)Z)$

Moreover, setting X = 1, we have (Z here is a random variable):

$\mathbb{E}_Q(X) = 1 = \mathbb{E}_P(Z)$

These results hold in general, especially for the Gaussian random variable and hence Brownian motion. Suppose we have a random (i.e., stochastic) process generated by (adapted to) a Brownian motion and it has a drift μ under a probability measure P. We can find an equivalent measure Q so that under Q, this random process has a 0 drift. Wiki has a picture that shows the same random process under the two different measures: each of the 30 paths in the picture has a different probability under P and Q.

The change of measure, Z, is a function of the original drift (as would be guessed) and is given by:

$Z=exp \left ( -\mu X - \frac{1}{2} \mu^2 \right )$

For a 0 drift process, hence no increment, the expectation of the future value of the process is the same as the current value (a laymen way of saying that the process is a martingale.) Therefore, with the ability to remove the drift of any random process (by finding a suitable Q using the Z formula), we are ready to do options pricing.

Now, if you understand my presentation and go back to the textbook version, you should have a much better understanding and easier read, I hope.

References:

## Mean-Variance Portfolio Optimization When Means And Covariances Are Unknown

Posted by on Feb 16, 2013 in AlgoQuant, Algorithmic Trading | 1 comment

[Lai, Xing and Chen, 2010], in the paper “Mean-Variance Portfolio Optimization When Means And Covariances Are Unknown”, proposed a ground breaking method to do portfolio optimization. In what follows we summarize their idea and use it to implement a periodic rebalancing strategy based on the AlgoQuant framework.

efficient frontier

Harry Markowitz won the Nobel prize for his work in mean-variance (MV) portfolio optimization in 1950s. The theory is widely regarded as fundamental in financial economics. It says, given a target return of a portfolio of m assets, the optimal (in terms of information ratio) weighting $w_{eff}$ is given by

$w_{eff}=\arg \min_w w^T \Sigma w \text{ subject to } w^T\mu = \mu_*, w^T1 = 1$

where $\mu$ is the expected future returns, and $\Sigma$ is the expected covariance matrix of future returns. This problem is readily solved by quadratic programming.

Nonetheless, the assumption that $\mu$ and $\Sigma$ are known in advance is very dubious. This has been referred to as the “Markowitz optimization engima”. The attempts made so far are to better forecast these estimators, namely $\hat{\mu}$ and $\hat{\Sigma}$, as accurately as possible. The maximum likelihood estimate (MLE) from the training sample is an example. It turns out, however, that MLE performs poorly because the estimators are quite different from the realized values [Michaud, 1989]. Since then, three approaches have been proposed to address the difficulty. The first approach uses a multi-factor model to reduce the dimensionality in estimating $\Sigma$ [Fan, Fan and Lv, 2008]. The second approach uses Bayes or other shrinkage estimates of $\Sigma$ [Ledoit and Wolf, 2004]. Both approaches attempt to use improved estimates of  $\Sigma$ for the plug-in efficient frontier. They have also been modified to provide better estimates of $\mu$, for example, in the quasi-Bayesian approach of [Black and Litterman, 1990]. The third approach uses bootstrapping to correct for the bias of $\hat{w}_{eff}$ as an estimate of $w_{eff}.$

The innovation in [Lai, Xing and Chen, 2010] is to move away from the traditional approach of finding better estimators. They instead assume inherent uncertainty, hence risk, in these estimators. In other words, because the mean and covariance matrix of future returns are estimated from past returns, the uncertainties of these estimators should be incorporated into the portfolio risk. The authors formulate a stochastic optimization framework which solves a more fundamental problem. Given the past returns $r_1, \ldots, r_n$,

maximize $\left \{ E(w^Tr_{n+1}) - \lambda Var(w^Tr_{n+1}) \right \}$

where $w^Tr_{n+1}$ is now regarded as a random variable conditional on the past returns, and $\lambda$ is a risk-aversion index (investor’s preference).

This stochastic optimization problem is not standard due to the $E(.)^2$ term in $Var(.)$ ($Var(X) = E[X^2] - (E[X])^2$). To tackle this problem, the authors solve instead an equivalent stochastic optimization problem:

$\max_{\eta} {E[w^T(\eta)r_{n+1}] - \lambda Var[w^T(\eta)r_{n+1}]}$

where

$w(\eta) = \arg \min_w {\lambda E[(w^Tr_{n+1})^2] - \eta E(w^Tr_{n+1}) }$

and

$\eta = 1 + 2\lambda E(W_B)$

Next, define the function

$C_{\lambda,\mu_n,V_n}(\eta) = E[w^T(\eta)\mu_n] + \lambda (E[w^T(\eta)\mu_n])^2 - \lambda E[w^T(\eta) V_n w(\eta)]$

where $\mu_n$ and $V_n$ are the first and second moments of returns. The problem now becomes maximizing $C_{\lambda,\mu_n,V_n}(\eta)$ which can be achieved by any univariate maximization algorithm such as Brent’s method (BrentMinimizer in SuanShu can be used to minimize the negative of this function).

The remaining task is to estimate $\mu_n$ and $V_n$ to plug into $C_{\lambda,\mu_n,V_n}(\eta)$. $\mu_n$ can be estimated by, for example, bootstrapping, a multi-factor model, a GARCH model, etc. Similarly, $V_n = \Sigma + \mu_n \mu_n^T$ can be estimated by first estimating the sample covariance matrix $\Sigma$. As $\Sigma$ may be unstable when the sample size is small, our implementation uses a covariance-shrinkage method described in [Ledoit and Wolf 2004].

Putting all these steps together, the algorithm to compute the optimal weighting looks like this:

1. Estimate $\mu_n$ and $V_n$ from past returns
2. Estimate $a = E[w^T(\eta)\mu_n], b = E[w^T(\eta)V_nw(\eta)]$
3. Construct the univariate real function $C_{\lambda,\mu_n,V_n}(\eta) = a + \lambda a^2 - \lambda b$
4. Maximize $C_{\lambda,\mu_n,V_n}(\eta)$ over $\eta$
5. Compute the optimal weighting $w^* = w(\eta^*)$ using the optimal $\eta^*$

We have implemented this algorithm in AlgoQuant as a component/module/signal. Using our component based programming paradigm, it is easy to modify the steps in the implementation without a lot of changes to the code. For instance, we can pass in the constructor of Lai2010NPEBModel a GridSearchCetaMaximizer object instead of a BrentCetaMaximizer object to change the algorithm to maximize $C(\eta)$.

In addition, we have coded up a trading strategy using Lai’s portfolio optimization algorithm as a signal. This strategy re-balances periodically a basket of assets according to the optimal weighting computed by the model. Specifically, onDepthUpdate(), collects the returns series using the component ReturnSeries with its update() method. Then, onTimerUpdate() is triggered periodically (say, every 3 months) to call the model (passing in the returns matrix) and to send orders to re-balance the assets accordingly. We backtest this trading strategy using the Simulator object together with the historical data from Yahoo! Finance (or other data sources).

An online demonstration can be found here.

### References

• Black, F. and Litterman, R. (1990). Asset Allocation: Combining Investor Views with Market Equilibrium. Goldman, Sachs & Co., New York.
• Fan, J, Fan, Y and Lv, J. (2008). High dimensional covariance matrix estimation using a factor model. J. Econometrics 147 186-197.
• Lai, T. L., Xing, H. and Chen Z. (2010). Mean-variance portfolio optimization when means and covariances are unknown. Annals of Applied Statistics 2011, Vol. 5, No. 2A, p. 798-823.
• Ledoit, P. and Wolf, M. (2004). Honey, I shrunk the sample covariance matrix. J. Portfolio Management 30 110–119.
• Michaud, R. O. (1989). Efficient Asset Management. Harvard Business School Press, Boston.

## FREE .NET/C# Numerical/Math library

Posted by on Dec 25, 2012 in SuanShu | 2 comments

On this Christmas Day, we are happy to announce that SuanShu.net is FREE for all! SuanShu.net has all the features as its Java sibling as well as has undergone the same many thousands of test cases daily.

There are a tutorial and examples that show you how to build a SuanShu application in Visual Studio. One major advantage of using SuanShu.net over the Java version is that it integrates seamlessly with Microsoft Excel. By incorporating SuanShu library in your spreadsheet, you literally have access to hundreds of numerical algorithms when manipulating and analyzing your data, significantly enhancing Excel’s productivity.

We hope that you enjoy using SuanShu.net in your work. If you have any interesting story, comments or feedback, we’d love to hear from you.

Posted by on Dec 21, 2012 in AlgoQuant, Programming | 2 comments

Bloomberg maintains tick-by-tick historical data for only 140 days. However, you may want to backtest your strategies with a longer history. In this case, you can archive these tickdata by yourself and do backtesting with the archived data. Since version 0.2, AlgoQuant supports downloading tick-by-tick data from Bloomberg and saving them as CSV files via the Bloomberg Java API v3 (assuming that you have access to a Bloomberg terminal).

After you download the AlgoQuant package, you will find that there is a folder lib/blpapi-dummy which contains the Bloomberg API jar file (blpapi3.jar). This file is a dummy for AlgoQuant to compile. To use the Bloomberg Java API, you need to replace the file with the real API jar file. If your machine has been deployed the Bloomberg API, you can find the real jar file in your hard drive, for example,

C:\blpAPI\APIv3\JavaAPI\v3.4.3.2\lib\blpapi3.jar

Note that the version number of the API may be automatically upgraded by Bloomberg.

The code for connecting, downloading and saving the tickdata is located in the package com.numericalmethod.algoquant.data.historicaldata.bloomberg.  You can find in the package a simple demo application “TickDataDownloadApp“, which accepts command-line arguments and downloads tickdata of a given period for a given security:

Usage: TickDataDownloadApp <security> <startDate (yyyy-MM-dd)> <endDate (yyyy-MM-dd)> <output dir>

Note that the start date should be within 140 days as it is the oldest history you can download from Bloomberg.

Here is how it works. The Bloomberg system provides a core service named “//blp/refdata” which allows downloading a wide range of market data. The code opens a session to connect to the localhost at port 8194 (change the settings in SimpleSessionFactory if you are using a different port). Then, it sends the IntradayTickRequest with the security symbol, start and end dates. Upon receiving the response, the BloombergTickDataFileWriter saves the data as zipped CSV files in the specified output folder (one file per trading day). For example,

TickDataDownloadApp "5 HK Equity" 2012-12-01 2012-12-21 data

will save the data as .csv.zip files in the folder “data/5 HK Equity“.

Since AlgoQuant is source-available, you are free to change the code to download different data, or save the data into your database instead of files.

Reference: Bloomberg API Version 3.x Developer’s Guide

## SuanShu 2.0

Posted by on Dec 13, 2012 in SuanShu | 0 comments

We are proud to announce the release of SuanShu 2.0! This release is the accumulation of customer feedbacks and our experience learnt in the last three years coding numerical computation algorithms. SuanShu 2.0 is a redesign of the software architecture, a rewrite of many modules, additions of new modules and functionalities driven by user demands and applications, numerous bug fixes as well as performance tuning. We believe that SuanShu 2.0 is the best numerical and statistical library ever available in Java, if not all, platform.

Here are highlights of the new features available since 2.0.

-          ordinary and partial differential equation solvers

-          Optimization: Quadratic Programming,  Sequential Quadratic Programming, (Mixed) Integer Linear Programming, Semi-Definite Programming

-          ARIMA fit

-          LASSO and LARS

-          Interpolation methods

-          Trigonometric functions and physical constants

-          Extreme Value Theory

Continuing our tradition, we will still provide trial license and academic license for eligible schools and research institutes. Moreover, we now provide another way to get a FREE SuanShu license – the contribution license. If you are able to contribute code to the SuanShu library, you can get a permanent license. For more information, see: http://numericalmethod.com/suanshu/

We hope that you will find the new release of SuanShu more helpful than ever in your work. If you have any comments to help us improve, please do let us know.

Happy birthday to TianTians and Merry Christmas to all!

## Trading and Investment as a Science

Posted by on Sep 3, 2012 in Algorithmic Trading, Investment, Seminars | 0 comments

Here is the synopsis of my presentation at HKSFA, September 2012. The presentation can be downloaded from here.

1.

Many people lose money playing the stock market. The strategies they use are nothing but superstitions. There is no scientific reason why, for example, buying on a breakout of the 250-day-moving average, would make money. Trading profits do not come from wishful thinking, ad-hoc decision, gambling, and hearsay, but diligent systematic study.
• Moving average as a superstitious trading strategy.

2.

Many professionals make money playing the stock market. One approach to investment decision or trading strategy is to treat it as a science. Before we make the first trade, we want to know how much money we expect to make. We want to know in what situations the strategy will make (or lose) money and how much.
• Moving average as a scientific trading strategy

3.

There are many mathematical tools and theories that we can use to quantify, analyse, and verify a trading strategy. We will show case some popular ones.
• Markov chain (a trend-following strategy)
• Cointegration (a mean-revision strategy)
• Stochastic differential equations (the best trading strategy, ever!)
• Extreme value theory (risk management, stop-loss)
• Monte Carlo simulation (what are the success factors in a trading strategy?)

## Using SuanShu on Amazon EC2

Posted by on May 21, 2012 in SuanShu | 0 comments

## Data Mining

Posted by on Feb 12, 2012 in Algorithmic Trading | 0 comments

The good quant trading models reveal the nature of the market; the bad ones are merely statistical artifacts.

One most popular way to create spurious trading model is data snooping or data mining. Suppose we want to create a model to trade AAPL daily. We download some data of, e.g., 100 days of AAPL, from Yahoo. If we work hard enough with the data, we will find a curve (model) that explains the data very well. For example, the following curve perfectly fits the data.

Suppose the prices are ${ x_1, x_2, \dots x_n }$

$\frac{(t-2)\dots(t-n)}{(1-2)\dots(1-n)}(x_1) + \frac{(t-1)\dots(t-n)}{(2-1)\dots(2-n)}(x_2) + \dots + \frac{(t-1)\dots(t-n+1)}{(n-1)\dots(n-n+1)}(x_n)$

Of course, most of us are judicious enough to avoid this obvious over-fitting formula. Unfortunately, some may fall into the trap of it in disguise. Let’s say we want to understand what factors contribute to the AAPL price movements or returns. (We now have 99 returns.) We come up with a list of 99 possible factors, such as PE, capitalization, dividends, etc. One very popular method to find significant factors is linear regression. So, we have

$r_t = \alpha + \beta_1f_{1t} + \dots + \beta_{99}f_{99t} + \epsilon_t$

Guess how well this fits? The goodness-of-fit (R-squared) turns out be 100% – a perfect fit! It can be proved that this regression is a complete nonsense. Even if we throw in random values for those 99 factors, we will also end up with a perfect fit regression. Consequently, the coefficients and t-stats mean nothing.
Could we do a “smaller” regression on a small subset of factors, e.g., one factor at a time, and hope to identify the most significant factor? This step-wise regression turns out to be spurious as well. For a pool of large enough factors, there is big probability of finding (the most) significant factors even when the factors values are randomly generated.

Suppose we happen to regress returns on only capitalization and finds that this factor is significant. Even so, we may in fact be doing some form of data snooping. This is because there are thousands other people testing the same or different factors using the same data set, i.e., AAPL prices from Yahoo. This community, taken as a whole, is doing exactly the same step-wise regression described in the last paragraph. In summary, empirical evidence alone is not sufficient to justify a trading model.

To avoid data snooping in designing a trading strategy, Numerical Method Inc. recommends our clients a four-step procedure.

1. Hypothesis: we start with an insight, a theory, or a common sense about how the market works.
2. Modeling: translate the insight in English into mathematics (in Greek).
3. Application: in-sample calibration and out-sample backtesting.
4. Analysis: understand and explain the winning vs. losing trades.

In steps 1 and 2, we explicitly write down the model assumptions, deduce the model properties, and compute the p&l distribution. We prove that under those assumptions, the strategy will always make money (on average). Whether these assumptions are true can be verified against data using techniques such as hypothesis testing. Given the model parameters, we know exactly how much money we expect to make. This is all done before we even look at a particular data set. In other words, we avoid data snooping by using the data set only until the calibration step and after we have created a trading model.

An example of creating a trend following strategy using this procedure can be found in lecture 1 of the course “Introduction to Algorithmic Trading Strategies”.