Bloomberg Tick-By-Tick Data Download

Bloomberg maintains tick-by-tick historical data for only 140 days. However, you may want to backtest your strategies with a longer history. In this case, you can archive these tickdata by yourself and do backtesting with the archived data. Since version 0.2, AlgoQuant supports downloading tick-by-tick data from Bloomberg and saving them as CSV files via the Bloomberg Java API v3 (assuming that you have access to a Bloomberg terminal). After you download the AlgoQuant package, you will find that there is a folder lib/blpapi-dummy which contains the Bloomberg API jar file (blpapi3.jar). This file is a dummy for AlgoQuant to compile. To use the Bloomberg Java API, you need to replace the file with the real API jar file. If your machine has been deployed the Bloomberg API, you can find the real jar file in your hard drive, for example, C:\blpAPI\APIv3\JavaAPI\v3.4.3.2\lib\blpapi3.jar Note that the version number of the API may be automatically upgraded by Bloomberg. The code for connecting, downloading and saving the tickdata is located in the package com.numericalmethod.algoquant.data.historicaldata.bloomberg.  You can find in the package a simple demo application “TickDataDownloadApp“, which accepts command-line arguments and downloads tickdata of a given period for a given security: Usage: TickDataDownloadApp <security> <startDate (yyyy-MM-dd)> <endDate (yyyy-MM-dd)> <output dir> Note that the start date should be within 140 days as it is the oldest history you can download from Bloomberg. Here is how it works. The Bloomberg system provides a core service named “//blp/refdata” which allows downloading a wide range of market data. The code opens a session to connect to the localhost at port 8194 (change the settings in SimpleSessionFactory if you are using a...

The Role of Technology in Quantitative Trading Research

I posted my presentation titled “The Role of Technology in Quantitative Trading Research” presented in HKU-HKUST-Stanford Conference in Quantitative Finance. Dec 9, 2011. Workshop On Recent Developments Of Financial Mathematics (REDFIN2011). Dec 13, 2011. You can find the powerpoint here. Abstract: There needs a technology to streamline the quantitative trading research process. Typically, quants/traders, from idea generation to strategy deployment, may take weeks if not months. This means not only loss of trading opportunity, but also a lengthy, tedious, erroneous process marred with ad-hoc decisions and primitive tools. From the organization’s perspective, comparing the paper performances of different traders is like comparing apples to oranges. The success of the firm relies on hiring the right geniuses. Our solution is a technological process that standardizes and automates most of the mechanical steps in quantitative trading research. Creating a new trading strategy should be as easy and fun as playing Legos by assembling together simpler ideas. Consequently, traders can focus their attention on what they are supposed to be best at – imagining new trading ideas/strategies. Excerpts: In reality, the research process for a quantitative trading strategy, from conceptual design to actual execution, is very time consuming, e.g., months. The backtesting step, in the broadest sense, takes the longest time. There are too many details that we can include in the backtesting code. To just name a few, data cleaning and preparation, mathematics algorithms, mock market simulation, execution and slippage assumptions, parameter calibration, sensitivity analysis, and worst of all, debugging. In practice, most people will ignore many details and make unfortunate “approximation”. This is one major reason why real and paper...

Java vs c++ performance

It is very unfortunate that some people are still not aware of the fact that Java performance is comparable to that of C++. This blog piece collects the evidence to support this claim. The wrong perception about Java slowness is by-and-large because Java 1 in 1995 was indeed slower than C++. Java has improved a lot since then, e.g., hotspot. It is now version 6 and soon will be version 7. Java is now a competitive technology comparing to C/C++. In fact, in order to realistically optimize for C/C++, you need to find the “right” programmer to code it. This programmer needs to be aware of all the performance issues of C/C++, profiling, code optimization such as loop unfolding, and may even need to write code snippets in assembly. An average Joe coding in C/C++ is probably not any faster than coding in Java. (I am in general against code optimization techniques because they make the code unreadable to humans, hence unmaintainable, such as a lot of the FORTRAN/C/C++ code found in Netlib and Statlib.) More importantly, most modern software runs on multiple cores. Code optimization techniques are dwarfed by parallel computing technologies. It is significantly easier and more efficient (and more enjoyable) to write concurrent programming code in Java than in C++. Therefore, to code high performance software, I personally prefer to code for multi-core, multi-CPU, and cloud in java rather than doing code optimization in C/C++. (I AM NOT SURE WHY FORTRAN SURVIVES IN 2011. HOW ARE YOU SUPPOSED TO READ THOUSDANDS LINES OF CODE ALL IN UPPER/LOWER CASES WITH A BUNCH OF C’S AND GOTO’S EVERYWHERE?)...

Open Source Trading Software Or Not?

I have built a few trading systems from scratch during my years with investment banks. So, I have learnt from the many mistakes made. I am recently reviewing some open source system, and would like to share some thoughts. In general, I am against building software in house. Funds and banks are not software firms. They should focus on making money instead of IT development. Producing software is best left to Microsoft, Google, Apple and IT vendors. These good folks spend their life time doing nothing other than building software. There is no chance that a (small) financial organization can build better systems than they do. Now, there is an exception to this rule of thumb. When it comes to building trading systems, I suggest that we should do this in house. Alternatively, you can hire a vendor and retain them on contract basis to co-develop the trading system, if you do not have proper talents in house. (It is notoriously difficult to hire very talented programmers in finance because the top quality guys go to GOOG, MSFT, AAPL or startups.) The worst is to download an open source trading system and simply hope that it will work. Let’s examine why. Firstly, for a piece of very complicated software such as trading system, it is unlikely that the downloaded code works out of the box. There are often too many configurations to do. Even with proper documentations (which is a big assumption), there are still too many plumbing to set up before you can make the first trade, e.g., connectivity to exchanges, data sources, news and announcements, writing and...

The Demand of Good Financial Engineers

It is very difficult to find people who can produce effective code for mathematical models. In general, mathematicians do not write good code and computer programmers do not understand the (advanced) mathematics. Yet, computer language is the only language we use to turn an idea into a product that people can use. I foresee that the truly valuable talents are those who have creative ideas and are able to implement them. They are the most-sought after in many engineering disciplines, e.g., finance. Speaking from personal experience, I have been working exclusively in the financial industry, Algorithmic Trading in particular. This field is where mathematics and computer science meet. As an algo-trader, my job is to develop mathematical models and write computer software for automatic execution. In addition, I lead a team of mathematicians to design trading models and a team of programmers to build these systems. From these years of job experience, I find it exceedingly difficult to hire the perfect candidates to work in this algorithmic trading industry. I have hired some very good statisticians and mathematicians from the top schools. They produce good research and design very sophisticated mathematical models. Unfortunately, they are not able to program to the professional standard their models that are ready to be used by other people. Often the programmers need to translate into C++/C#/Java their prototypes in Matlab/R. On the other hands, it is unrealistic to expect the programmers to develop these mathematical models simply because they do not have the training. In summary, my dilemma is: mathematicians cannot differentiate between inheritance and interface; programmers do not know about hidden Markov...

What is missing in our computer science curriculum?

From my years of experience of recruiting computer science students, it seems that universities fail to train students who can build good software. When I hire freshly graduated students, I often need to rewrite their code (sometimes from scratch) before putting the code in production. Most graduates know what inheritance and interface are, but few of them can properly choose and defend which to use. Most graduates have taken classes in “Operating Systems” and thus can talk a lot about threads and scheduling, but few actually have read books like the one from Doug Lea. They cannot be entrusted to write multi-threaded code. Most graduates memorize the visited nodes when writing a function to detect cycle in a linked list, but few of them are aware of Floyd’s cycle-finding algorithm. The funnies thing is that a lot of them think that they are writing object-oriented code because they have replaced ‘struct’ in C with ‘class’ in C++. They are yet to read a book on design pattern. In other words, these graduates cannot use the programming languages effectively (ineffective). They cannot communicate their ideas to colleagues in their code (incomprehensible). They do not evaluate the space and time complexities of their algorithms (inefficient). They produce spaghetti code that is so difficult to modify (unmaintainable) so we have this motto in the industry: if it ain’t broke, don’t fix it. It is next to impossible to add new requirements without rewriting it from scratch (inextensible). Often when I do code review, the code looks like scribble to me. I have no way to tell whether it is right or wrong...