Forex 3.
Forex Trading Diary #3 - Open Sourcing the Forex Trading System.
Forex Trading Diary #3 - Open Sourcing the Forex Trading System.
In today's entry of the Forex Trading Diary I want to discuss the longer term plan for the forex trading system. In addition I want to outline how I've used Python's Decimal data-type to make calculations more accurate.
To date, we've been experimenting with the OANDA Rest API in order to see how it compared to the API provided by Interactive Brokers. We've also seen how to add in a basic portfolio replication element as the first step towards a proper event-driven backtesting system. I've also had some helpful comments on both previous articles (#1 and #2), which suggests that many of you are keen on changing and extending the code yourselves.
Open Sourcing the Forex Trading System.
For the reasons outlined above I have decided to open-source the forex trading system . What does this mean? It means that all current and future code will be available for free, under a liberal MIT open source license, on the Github version control website at the following URL: https://github.com/mhallsmoore/qsforex.
For those of you who have used git and Github before, you'll be able to git clone the repo and start modifying it for your own purposes.
The QuantStart Automated Forex Trading System is now open-source under a liberal MIT license. You can find the latest code on Github under the qsforex repository at https://github.com/mhallsmoore/qsforex.
For those of you who are new to source version control you will probably want to read up on how git (and version control in general) works with the fantastic free ebook Pro Git. It is worth spending some time learning about source control as it will save you a huge amount of future headache if you spend a lot of time programming and updating projects!
The "quick start" for a Ubuntu system is to install git:
sudo apt-get install git-core.
You will then need to make a directory for the qsforex project to live in and "clone" the project from the Github site as follows:
mkdir -p ~/projects/ cd ~/projects/ git clone https://github.com/mhallsmoore/qsforex.git.
At this point you will need to create a virtual environment in which to run the code:
mkdir -p ~/venv/qsforex cd ~/venv/qsforex virtualenv . source ~/venv/qsforex/bin/activate.
You will then need to install the requirements (this will take some time!):
pip install -r ~/projects/qsforex/requirements.txt.
Finally you will need to create a symbolic link in your Python virtual environment to allow you to type import qsforex in your code (and run it!):
sln -s ~/projects/qsforex/ ~/venv/qsforex/lib/python2.7/site-packages/qsforex.
As I mentioned in the previous entries you will need to create the necessary environment variables for your OANDA authentication credentials. Please see diary entry #2 for instructions on how to do this.
Please pay attention to the README associated with the repo, as it contains installation instructions, a disclaimer and a warranty about using the code.
Since the software is in "alpha" mode, these instructions will become more straightforward as time progresses. In particular I will try to wrap the project into a Python package so that it can be easily installed via pip.
If you have any questions about the installation procedure, then please don't hesitate to contact our team on support@quantstart.com.
Longer-Term Plan.
The "philosophy" of the forex trading system, as with the rest of the QuantStart site, is to try and mimic real-life trading as much as possible in our backtesting. This means including the details that are often excluded from more "research oriented" backtesting situations. Latency, server outages, automation, monitoring, realistic transaction costs will all be included within the models to give us a good idea of how well a strategy is likely to perform.
Since we will have access to tick data (bid/ask timestamps) we will be able to incorporate the spread into the transaction costs. We can also model slippage. It is less straighforward to model market impact, although this is less of a concern at smaller trading amounts.
In addition to transaction costs we want to model robust portfolio management using risk overlays and position sizing.
So what is currently included in the Forex Trading System to date?
Event-Driven Architecture - The forex trading system has been designed as an event-driven system from the ground up, as this is how an intraday trading system will be implemented in a live environment. Price Streaming - We have a basic price streaming object. This currently handles subscription to only a single pair, but we can easily modify this to subscribe to multiple currency pairs. Signal Generation - We can incorporate trading strategies (based directly off past and current tick prices) using the Strategy object, which creates SignalEvent objects. Order Execution - We have a naive order execution system that blindly sends orders from the Portfolio to OANDA. By "blindly" I mean that there is no risk management or position sizing being carried out, nor any algorithmic execution that might lead to reduced transaction costs. GBP Base Currency - To keep things simple, I've only written the system for GBP base currency. This is perhaps the most important aspect to modify given how many of you will have practice accounts denominated in USD, EUR, CAD, JPY, AUD and NZD! GBP/USD Trading - I picked "the cable" as the currency pair to test the initial Position and Portfolio objects with. Handling multiple currency pairs is an important next step. This will involve modification to the position and portfolio calculations. Decimal Handling - Any production trading system must correctly handle currency calculations. In particular, currency values should not be stored as floating point data-types, since the rounding errors will accumulate. Please see this fantastic article on floating point representations for more details. Long/Short Trading - Between diary entries #2 and #3 I added the ability to short a currency pair (as opposed to only being able to go long). Crucially, this is also unit tested. Local Portfolio Handling - In my opinion carrying out a backtest that inflates strategy performance due to unrealistic assumptions is annoying at best and extremely unprofitable at worst! Introducing a local portfolio object that replicates the OANDA calculations means that we can check our internal calculations while carrying out practice trading , which gives us greater confidence when we later use this same portfolio object for backtesting on historical data. Unit Tests for Position/Portfolio - While I've not mentioned it directly in diary entries #1 and #2, I've actually been writing some unit tests for the Portfolio and Position objects. Since these are so crucial to the calculations of the strategy, one must be extremely confident that they perform as expected. An additional benefit of such tests is that they allow the underlying calculation to be modified, such that if all tests still pass, we can be confident that the overall system will continue to behave as expected.
At this stage the Forex Trading System is lacking the following functionality:
Slippage Handling - The system is currently generating a lot of slippage due to the high-frequency nature of the tick data provided from OANDA. This means that the portfolio balance calculated locally is not reflecting the balance calculated by OANDA. Until correct event-handling and slippage adjustment is carried out, this will mean that a backtest will not correctly reflect reality. Multiple Base Currencies - We are currently restricted to GBP. At the very least we need to include the major currency denominations - USD, EUR, CAD, AUD, JPY and NZD. Multiple Currency Pairs - Similarly we need to support the major currency pairs beyond "Cable" (GBP/USD). There are two aspects to this. The first is to correctly handle the calculations when neither the base or quote of a currency pair is equal to the account denomination currency. The second aspect is to support multiple positions so that we can trade a portfolio of currency pairs. Risk Management - Many "research" backtests completely ignore risk management. Unfortunately this is generally necessary for brevity in describing the rules of a strategy. In reality we -must- use a risk overlay when trading, otherwise it is extremely likely that we will suffer a substantial loss at some stage. This is not to say that risk management can prevent this entirely, but it certainly makes it less likely! Portfolio Optimisation - In an institutional setting we will have an investment mandate, which will dictate a robust portfolio management system with various allocation rules. In a retail/personal setting we may wish to use a position sizing approach such as the Kelly Criterion to maximise our long-term compounded growth rate. Robust Strategies - I have only demonstrated some simple random signal generating "toy" strategies to date. Now that we are beginning to create a reliable intraday forex trading system, we should start carrying out some more interesting strategies. Future diary entries will concentrate on strategies drawn from a mixture of "technical" indicators/filters as well as time series models and machine learning techniques. Remote Deployment - Since we are potentially interested in trading 24 hours (at least during the week!) we require a more sophisticated setup than running the backtester on a local desktop/laptop machine at home. It is vital that we create a robust remote server deployment of our system with appropriate redundancy and monitoring. Historical Backtesting - We have built the Portfolio object to allow us to perform realistic backtesting. At this stage we are missing a historical tick data storage system. In subsequent articles we will look at obtaining historical tick data and storing it in an appropriate database, such as HDF5. Trade Database - Eventually we will wish to store our live trades in our own database. This will allow us to carry out our own analytics on live trading data. A good recommendation for a relational database would be PostgreSQL or MySQL. Monitoring and High Availability - Since we are considering a high-frequency intraday system, we must put comprehensive monitoring and high availability redundancy in place. This means reporting on CPU usage, disk usage, network I/O, latency and checking that any periodic scripts are set to keep running. In addition we need a backup and restore strategy. Ask yourself what backup plans you would have in place if you had large open positions, in a volatile market, and your server suddenly died. Believe me, it happens! Multiple Broker/FIX Integration - At the moment we are strongly coupled to the OANDA broker. As I said this is simply because I came across their API and found it to be a modern offering. There are plenty of other brokers out there, many of which support the FIX protocol. Adding a FIX capability would increase the number of brokers that could be used with the system. GUI Control and Reporting - Right now the system is completely console/command line based. At the very least we will need some basic charting to display backtest results. A more sophisticated system will incorporate summary statistics of trades, strategy-level performance metrics as well as overall portfolio performance. This GUI could be implemented using a cross-platform windowing system such as Qt or Tkinter. It could also be presented using a web-based front-end, utilising a web-framework such as Django.
As can be seen there is a lot of functionality left on the roadmap! That being said, each new diary entry (and potential contributions from the community!) will move the project forward.
Decimal Data-Types.
Now that we have discussed the longer term plan I want to present some of the changes I have made to the code since diary entry #2. In particular, I want to describe how I modified the code to handle the Decimal data-type instead of using floating point storage. This is an extremely important change as floating point representations are a substantial source of long-term error in portfolio and order management systems.
Python natively supports decimal representations to an arbitrary precision. The functionality is contained within the decimal library.
In particular we need to modify -every- value that appears in a Position calculation to a Decimal data-type. This includes the units, exposure, pips, profit and percentage profit. This ensures we are in full control of how rounding issues are handled when dealing with currency representations that have two decimal places of precision. In particular we need to choose the method of rounding. Python supports a few different types, but we are going to go with ROUND HALF DOWN , which rounds to the nearest integer with ties going towards zero.
Here is an example of how the code is modified to handle Decimal data-types from their previous floating point representations. The following is a list of position.py :
from decimal import Decimal, getcontext, ROUND HALF DOWN class Position(object): def init ( self, side, market, units, exposure, avg price, cur price ): self.side = side self.market = market self.units = units self.exposure = Decimal(str(exposure)) self.avg price = Decimal(str(avg price)) self.cur price = Decimal(str(cur price)) self.profit base = self.calculate profit base(self.exposure) self.profit perc = self.calculate profit perc(self.exposure) def calculate pips(self): getcontext.prec = 6 mult = Decimal("1") if self.side == "SHORT": mult = Decimal("-1") return (mult * (self.cur price - self.avg price)).quantize( Decimal("0.00001"), ROUND HALF DOWN ) def calculate profit base(self, exposure): pips = self.calculate pips() return (pips * exposure / self.cur price).quantize( Decimal("0.00001"), ROUND HALF DOWN ) def calculate profit perc(self, exposure): return (self.profit base / exposure * Decimal("100.00")).quantize( Decimal("0.00001"), ROUND HALF DOWN ) def update position price(self, cur price, exposure): self.cur price = cur price self.profit base = self.calculate profit base(exposure) self.profit perc = self.calculate profit perc(exposure)
Note that we must provide Decimal with a string argument, rather than a floating point argument. This is because a string is precisely specifying the precision of the value, whereas a floating point type will not.
Note also that when we begin storing our trades in a relational database (as described above in the roadmap) we will need to make sure we once again use the correct data-type. PostgreSQL and MySQL support a decimal representation. It is vital that we utilise these data-types when we create our database schema, otherwise we will run into rounding errors that are extremely difficult to diagnose!
For those who are interested in a deeper discussion of these issues, in mathematics and computer science, the subject of Numerical Analysis covers floating point storage issues, among many other interesting topics.
In subsequent diary entries we are going to discuss how I have applied unit testing to the code and how we can extend the software to more currency pairs by modifying the position calculations.
Full Python Code.
Since the full source code for the project is now open source, under a MIT license, it can always be found out at https://github.com/mhallsmoore/qsforex, with the accompanying documentation.
If you would like to read the other entries in the series, please follow the links below:
QSAlpha.
Join the QSAlpha research platform that helps fill your strategy research pipeline, diversifies your portfolio and improves your risk-adjusted returns for increased profitability.
The Quantcademy.
Join the Quantcademy membership portal that caters to the rapidly-growing retail quant trader community and learn how to increase your strategy profitability.
Successful Algorithmic Trading.
How to find new trading strategy ideas and objectively assess them for your portfolio using a Python-based backtesting engine.
Advanced Algorithmic Trading.
How to implement advanced trading strategies using time series analysis, machine learning and Bayesian statistics with R and Python.