Strategy: XGBoost Classification + Technical Indicators


On many Kaggle public leaderboards, you will see an algorithm called “XGBoost”, or “Xtreme Gradient Boosting”. This is an optimized algorithm under the Gradient Boosting framework. The key difference between the normal boosting models and XGBoost, as far as I can see, is that XGBoost allows for parallel processing while keeps the accuracy on its performance. This dramatically makes large-scale machine learning algorithms (since boosting is essentially an ensemble of several weak learning algorithms) possible on personal computers (even on my MBP 2015), and further empowers people like me to apply them onto more practical scenarios, e.g. stock price prediction as we are gonna talk about today. Highest tribute to Tianqi Chen and his great paper!


Statistic Value
Backtest Interval 20130101 - 20170101
Initial Capital 1,000,000
Annualized Return (Strategy) TBD
Annualized Return (Benchmark) TBD
Sharpe Ratio TBD
Maximum Drawdown TBD

(Performance screenshot not available at present. Will be uploaded later.)

Intuition: Long-term vs. Short-term Prediction

I got the inspiration from this paper. The author raised an interesting but also convincing point when doing stock price prediction: the long-term trend is always easier to predict than the short-term. Again, let’s take AAPL for example. The Apple Inc. witnessed a close at $139.14 per share today, a slight rise by 0.33 percent point. What is the expected price change in percentage for, say, tomorrow? Also let’s assume that you have access to all the historical data you want, regardless of the size of the dataset or other computational issues? What if an chief engineer suddenly decides to resign from the company and the news leaks instantly through social networks? What if Samsung or another mobile phone company launches its newest prototype right after your prediction? What if, just for fun, the new prototype of Samsung (or whatever) should explode during the presentation, on the spot? The randomness is so changable that our prediction could even be totally the opposite of the true story. However, in a sense of long-term prediction, things get easier, since the overall relationship between Apple and Samsung would least likely change in the near future, and the general performance of AAPL is in principle dependent on how Apple Inc. behave in the following months. So GIVEN THAT COOPER IS NOT INSANE, BUT A RATHER CAUTIOUS CEO, we can expect a higher accuracy when predicting the average price change of AAPL in the next three months (60 tradingdays or so).

Notice again that we are not talking about predicting the close of the day three months from now. We are talking about the AVERAGE price change of the following three months. Now, let’s turn to see what predictor we can use to perform the idea.

Tool: Technical Indicators & XGBoost Classification

We are using the classification algorithm in XGBoost. The regression algorithm is slower but more intuitive, but binary classification is just faster in this case, which allows for efficient backtesting later on. The prediction is based on a certain length of historical data including open, high, low, close and volume, along with other 77 techinical indicators that are not extracted until before the prediction. This is of course far from “all historical data” in the market, not to mention the off-market rumors, social and political concerns, etc. But this is enough for illustration here. Therefore, a good amendment for this strategy is surely to append these left-out indicators. For NLA and similar applications in stock price prediction, you can check my post about making Mr. Trump’s tweets into prediction indicators. I didn’t acutally write the project because I found a really mature application already written on Github, but I would definitely rewrite it myself during the summer vacation. If you find the post not changed even after that, please feel free to mail me and “harshly” urge me for the overdue task.


The codes mainly consists of three parts:

import sys
import numpy as np
import tushare as ts
import pandas as pd
import talib as tb
from xgboost import XGBClassifier
from datetime import datetime as dt
from dateutil.relativedelta import relativedelta
from sklearn.model_selection import StratifiedKFold, GridSearchCV
from sklearn.metrics import accuracy_score

# define pivot variables for easy use
def technical(df):
    open = df['open'].values
    close = df['close'].values
    high = df['high'].values
    low = df['low'].values
    volume = df['volume'].values
    # define the technical analysis matrix
    retn = np.array([
        tb.MA(close, timeperiod=5),                                         # 1
        tb.MA(close, timeperiod=10),                                        # 2
        tb.MA(close, timeperiod=20),                                        # 3
        tb.MA(close, timeperiod=60),                                        # 4
        tb.MA(close, timeperiod=90),                                        # 5
        tb.MA(close, timeperiod=120),                                       # 6

        tb.ADX(high, low, close, timeperiod=20),                            # 7
        tb.ADXR(high, low, close, timeperiod=20),                           # 8

        tb.MACD(close, fastperiod=12, slowperiod=26, signalperiod=9)[0],    # 9
        tb.RSI(close, timeperiod=14),                                       # 10

        tb.BBANDS(close, timeperiod=5, nbdevup=2, nbdevdn=2, matype=0)[0],  # 11
        tb.BBANDS(close, timeperiod=5, nbdevup=2, nbdevdn=2, matype=0)[1],  # 12
        tb.BBANDS(close, timeperiod=5, nbdevup=2, nbdevdn=2, matype=0)[2],  # 13

        tb.AD(high, low, close, volume),                                    # 14
        tb.ATR(high, low, close, timeperiod=14),                            # 15

        tb.HT_DCPERIOD(close),                                              # 16

        tb.CDL2CROWS(open, high, low, close),                               # 17
        tb.CDL3BLACKCROWS(open, high, low, close),                          # 18
        tb.CDL3INSIDE(open, high, low, close),                              # 19
        tb.CDL3LINESTRIKE(open, high, low, close),                          # 20
        tb.CDL3OUTSIDE(open, high, low, close),                             # 21
        tb.CDL3STARSINSOUTH(open, high, low, close),                        # 22
        tb.CDL3WHITESOLDIERS(open, high, low, close),                       # 23
        tb.CDLABANDONEDBABY(open, high, low, close, penetration=0),         # 24
        tb.CDLADVANCEBLOCK(open, high, low, close),                         # 25
        tb.CDLBELTHOLD(open, high, low, close),                             # 26
        tb.CDLBREAKAWAY(open, high, low, close),                            # 27
        tb.CDLCLOSINGMARUBOZU(open, high, low, close),                      # 28
        tb.CDLCONCEALBABYSWALL(open, high, low, close),                     # 29
        tb.CDLCOUNTERATTACK(open, high, low, close),                        # 30
        tb.CDLDARKCLOUDCOVER(open, high, low, close, penetration=0),        # 31
        tb.CDLDOJI(open, high, low, close),                                 # 32
        tb.CDLDOJISTAR(open, high, low, close),                             # 33
        tb.CDLDRAGONFLYDOJI(open, high, low, close),                        # 34
        tb.CDLENGULFING(open, high, low, close),                            # 35
        tb.CDLEVENINGDOJISTAR(open, high, low, close, penetration=0),       # 36
        tb.CDLEVENINGSTAR(open, high, low, close, penetration=0),           # 37
        tb.CDLGAPSIDESIDEWHITE(open, high, low, close),                     # 38
        tb.CDLGRAVESTONEDOJI(open, high, low, close),                       # 39
        tb.CDLHAMMER(open, high, low, close),                               # 40
        tb.CDLHANGINGMAN(open, high, low, close),                           # 41
        tb.CDLHARAMI(open, high, low, close),                               # 42
        tb.CDLHARAMICROSS(open, high, low, close),                          # 43
        tb.CDLHIGHWAVE(open, high, low, close),                             # 44
        tb.CDLHIKKAKE(open, high, low, close),                              # 45
        tb.CDLHIKKAKEMOD(open, high, low, close),                           # 46
        tb.CDLHOMINGPIGEON(open, high, low, close),                         # 47
        tb.CDLIDENTICAL3CROWS(open, high, low, close),                      # 48
        tb.CDLINNECK(open, high, low, close),                               # 49
        tb.CDLINVERTEDHAMMER(open, high, low, close),                       # 50
        tb.CDLKICKING(open, high, low, close),                              # 51
        tb.CDLKICKINGBYLENGTH(open, high, low, close),                      # 52
        tb.CDLLADDERBOTTOM(open, high, low, close),                         # 53
        tb.CDLLONGLEGGEDDOJI(open, high, low, close),                       # 54
        tb.CDLLONGLINE(open, high, low, close),                             # 55
        tb.CDLMARUBOZU(open, high, low, close),                             # 56
        tb.CDLMATCHINGLOW(open, high, low, close),                          # 57
        tb.CDLMATHOLD(open, high, low, close, penetration=0),               # 58
        tb.CDLMORNINGDOJISTAR(open, high, low, close, penetration=0),       # 59
        tb.CDLMORNINGSTAR(open, high, low, close, penetration=0),           # 60
        tb.CDLONNECK(open, high, low, close),                               # 61
        tb.CDLPIERCING(open, high, low, close),                             # 62
        tb.CDLRICKSHAWMAN(open, high, low, close),                          # 63
        tb.CDLRISEFALL3METHODS(open, high, low, close),                     # 64
        tb.CDLSEPARATINGLINES(open, high, low, close),                      # 65
        tb.CDLSHOOTINGSTAR(open, high, low, close),                         # 66
        tb.CDLSHORTLINE(open, high, low, close),                            # 67
        tb.CDLSPINNINGTOP(open, high, low, close),                          # 68
        tb.CDLSTALLEDPATTERN(open, high, low, close),                       # 69
        tb.CDLSTICKSANDWICH(open, high, low, close),                        # 70
        tb.CDLTAKURI(open, high, low, close),                               # 71
        tb.CDLTASUKIGAP(open, high, low, close),                            # 72
        tb.CDLTHRUSTING(open, high, low, close),                            # 73
        tb.CDLTRISTAR(open, high, low, close),                              # 74
        tb.CDLUNIQUE3RIVER(open, high, low, close),                         # 75
        tb.CDLUPSIDEGAP2CROWS(open, high, low, close),                      # 76
        tb.CDLXSIDEGAP3METHODS(open, high, low, close)                      # 77
    return retn

class stock:
    def __init__(self, id): = id
        self.train = None
        self.test = None

    def load(self, window_length=120, ratio=0, back_length=10, forward_length=60, test_size=0.25, verbose=False):

        # load dataset through tushare
        open = history_bars(, window_length, '1d', 'open')
        close = history_bars(, window_length, '1d', 'close')
        high = history_bars(, window_length, '1d', 'high')
        low = history_bars(, window_length, '1d', 'low')
        volume = history_bars(, window_length, '1d', 'volume')
        df = pd.DataFrame({'open': open, 'close': close, 'high': high, 'low': low, 'volume': volume}).astype(float)
        tot_len = len(df)

        # use the technical function to load all the technical data
        tech = technical(df)

        # define the label function, ratio required
        def label(df, ratio, back_length, forward_length):
            close = df['close']
            r = 0.04  # discount rate
            mean_diff = sum([-close.diff(-i) * np.exp(-r * i) for i in range(1, forward_length + 1)]) / forward_length
            mean_p_change = (mean_diff / close)[back_length: -forward_length]
            return (mean_p_change > ratio).astype(int)

        # split data into X and y
        X = df[['open', 'close', 'high', 'low', 'volume']]
        X_shift = [X]
        for i in range(1, back_length):
            X_shift.append(df[['open', 'close', 'high', 'low', 'volume']].shift(i))
        if forward_length == 0:
            return np.concatenate([tech, np.log(pd.concat(X_shift, axis=1).values)], axis=1)[back_length:]
        X = np.concatenate([tech, np.log(pd.concat(X_shift, axis=1).values)], axis=1)[back_length: -forward_length]
        y = label(df, ratio, back_length, forward_length)

        # split data into train and test sets
        test_len = int(tot_len * test_size)
        train_len = tot_len - test_len
        X_train, X_test = X[:train_len], X[-test_len:]
        y_train, y_test = y[:train_len], y[-test_len:]

        # update the train and test dataset
        self.train = [X_train, y_train]
        self.test = [X_test, y_test]

class classifier:
    def __init__(self):
        self.model = XGBClassifier(n_estimators=300,

    def train(self, data):
        X_train, y_train = data.train[0], data.train[1]
        X_test, y_test = data.test[0], data.test[1], y_train,
                       eval_set=[(X_train, y_train), (X_test, y_test)],

        y_pred = self.model.predict(X_train)
        predictions = [round(value) for value in y_pred]
        # evaluate predictions
        train_acc = accuracy_score(y_train, predictions)

        # make predictions for test data
        y_pred = self.model.predict(X_test)
        predictions = [round(value) for value in y_pred]
        # evaluate predictions
        test_acc = accuracy_score(y_test, predictions)

        print('{} train acc: {:.2f}%, test acc: {:.2f}%'.format(, train_acc * 100, test_acc * 100))
        return test_acc

    def predict(self, data):
        return self.model.predict(data)

def close_all(context):
    for s in context.portfolio.positions:
        order_target_percent(s, 0)

def modelize(context, bar_dict):
    if context.flag % (context.FL // 20) != 0:
        context.flag += 1
        return None

    print('-' * 49)


    context.pool = [
        '600804.XSHG', ]
    context.stock = [stock(s) for s in context.pool]

    print('loading training dataset')
    stt = ( + relativedelta(months=-context.WL // 20)).strftime('%Y-%m-%d')
    end ='%Y-%m-%d')
    for i in range(context.N):
        print(context.stock[i].id + ' loaded')
    print('-' * 49)

    for i in range(context.N):
        context.score[i] = context.clf[i].train(context.stock[i])
    print('-' * 49)

    stt = ( + relativedelta(months=-1)).strftime('%Y-%m-%d')
    sig = lambda s: '+' if s else '-'
    for i in range(context.N):
        context.signal[i] = context.clf[i].predict(context.stock[i].load(window_length=context.BL * 2,
                                                                         forward_length=0))[-1] * \
            int(context.score[i] > 0.9)
        print('{}: {}'.format(context.stock[i].id, sig(context.signal[i])))
    print('-' * 49)

    tot = sum(context.signal)
    if tot:
        for i in range(context.N):
            if context.signal[i]:
                order_target_percent(context.pool[i], 1 / tot)

def init(context):
    context.N = 10          # number of stocks in pool
    context.BL = 20         # backward length
    context.FL = 60         # forward length
    context.WL = 240        # total window length
    context.ratio = 0.000   # critical ratio as threshold
    context.flag = 0
    context.pool = []
    context.stock = []
    context.signal = [0] * context.N
    context.score = [0] * context.N
    context.clf = [classifier()] * context.N
    scheduler.run_monthly(modelize, tradingday=1)

__config__ = {
    "base": {
        "strategy_type": "stock",
        "start_date": "2011-01-01",
        "end_date": "2017-01-01",
        "frequency": "1d",
        "matching_type": "next_bar",
        "future_starting_cash": 1000000,
        "commission_multiplier": 0.01,
        "benchmark": "000300.XSHG",
    "extra": {
        "log_level": "error",
    "mod": {
        "progress": {
            "enabled": True,
            "priority": 400,