Skip to main navigation Skip to search Skip to main content

Improving trade execution with deep reinforcement learning

  • Isaac Tonkin

Student thesis: Doctoral Thesis

Abstract

This thesis investigates the application of deep reinforcement learning to trade execution. Trade execution is an ongoing optimisation problem in finance, that focusses on attaining the best prices through forecasting future price movements and liquidity. This is particularly relevant to institutional investors transacting large positions. Deep reinforcement learning is particularly well-suited to this problem given its ability to capture highly non-linear relationships within a sequential decision-making context. However, research that employs deep reinforcement learning for this task is at a nascent stage, with most focussed on established mathematical methods. Numerous reinforcement learning formulations have been explored, but progress has been hindered by the lack of comparisons between published results.

A new comparative framework, referred to as the Deep Reinforcement Learning Trade Execution (DRLTE) Framework, is established to facilitate the development and evaluation of deep reinforcement learning approaches to trade execution. To ascertain the key methodological considerations required for this framework, a systematic literature review is conducted. Based on the findings of this review, recommended guidelines are established for each of these key considerations. This framework enables approaches to be evaluated on a consistent basis. This is a critical advancement necessary to address the lack of comparison between existing deep reinforcement learning approaches.

The DRLTE Framework is then implemented to compare several existing approaches using a comprehensive suite of benchmarks, metrics, diagnostics and analytics. The core of the DRLTE Framework is the simulated exchange that is used to simulate the placement of market and limit orders using tick and limit order book data. Experiments are conducted using data from ten highly liquid stocks listed on the Australian Stock Exchange over a five-year period. Extensive pre-processing and robustness checks are conducted with a focus on finance-specific considerations, an aspect that has not been widely explored in prior research in this area.

It is found that the deep reinforcement learning approaches consistently outperformed the benchmark strategies on a median basis and did so with a lower standard deviation. Two new approaches were also developed, with the most effective being the one that had the most degrees of freedom in the action space, allowing for both price and volume to be adjusted simultaneously. This finding highlights the ability of deep reinforcement learning techniques to learn effective execution strategies, even when relatively unconstrained. The results of this analysis also provide insight into the future development of the reinforcement learning formulation.

As investors continually seek to improve the profitability of their trading strategies, further enhancements are made to the existing formulation by incorporating historical and forecasted context into the state space of the agents. Context provides additional information on market conditions. Historical context is incorporated using stock-level momentum and market-level momentum, while forecasted context is incorporated using a forecast of return and a forecast of volatility. The substantial improvement provided by the inclusion of these items of context demonstrates that it is a promising new avenue for future performance enhancements. In addition, this work is the first in this field to incorporate both historical and forecasted context and to demonstrate that synergistic gains can be attained.

The thesis concludes with a summary of the results and suggests avenues for future research. Proposed future work includes the continued development of new reinforcement learning formulations for trade execution, the inclusion of finance-specific considerations such as investor preferences and risk measures, and the use of the new framework to compare and refine the highest performing approaches.
Date of Award18 Jun 2026
Original languageEnglish
SupervisorGeoffrey Harris (Supervisor), Adrian Gepp (Supervisor) & Bruce Vanstone (Supervisor)

Cite this

'