Imitation Learning
Should an AI copy your best trader?
Every trading desk has an asset that never shows up on the balance sheet: years of logged decisions from its best people. When to lean in, when to widen a quote, when to sit on their hands.
Writing down why they did it is almost impossible. Their real objective mixes P&L, risk appetite, inventory, market impact and compliance in ways no reward function captures.
That's exactly the problem Imitation Learning (IL) was built for: learn a policy from expert demonstrations instead of a hand-crafted objective [1][2].
The obvious version is behavior cloning: treat every (market state → trader action) pair as a label and train a model. It works… until it doesn't.
The catch is distribution shift. A cloned policy makes a small error, lands in a state the expert never visited, makes a bigger error, and drifts away (see the chart). In markets this is amplified:
Regime changes are distribution shift at its worst
Experts are imperfect, so you clone their biases and bad days too
Logs miss the phone call or headline that actually drove the decision
Your own orders move the market, so offline data can't show the counterfactual
So is IL useless for trading? No. The field has spent 20+ years on exactly these failure modes:
Imitate an oracle, not just a human. DAgger-style methods query the expert in the states the learner actually reaches [3]. In finance, that expert can be a hindsight-optimal policy. Kim et al. use behavior cloning from a "prophetic expert" in statistical arbitrage [6].
Recover the why, not just the what. Inverse RL infers the reward a trader seems to optimize. Yang et al. used it to tell high-frequency strategies apart from other trading styles with >90% accuracy on a simulated E-mini S&P 500 market [5]. That's valuable for risk and surveillance, not just automation.
Start from demonstrations, then improve. AlphaGo began with imitation and surpassed it through self-play [4]. Trading RL systems increasingly use IL as a warm start [7].
Stay close to decisions where it shines: execution and market making, with frequent, short-horizon actions and plenty of data.
Our view at HQL: learn FROM your experts, don't just COPY them.
We're about to start a new research project applying imitation learning to trading decisions. We'll share what we learn along the way, including what doesn't work.
If you run a desk sitting on years of expert decisions, or you've tried to clone a trader and watched it drift, we'd love to compare notes.
#QuantitativeFinance #AlgorithmicTrading #MachineLearning #ImitationLearning #ReinforcementLearning #AI
References
[1] D. Garg, "Learning to Imitate," Stanford AI Lab Blog, 2022. https://ai.stanford.edu/blog/learning-to-imitate/
[2] A. Hussein et al., "Imitation Learning: A Survey of Learning Methods," ACM Computing Surveys 50(2), 2017. https://doi.org/10.1145/3054912
[3] S. Ross, G. Gordon, J. A. Bagnell, "A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning" (DAgger), AISTATS 2011. https://arxiv.org/abs/1011.0686
[4] R. Tedrake, Underactuated Robotics, Ch. "Imitation Learning," MIT. https://underactuated.mit.edu/imitation.html
[5] S. Y. Yang et al., "Behavior Based Learning in Identifying High Frequency Trading Strategies," IEEE CIFEr 2012. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1955965
[6] "The Evolution of Reinforcement Learning in Quantitative Finance: A Survey," 2024 (§6.4 Imitation Learning). https://arxiv.org/abs/2408.10932
[7] "Improving algorithmic trading consistency via human alignment and imitation learning," Expert Systems with Applications, 2024. https://www.sciencedirect.com/science/article/abs/pii/S0957417424012168
Further reading: NVIDIA, "What Is Imitation Learning?" https://www.nvidia.com/en-eu/glossary/imitation-learning/ · Z. Lőrincz, "A Brief Overview of Imitation Learning" https://smartlabai.medium.com/a-brief-overview-of-imitation-learning-8a8a75c44a9c