Associate Professor at the Department of Computer and Software Engineering, Polytechnique Montréal
Mila, Core Academic Member

Associate Professor (Status-Only) at the Department of Computer Science, University of Toronto

Research Goal: Understanding the computational and statistical mechanisms required to design efficient RL agents that interact with their environment and adaptively improve their long-term performance.

Refer to Research and Publications for more information on my research, and to Bio and CV for more information about my academic background.

Links

• Email: {my last name} at cs.toronto.edu
• Google Scholar
• CV (November 2024) and Bio
• Research Statement (2024): Rethinking Reinforcement Learning
• Teaching Statement (2024): Nurturing Expertise in ML and RL through Growth Mindset and Deliberate Practice
• Prospective students (tldr: I will recruit grad students at Polytechnique Montréal and Mila for September 2025.)
• Regularization in Reinforcement Learning (PhD Dissertation)
• SoloGen@Twitter and SoloGenBlog@Twitter (Persian)
• Blog: Anti Memoirs (Persian) [dysfunctional now!]

• (2024 Fall) I joined Polytechnique Montréal as an associate professor and Mila as a core academic member.
• (2024 Winter) I got Ontario Early Research Awards (ERA) on Accelerated Reinforcement Learning Algorithms.
• (2024 Winter) I taught the undergraduate course on Neural Networks and Deep Learning at DCS, U of T. This was my last course at the University of Toronto.
• (2022-2024) I did not keep the news updated from 2022 Fall until 2024. Many things have happened! Check out our papers instead!
• (2022 Fall) Tyler Kastner joined my team and Murat Erdogdu’s as a PhD student. Welcome!
• (2022 Fall) I taught the graduate level Introduction to Machine Learning course at DCS. This is a slightly updated version of the Fall 2021.
• (2022 Summer) Farnam Mansouri did his MSc on risk-aware RL, and joined University of Waterloo afterwards. Stay tuned for some of his interesting results!
• (2021 Fall) I taught the graduate level Introduction to Machine Learning course at DCS. I recorded the lectures, which you can find at this YouTube playlist.
• (2021 Spring & Fall) Allen Bao, an MScAc student, joined my lab in Spring, and in collaboration with AMD worked on Gameplay Test Automation with Reinforcement Learning. He graduated in the Fall. Congratulations! He is currently Senior Software Development Engineer (ML) at AMD.
• (2021 Summer) Dr. Yangchen Pan defended his PhD! He is my first graduated PhD student and I am very proud of him. He is currently a Departmental Lecturer at the University of Oxford. Congratulations on both achievements!
• (2021-2023) I have not updated this place between 2021 Spring until 2023 Spring. I retroactively add a few important news above.
• (2021 Spring) I developed and taught a new course on Reinforcement Learning. All videos can be accessed through this playlist. The course is accompanied by the Lecture Notes in Reinforcement Learning. 

Continue reading “News”

Textbook

I have been working on a textbook on reinforcement learning started from when I taught the new Introduction to Reinforcement Learning course in Spring 2021. The current version is from 2021. I am hoping to have a major update in 2025.

• A.M. Farahmand, Lecture Notes on Reinforcement Learning, 2021.

  The textbook is introductory in the sense that it does not assume prior exposure to reinforcement learning. It is not, however, focused on being a collection of algorithms or only providing high-level intuition. Instead, it tries to build the mathematical intuition behind many important ideas and concepts often encountered in RL. We prove many basic, or sometimes not so basic, results in RL. If the proof of some result is too complicated, we prove a simplified version of it.

  If you are a university instructor and wish to use slides for your own course, please contact me.

Papers

This list is regularly updated. You can also check my Google Scholar page.

2024

• Amin Rakhsha, Mete Kemertas, Mohammad Ghavamzadeh, and A.M. Farahmand, “Maximum Entropy Model Correction in Reinforcement Learning,” International Conference on Learning Representations (ICLR), 2024. (PDF; arXiv; OpenReview)
• Mark Bedaywi, Amin Rakhsha, A.M. Farahmand, “PID Accelerated Temporal Difference
  Algorithms,” Reinforcement Learning Conference (RLC), 2024. (PDF; arXiv)
• Claas Voelcker, Tyler Kastner, Igor Gilitschenski, A.M. Farahmand, “When does Self-Prediction help? Understanding Auxiliary Tasks in Reinforcement Learning,” Reinforcement Learning Conference (RLC), 2024. (PDF; arXiv)
• Marcel Hussing, Claas Voelcker, Igor Gilitschenski, and A.M. Farahmand, Eric Eaton “Dissecting Deep RL with High Update Ratios: Combatting Value Overestimation and Divergence,” Reinforcement Learning Conference (RLC), 2024. (PDF; arXiv)
• Avery Ma, A.M. Farahmand, Yangchen Pan, Philip Torr, Jindong Gu, “Improving Adversarial Transferability via Model Alignment,” European Conference on Computer Vision (ECCV), 2024. (PDF; arXiv; Code)

2023

• Tyler Kastner, Murat A. Erdogdu, and A.M. Farahmand, “Distributional Model Equivalence for Risk-Sensitive Reinforcement Learning,” Neural Information Processing Systems (NeurIPS), 2023. (PDF; OpenReview)
• Avery Ma, Yangchen Pan, and A.M. Farahmand, “Understanding the robustness difference between stochastic gradient descent and adaptive gradient methods,” Transactions on Machine Learning Research (TMLR), 2023. (PDF; OpenReview) [Featured certificate: ~3.5% of accepted papers]
• Claas A. Voelcker, Arash Ahmadian, Romina Abachi, Igor Gilitschenski, and A.M. Farahmand, “λ-AC: Learning latent decision-aware models for reinforcement learning in continuous state-spaces,” 2023. (arXiv)
• Mete Kemertas, Allan Jepson, and A.M. Farahmand, “Efficient and Accurate Optimal Transport with Mirror Descent and Conjugate Gradients”, 2023. (arXiv)

2022

• Amin Rakhsha, Andrew Wang, Mohammad Ghavamzadeh, and A.M. Farahmand, “Operator Splitting Value Iteration,” Neural Information Processing Systems (NeurIPS), 2022. (PDF; OpenReview)
• Claas A. Voelcker, Victor Liao, Animesh Garg, and A.M. Farahmand, “Value Gradient Weighted Model-Based Reinforcement Learning,” International Conference on Learning Representations (ICLR), 2022. (PDF; OpenReview)
• Guiliang Liu, Ashutosh Adhikari, A.M. Farahmand, and Pascal Poupart, “Learning Object-Oriented Dynamics for Planning from Text,” International Conference on Learning Representations (ICLR), 2022. (PDF; OpenReview; GitHub)
• Jincheng Mei, Yangchen Pan, A.M. Farahmand, Martha White, Hengshuai Yao, Mohsen Rohani and, Jun Luo, “Understanding and Mitigating the Limitations of Prioritized Replay,” Conference on Uncertainty in Artificial Intelligence (UAI), 2022. (PDF)

2021

• A.M. Farahmand and Mohammad Ghavamzadeh, “PID Accelerated Value Iteration Algorithm,” International Conference on Machine Learning (ICML), 2021. (PDF; Extended Version PDF)
• Erfan Pirmorad, Faraz Khoshbakhtian, Farnam Mansouri, A.M. Farahmand, “Deep Reinforcement Learning for Online Control of Stochastic Partial Differential Equations,” NeurIPS Workshop on the Symbiosis of Deep Learning and Differential Equations, 2021. (PDF; OpenReview)

2020

2019

• A.M. Farahmand, “Value Function in Frequency Domain and the Characteristic Value Iteration Algorithm,” In the Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2019. (PDF; Extended Version PDF)
• Mohamed Akrout, A.M. Farahmand, Tory Jarmain, Latif Abid, “Improving Skin Condition Classification with a Visual Symptom Checker Trained using Reinforcement Learning,” International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), 2019. (arXiv)
• Yangchen Pan, Hengshuai Yao, A.M. Farahmand, and Martha White, “Hill Climbing on Value Estimates for Search-control in Dyna,” International Joint Conference on Artificial Intelligence (IJCAI), 2019. (PDF)
• Marc T. Law, Jake Snell, A.M. Farahmand, Raquel Urtasun, and Richard S. Zemel, “Dimensionality Reduction for Representing the Knowledge of Probabilistic Models,” International Conference on Learning Representations (ICLR), 2019. (PDF) (OpenReview)
• Mouhacine Benosman, A.M. Farahmand, Meng Xia, “Learning-Based Iterative Modular Adaptive Control for Nonlinear Systems,” International Journal of Adaptive Control and Signal Processing, Vol. 33, No. 2, pp. 335–355, 2019. (Publisher’s Version)

2018

• A.M. Farahmand, “Iterative Value-Aware Model Learning,” Neural Information Processing Systems (NeurIPS), 2018. (PDF; Extended Version PDF)
• Yangchen Pan, A.M. Farahmand, Martha White, Saleh Nabi, Piyush Grover, and Daniel Nikovski, “Reinforcement Learning with Function-Valued Action Spaces for Partial Differential Equation Control,” International Conference on Machine Learning (ICML), 2018. (PDF; Extended Version PDF)

2017

• A. M. Farahmand, Sepideh Pourazarm, and Daniel Nikovski, “Random Projection Filter Bank for Time Series Data,” In the Proceedings of Advances in Neural Information Processing Systems (NeurIPS), December 2017. (PDF; Extended Version PDF; NeurIPS website)
• A. M. Farahmand, André M.S. Barreto, and Daniel Nikovski, “Value-Aware Loss Function for Model-based Reinforcement Learning,“ The 20th International Conference on Artificial Intelligence and Statistics (AISTATS), April 2017. (PDF; Extended Version PDF) Note: An extended abstract version of this paper appeared at EWRL 2016.
• A. M. Farahmand, Saleh Nabi, and Daniel Nikovski, “Deep Reinforcement Learning for Partial Differential Equation Control,” American Control Conference (ACC), May 2017 (PDF)
• Sepideh Pourazarm, A. M. Farahmand, and Daniel Nikovski, “Fault Detection and Prognosis of Time Series Data with Random Projection Filter Bank,” In the Proceedings of Annual Conference of Prognostics and Health Management Society (PHM), October 2017. (PDF; PHM Version) [This is a mostly empirical version of the Random Projection Filter Bank work geared towards fault detection and prognosis applications. See the NeurIPS paper for more theoretical version (with proof/without proofs).]

Continue reading “Publications”