GoKawiilThis post is the third and final post in our series on quadcopter simulation. The first post and second post derived and simulated the quadcopter’s equations of motion, first in 2D and then in 3D. In this post we train a reinforcement learning policy to fly the quadcopter, first to hover at a fixed point, then to fly through a sequence of gates.
The previous posts were tutorial-like, we started from a free body diagram and, step-by-step, arrived to simulation code. This post instead is more recipe-like, a collection of tricks and techniques I found useful to train an RL policy for quadcopter racing. There is no rigorous proof of why these methods work, only empirical.
Because it is a recipe and not a walkthrough, the post shows only the code that matters for each decision. The full environment is on GitHub at mrandri19/quadcopter-racing.
Here is the trained policy flying the eight-gate loop track. The visualizations throughout the post are made with Rerun.
An incremental approach
Trying to implement a drone racing simulation, RL environment, reward, and model all at once just doesn’t work. Too many things can be slightly off, affecting the policy’s performance, with no good way of debugging it. So we split the task in three stages, each building on top of each other:
The “Hello, world! of RL”: training a policy to solve the inverted pendulum control problem with PPO and a vectorized stable-baselines3 env. Building a custom quadcopter RL environment using MuJoCo, choosing action and observation spaces, splitting high-level RL-based and low-level P-controller, and a hovering reward. Extending the environment from hovering to racing, with a new reward, observation, and random initialization design.
Solving inverted pendulum with vectorized PPO
Let’s start from the basics and setup our vectorized RL problem. We use stable-baselines3’s PPO together with a vectorized InvertedPendulum-v5 (cartpole) environment, and check that the policy reaches the maximum episode length.
from stable_baselines3 import PPO from stable_baselines3.common.env_util import make_vec_env from stable_baselines3.common.policies import ActorCriticPolicy def main () -> None : env = make_vec_env ( "InvertedPendulum-v5" , n_envs = 128 ) model = PPO ( policy = ActorCriticPolicy , env = env , learning_rate = 3e-3 , # 3e-4 (default lr) * sqrt(n_envs) ~= 3e-3 n_steps = 512 , # lower than default 2048, no need to have that many steps with 128 envs. batch_size = 1024 , # higher than default 64 for better efficiency. n_epochs = 5 , # lower than default 10, speeds up training, no need to refit 10 times verbose = 1 , ) model . learn ( total_timesteps = 750_000 )
... continue reading