Reinforcement Learning with Unreal Learning Agents

I finished reinforcement learning tutorial using Unreal's Learning Agents plugin some time ago. This post is for people who did or interested in doing the tutorial. It was really cool seeing the cars learning how to drive on a track they have never seen before. But I didn't really understand much about the underlying mechanism, namely reinforcement learning.

My interest for this just grew as I started reading Reinforcement Learning book from Richard S. Sutton and Andrew G. Barto and I decided to revisit the same tutorial to understand what is going on under the hood. The book is freely avaiable here. While doing some reading I bumped into a debugging tool called TensorBoard, the standard for debugging artificial neural networks. Honestly it gave me the same feeling I had when I first learned about proper debuggers almost 2 decades ago.

To make it work with Unreal Editor, you need to enable it in Trainer Training Settings in your BP_SportsCarManager blueprint. Alt text

BP_SportsCarManager contains Trainer Training Settings variable

tensorboard_check.png

You also need to install TensorBoard specifically for the Python environment that ships with Unreal Editor. I really like that Unreal comes with its own Python Environment, unlike Unity. It is easier to start and I personally hate installing libraries without knowing what they are for.

pip -m install tensorboard

Even though I successfully installed TensorBoard, I got the following warning when I started the simulation. LogLearning: Display: Subprocess: Warning: Failed to Load TensorBoard: No module named 'tensorboard'. Please add manually to site-packages. LogLearning: Display: Subprocess: "UseTensorBoard": false,

Since Unreal logs are really crappy, it took me some time to figure this out. I needed to add this path C:/Program Files/Epic Games/UE_5.5/Engine/Binaries/ThirdParty/Python3/Win64/Lib/site-packages to the Additional Paths list under Edit>Project Settings>Plugins>Python in Unreal Editor.

Alt text

Python plugin settings

After adding the path to the list, I got this log which means that logging is active. LogLearning: Display: Subprocess: "UseTensorBoard": true,

And then finally to see the visualized data, you need to run TensorBoard. Go to C:/Program Files/Epic Games/UE_5.5/Engine/Binaries/ThirdParty/Python3/Win64/Scripts and you should have tensorboard.exe there. If not go to the installation step. By running TensorBoard .\tensorboard.exe --logdir="[YourProjectPath]\Intermediate\LearningAgents\TensorBoard\runs"

If it starts successfully, you should see a log like this TensorBoard 2.20.0 at http://localhost:6006/ (Press CTRL+C to quit) The Address is where you can see the visualized data, just open your browser and paste the address

You need to run your simulation for a while to see some data in TensorBoard. Alt text

TensorBoard dashboard

The tensorboard visualization made immediate sense to me since I was reading about rewards and general theory behind it. I guess I got lucky here.

Now we can see what is going in our car's brain, thanks to TensorBoard and now it is time to play around with rewards. I'll write this in more detail in another post but basically rewards are how a reinforcement learning method works. Rewards can be either positive or negative which sounds more like punishment. :P

** Show Unreals reward nodes here and explain the values

** and how bad reward values affect tensorboard graphs

Cases

Very low crash reward(-1000) how it makes the car very hesitant to move and go slow becuase not crashing is 1000x more important than moving
Zero crash reward, how the car just darts and