Technical ExplorationMachine Learning

ML-Agents Maze Training

Training a Pac-Man-inspired agent through perception, discrete decisions, actions, and reward feedback across fixed and randomized maze experiments.

Role

Collaborative ML-Agents developer

Engine

Unity

Platform

Windows

Team

5-person team

Dates

11-25 Sep 2025

Outcome

Technical exploration

What I worked on

A Unity ML-Agents study where a maze agent learns to collect twenty pellets while avoiding two randomly moving enemies.

We designed and implemented the training environment collaboratively as a five-person team.
The agent observes grid-relative locations for itself, enemies, and pellets; the randomized-maze version also exposes wall locations.
The action space uses four discrete movement directions.
Reward shaping combines pellet rewards, a capture penalty with episode restart, an accumulating penalty for movement without progress, and a completion bonus.
We trained 64 environments in parallel to accelerate iteration and evaluated the progression through cumulative-reward graphs and recorded win counts.

Technical details

Programming

Supporting technology

Unity ML-Agents

Development breakdown

Behind the systems

Environment and observation design

Each episode starts with twenty pellets, one agent, and two enemies that move randomly. Observations use custom maze-grid coordinates rather than raw world coordinates. The harder version also reports wall locations from randomized layouts, increasing the amount of state the policy must interpret.

Reward shaping and discrete actions

The agent chooses among four discrete movement directions. It receives a positive reward for collecting a pellet, a negative reward and restart when caught, a small accumulating penalty for movement without progress, and a completion bonus after clearing all twenty pellets.

Parallel training and recorded progress

We duplicated the environment 64 times to train faster. A phone recording was used because the training PC was already under load; during the clip, the cumulative win count rises from 58,095 to 58,125 while some parallel episodes still fail.

Training progression and evaluation

The cumulative-reward plots show a substantial rise during training followed by a long plateau and later improvement. The fixed-maze run became generally capable of clearing the level, though random enemy behavior still produced unsuccessful episodes.

Problems and solutions

Preventing stationary oscillation

Early agents learned to vibrate in place and react to enemies without making meaningful progress toward pellets.

We added a small negative reward for movement without collecting a pellet. The penalty is minor per step but accumulates when the agent repeatedly wiggles without progressing.

Generalization versus learnability

Randomizing maze walls, the agent, enemies, and enemy movement greatly increased the observation space and training difficulty.

We compared the randomized layout with a fixed-maze setup. The fixed layout retained randomized actors and enemies while reducing structural variability enough for more reliable learning.

Training captures

Reflection

The exercise showed how observation design and reward shaping can matter as much as the learning loop itself. The final agent performed well in most runs, but occasional failures remained under randomized enemy behavior.