01 The environment
The agent observes its position and orientation in 3D plus a vector from the tip of its beak to the nearest flower's nectar hitbox. The baseline is purely extrinsic: reward arrives only when the beak lands inside the hitbox.
The question
Standard practice rewards an agent only for the goal itself. Would paying it for behavior that tends to lead there train it faster? I built a hummingbird agent in Unity that collects nectar from flowers, and gave one copy a reason to explore the right area.
How it was built
02 The intrinsic reward
Triangular prism triggers in a circular pizza slice pattern at the same Z level as the flowers. Entering one pays a small reward and disables that trigger for the episode, so the agent is paid for covering new ground rather than hovering where it already was.
The idea is narrow on purpose: reward exploring the altitude where flowers live, before the agent has touched one. If it works, it should show up as faster convergence.
03 Training and evaluation
Both models trained through Unity ML-Agents against a Python backend, with a config file pinning the architecture and hyperparameters so the only difference between runs was the reward. Compared in TensorBoard.
Technologies
- Unity
- Unity ML-Agents
- Python API for the network interface
- C# for scripts and trigger logic
- Anaconda
- TensorBoard
Results
What came out of it
The location based intrinsic reward converged faster than the extrinsic only baseline, spending more of the episode where reward was possible. Reward shaping is worth deciding early, since it shapes what the agent spends its time on.