Reward Shaping via Large Language Models for Sparse-Reward Reinforcement Learning Tasks

Reinforcement learning agents struggle with sparse reward signals, often requiring millions of environment interactions to discover reward-yielding behaviors in complex domains. We explore using large language models (LLMs) as reward shapers: given a task description, an LLM generates dense, semantically grounded intermediate rewards that guide the agent toward task completion. Our framework, LLM-RewardShape, queries the LLM to evaluate action sequences against natural language task descriptions and translates semantic evaluations into scalar reward supplements. Applied to NetHack, MiniGrid, and a custom robotic manipulation environment, LLM-RewardShape reduces sample complexity by 3.7–8.4× compared to PPO with sparse rewards and outperforms prior reward shaping methods by 23–41% in final task success rate.