Reinforcement Learning (RL) is a paradigm of machine learning and optimal control where an agent learns to make decisions by interacting with an environment to maximize a cumulative reward. Unlike supervised learning, the agent isn’t given correct actions but instead experiments with actions, learning through feedback (rewards). The agent observes the current state of the environment, takes an action, and receives a reward (scalar feedback signal), then the environment transitions to a new state. This loop continues over time (see Figure 1). The agent seeks a policy (mapping states to actions) that maximizes expected cumulative rewards. Key concepts:
Agents face the exploration vs. exploitation dilemma: exploring new actions to find higher rewards vs. exploiting known rewarding actions.
Markov Decision Processes (MDPs):
RL problems often use Markov Decision Processes (MDPs), a mathematical framework for sequential decision-making under uncertainty. An MDP is defined by a tuple:
$$ \mathcal{M} = (\mathcal{S}, \mathcal{A}, P, R, \gamma) $$
where:
MDPs satisfy the Markov property: future states depend only on current state and action. The optimal policy `\pi^*(s)` maximizes expected long-term rewards. MDP solutions involve computing value functions `V(s)` or action-value functions `Q(s,a)`. The Bellman optimality equation for value functions:
$$ V^*(s) = \max_{a \in \mathcal{A}} \sum_{s'} P(s'|s,a)\left[ R(s,a,s') + \gamma V^*(s') \right] $$
Similarly, the optimal Q-value:
$$ Q^*(s,a) = \sum_{s'}P(s'|s,a)\left[R(s,a,s') + \gamma \max_{a'}Q^*(s',a')\right] $$
Model-Free vs. Model-Based RL:
A key distinction:
Hybrid approaches like Dyna-Q use learned models to simulate additional experiences.
Applications of RL in Engineering Optimization:
RL applies broadly to engineering tasks involving sequential decision-making or control:
These applications demonstrate RL’s effectiveness in engineering optimization, addressing complex, uncertain, and dynamic conditions.