Dynamic Optimization

Reinforcement Learning (RL) is a paradigm of machine learning and optimal control where an agent learns to make decisions by interacting with an environment to maximize a cumulative reward. Unlike supervised learning, the agent isn’t given correct actions but instead experiments with actions, learning through feedback (rewards). The agent observes the current state of the environment, takes an action, and receives a reward (scalar feedback signal), then the environment transitions to a new state. This loop continues over time (see Figure 1). The agent seeks a policy (mapping states to actions) that maximizes expected cumulative rewards. Key concepts:

• States: Observations of the environment.
• Actions: Decisions made by the agent.
• Rewards: Immediate feedback signals.
• Policy: Strategy to select actions.

Agents face the exploration vs. exploitation dilemma: exploring new actions to find higher rewards vs. exploiting known rewarding actions.

Markov Decision Processes (MDPs):

RL problems often use Markov Decision Processes (MDPs), a mathematical framework for sequential decision-making under uncertainty. An MDP is defined by a tuple:

$$ \mathcal{M} = (\mathcal{S}, \mathcal{A}, P, R, \gamma) $$

where:

• `\mathcal{S}` = Set of states.
• `\mathcal{A}` = Set of actions.
• `P(s'|s,a)` = Transition probability to state `s'` given state `s` and action `a`.
• `R(s,a,s')` = Reward from transitioning state `s` to `s'` via action `a`.
• `\gamma \in [0,1)` = Discount factor, weighing future vs. immediate rewards.

MDPs satisfy the Markov property: future states depend only on current state and action. The optimal policy `\pi^*(s)` maximizes expected long-term rewards. MDP solutions involve computing value functions `V(s)` or action-value functions `Q(s,a)`. The Bellman optimality equation for value functions:

$$ V^*(s) = \max_{a \in \mathcal{A}} \sum_{s'} P(s'|s,a)\left[ R(s,a,s') + \gamma V^*(s') \right] $$

Similarly, the optimal Q-value:

$$ Q^*(s,a) = \sum_{s'}P(s'|s,a)\left[R(s,a,s') + \gamma \max_{a'}Q^*(s',a')\right] $$

Model-Free vs. Model-Based RL:

A key distinction:

• Model-based RL: Uses or learns a model `P(s'|s,a)` and reward function, enabling planning and simulation (e.g., AlphaZero).
• Model-free RL: Learns directly from trial-and-error interaction, without explicit models. Common and simpler, but typically requires more environment interactions.

Hybrid approaches like Dyna-Q use learned models to simulate additional experiences.

Applications of RL in Engineering Optimization:

RL applies broadly to engineering tasks involving sequential decision-making or control:

• Chemical Engineering: Process control and reaction optimization (e.g., reactor settings, energy minimization, yield improvement).
• Mechanical Engineering: Robotic control, autonomous systems (e.g., robotic arms, drones, inverted pendulum).
• Automotive: Autonomous driving (lane-keeping, cruise control, collision avoidance).
• Industrial Energy Management: HVAC optimization (e.g., DeepMind reduced Google data center cooling energy by ~40%).

These applications demonstrate RL’s effectiveness in engineering optimization, addressing complex, uncertain, and dynamic conditions.