Dynamic Optimization

This code trains a DDPG agent on the Pendulum problem. With 200 episodes, it should achieve a decent policy (though possibly not perfect; longer training yields better results). In practice, one might incorporate learning rate schedules, noise decay, or more episodes.

'''Pendulum Problem with DDPG:''' The Pendulum-v1 environment is a classic control problem: the agent applies a torque to a pendulum (which starts hanging downward) and tries to swing it up and balance it upright. The state is 3-dimensional (angle represented as cos and sin, and angular velocity) and the action is 1-dimensional continuous (torque ''-2'' to ''+2'' N·m). The reward is highest (less negative) when the pendulum is upright and not moving (and using minimal torque).

* Implement a simplified DDPG for Pendulum.
* Build the actor and critic networks, a replay buffer, and the training loop. We’ll use '''PyTorch''' in code (though one could use TensorFlow similarly). The code will highlight how to connect to the gymnasium environment and perform learning.
* In practice, use an existing library like [[https://stable-baselines3.readthedocs.io/en/master/|Stable Baselines3]].

'''Set up the Pendulum Environment and Networks:'''

Import the necessary libraries. Retrieve ''state_dim'', ''action_dim'', and ''max_action'' (to help scale outputs). Next, define the neural network models for actor and critic. A simple feedforward network is sufficient for this small problem:

The actor outputs a continuous action using a ''tanh'' activation to ensure it stays within ''[-1,1]'', then multiplies by ''max_action'' to scale to [-2,2]. The critic concatenates state and action and outputs a scalar Q-value. Use two hidden layers of 128 units as a reasonable size for this small problem.

'''Replay Buffer:''' DDPG relies on a replay buffer to store transitions and sample them for training. Implement a simple ring-buffer:

The buffer stores transitions and samples a random batch for training as tensors.

Create the networks and set target networks initially equal to the main networks. Use Adam optimizers for both actor and critic. Create the replay buffer. The learning rates (1e-3) are chosen as typical starting points.

'''Training Loop:''' Train the DDPG agent. Here is an outline of the loop and key steps:

* Loop over episodes. For each episode, reset the environment and then loop for a certain number of steps (use 500 as an upper bound to ensure it doesn't loop forever if the environment doesn’t terminate early; Pendulum naturally truncates around 200 steps by default).

* '''Action selection:''' Get the action from the actor network ''actor(state_tensor)'' and then add Gaussian noise for exploration. The noise scale decays by ''exploration_noise * max_action'' (so here 0.1 * 2.0 = 0.2 std dev initial noise). This encourages exploration of different torques. Clip the action to the allowed range.

* '''Storing transitions:''' Store ''state, action, reward, next_state, done'' in the ''ReplayBuffer''.

* '''Training updates:''' Wait until at least one batch worth of samples is in the buffer, then each time step do one gradient update for actor and critic:
** Compute '''target Q''': using the target networks, compute {`y_i = r_i + \gamma (1-d_i)\,Q_{\text{target}}(s_{i+1}, \mu_{\text{target}}(s_{i+1}))`}. ''dones'' is a 0/1 indicator for terminal states; use ''(1 - done)'' to zero-out the future term if the episode ended.

** '''Actor loss:''' maximize the critic’s Q, so minimize ''-critic(s, actor(s))'' (negative sign for gradient descent). This implements the deterministic policy gradient update.
** '''Target network update:''' perform a soft update with factor {`\tau`}. A small {`\tau`} of ''0.005'' means the target networks change very slowly, providing a stable reference for targets.

Gymnasium Environments are custom RL setups:

The '''Pendulum-v1''' environment is a classic control task:

Having covered the theory, we now discuss practical implementation using the '''gymnasium''' library (formerly OpenAI Gym). Gymnasium provides standard interfaces for reinforcement learning environments, facilitating experimentation with various RL algorithms.

** Compute '''target Q''': using the target networks, compute {`y_i = r_i + \gamma (1-d_i)\,Q_{\text{target}}(s_{i+1}, \mu_{\text{target}}(s_{i+1}))`}. ''dones'' is a 0/1 indicator for terminal states; we use ''(1 - done)'' to zero-out the future term if the episode ended.

'''Pendulum Problem with DDPG:''' The Pendulum-v1 environment is a classic control problem: the agent applies a torque to a pendulum (which starts hanging downward) and tries to swing it up and balance it upright. The state is 3-dimensional (angle represented as cos and sin, and angular velocity) and the action is 1-dimensional continuous (torque ''-2'' to ''+2'' N·m). The reward is highest (less negative) when the pendulum is upright and not moving (and using minimal torque). This is a perfect testbed for DDPG.

* '''Action selection:''' We get the action from the actor network ''actor(state_tensor)'' and then add Gaussian noise for exploration. The noise scale decays by ''exploration_noise * max_action'' (so here 0.1 * 2.0 = 0.2 std dev initial noise). This encourages exploration of different torques. We then clip the action to the allowed range.

* '''Storing transitions:''' We store ''state, action, reward, next_state, done'' in the ''ReplayBuffer''.

** '''Critic loss:''' mean squared error between ''current_Q = critic(s,a)'' and ''target_Q''. This corresponds to minimizing {`(Q(s,a) - y)^2`}.
** '''Actor loss:''' we want to maximize the critic’s Q, so we minimize ''-critic(s, actor(s))'' (negative sign for gradient descent). This implements the deterministic policy gradient update.
** '''Target network update:''' We perform a soft update with factor {`\tau`}. A small {`\tau`} of ''0.005'' means the target networks change very slowly, providing a stable reference for targets.

*We will implement a simplified DDPG for Pendulum.
* This involves building the actor and critic networks, a replay buffer, and the training loop. We’ll use '''PyTorch''' in code (though one could use TensorFlow similarly). The code will highlight how to connect to the gymnasium environment and perform learning.
* In practice, one might use an existing library like [[https://stable-baselines3.readthedocs.io/en/master/|Stable Baselines3]] for quick results, but here we write it out to understand the process.)

(:keywords gymnasium, reinforcement learning, RL environments, Pendulum-v1, DDPG, engineering applications:)

!!! Custom Engineering Environment

Environments are custom RL setups:

# Copy weights from actor to target_actor, and critic to target_critic

!!! Gymnasium Examples for Engineering

# Initialize environment and replay buffer.
# Collect experiences and perform network updates:

** Critic loss: minimize squared difference from target Q.
** Actor loss: maximize critic Q-values.
** Use soft updates for target networks.

* Reward: {`-(\Delta \theta^2 + 0.1*\dot{\theta}^2 + 0.001*\tau^2)`}

 ** Critic loss: minimize squared difference from target Q.
 ** Actor loss: maximize critic Q-values.
 ** Use soft updates for target networks.

* '''CartPole''': Balancing pole, discrete actions, robotics/control theory.
* '''MountainCar''': Illustrates energy management.
* '''Acrobot''': Robotic arm analogy.
* '''LunarLander''': Aerospace engineering control.
* '''BipedalWalker''': Biomechanical control using continuous actions.
* '''HVAC or Water Resource Management''': Custom environments applicable to industrial optimization.

Gymnasium provides a test environment to implement RL algorithms across engineering domains.