Differentiable Weights-Varying Nonlinear MPC via Gradient-based Policy Learning: An Autonomous Vehicle Guidance Example

We actively maintain practical and empirically validated stability and feasibility through specific regularizations and safeguards within the learning framework.

Our approach to practical stability and feasibility while performing local adaptation of our MPC cost function weights relies on three types of strategies: (1) NMPC problem design, (2) stable and accurate gradients calculation, and (3) architectural safeguards in the learning policy. We will discuss each strategy in the following sections.

1. Underlying NMPC Formulation

We rely on established trajectory-tracking NMPC formulations and explicitly separate the learning of cost function weights from the constraints.

• Separation of Cost Function Weights Learning and Constraints: It is crucial to note that the learning process adapts the cost function weights $\boldsymbol{\theta}$, but does not alter the state-, input-, and terminal constraints $\boldsymbol{h}\big(\boldsymbol{x}(t), \boldsymbol{u}(t)\big) \le \bar{\boldsymbol{h}}$ and $\boldsymbol{h}^e\big(\boldsymbol{x}(T_p)\big) \le \bar{\boldsymbol{h}}^e$. All constraints, including the combined longitudinal-lateral acceleration limits, remain unchanged.
• Terminal Cost Role: In our setting, the terminal cost $m(x(T_p);\theta)$ does not approximate an infinite horizon cost-to-go derived from the Hamilton-Jacobi-Bellman equation, typically one of the central pillars for stability proofs of an MPC. Instead, the terminal cost is defined as the state-dependent component of the stage cost. This design aligns with the NMPC formulations by Grüne [1], Limon et al. [2], Boccia et al. [3], Köhler et al. [4], and Soloperto et al. [5], which all avoid calculating terminal sets or costs. Köhler explicitly argues the benefit of this design for the "general tracking of dynamic reference trajectories" [4]. These theoretical approaches prove that a sufficiently long prediction horizon ensures stability. We experimentally determine an adequate prediction horizon of 2.55 s (Paper, Table I), which effectively balances practical stability and computational load in our application.
• Feasible Warm-Starts: To improve practical feasibility, we utilize the SQP_RTI solution method (see Appendix A). The solver is warm-started with the shifted optimal trajectory from the previous MPC step, providing an interior-point feasible initial guess that helps preserve constraint satisfaction under bounded disturbances.

2. Differentiable MPC Solver Properties for Gradient Calculation

To avoid numerical instabilities during cost function weights learning, we adopt the following strategies:

• Smoothing for Stability: Active set changes can introduce non-differentiable kinks in the solution map, as shown in Frey et al. [6]. To mitigate this and ensure stable gradients, we solve the KKT system to an accuracy of $\tau = 10^{-6}$ (Section IV-A). As shown by Kordabad et al. [7] and Frey et al. [6], setting $\tau > 0$ ensures that solution sensitivities remain bounded in magnitude, whereas $\tau \approx 0$ can lead to unbounded sensitivities at kinks caused by active set changes.
• Sensitivity Accuracy: We adopt an exact Hessian formulation (Section IV-A) in the numerical MPC solver. Frey et al. (Fig. 2) [6] show that using exact Hessians significantly improves the calculation of the solution's sensitivities, in comparison with Gauss-Newton Hessian approximations.

These design choices reduce the risk that the MPC solver returns inaccurate sensitivities, which may negatively affect the learning process and lead to instability or infeasibility.

3. Architectural Safeguards

During our experiments, we identified specific scenarios where an unconstrained learning process drove the MPC into unstable or infeasible regions. We implemented the following architectural safeguards to address these observed failure modes:

• Infeasibility Fallback Mechanism: In cases where the numerical NMPC solver fails to converge, the computed gradients and learning updates for that timestep are discarded. The weights are set to the last feasible set, and the MPC is reinitialized. If the infeasibility persists for two consecutive timesteps, the training episode is terminated, and the previous policy with the best loss over one lap is saved (Section III).
• Enforcing Positive Cost Weights via a Softplus Activation: During certain maneuvers, such as braking zones, we observed that the gradient descent algorithm could occasionally drive specific cost weights toward negative values in an attempt to optimize its path tracking behavior. This violates the requirements of the HPIPM QP solver, which needs a least-square convex cost function, leading to solver infeasibility. To impose the positivity of all cost function weights during training, we implemented a softplus activation function at the output of our policy network (Section III). This guarantees that all cost weights $\boldsymbol{\theta}_k$ remain strictly positive definite, ensuring the underlying optimization problem remains well-posed and feasible throughout the learning process. The softplus activation function provides an additional benefit, as theoretical stability guarantees for weights-varying MPC, such as the approach by Baumgärtner et al. [8], require continuous stage costs, which a softplus activation function offers. This continuity is crucial for the modularity described in this response's summary, ensuring that the necessary assumptions for stability are preserved whenever a provably stable MPC is integrated into our learning framework. However, formal considerations regarding the NMPC stability during online weight adaptation are left for future work.
• Stabilizing Updates via Gradient Clipping, Accumulation, and Conservative Learning Rates: We observed momentary, large spikes in the loss function gradient terms of our Differentiable Weights-Varying MPC. Such large gradients caused the policy to make erratic, large updates that could push the controller into unstable regions in a single step. To mitigate this, we implemented gradient clipping and accumulated gradients over a batch before applying an update (Section III). In our application, gradients are clipped to the range $[-0.1, +0.1]$, and we set the batch size $N_{batch}$ to 10 timesteps.
  
  Additionally, we used a conservative learning rate ($2.9 \times 10^{-5}$ for our experiments), which is low enough to prevent rapid learning that could introduce instability. This combined approach effectively smooths out the policy updates and helps avoid sudden behavioral changes that could destabilize the vehicle.
• Including the Control Inputs into the Loss Function: In the initial phases of our experiments, our loss function (Equation 4) focused primarily on minimizing path tracking and velocity deviations. However, we observed that without explicit regularization terms in the loss, the learned policy lacked sufficient incentive to maintain stable control inputs. In these cases, larger jerk and steering rate inputs could lead to expected, momentarily improved path tracking behavior by the MPC, but introduce instability in subsequent timesteps, ultimately leading our vehicle to spin out. We therefore introduced conditional penalties (Equations 4-5, Section III) that apply a standard quadratic cost within a nominal operating range, but transition to a steep exponential penalty when control inputs exceed defined thresholds. This formulation enables the policy to utilize the vehicle's full agility within safe limits while imposing incentives against instability. We observed that scenarios that previously resulted in spin-outs now track the reference line in a stable manner.

Summary: While we do not claim theoretical guarantees for recursive feasibility or stability, we provide useful measures to favor empirical/practical stability and feasibility. By building directly upon the ACADOS framework, our implementation offers clear, standardized interfaces. This modularity allows users to easily replace the underlying control formulation with their own custom MPC implementation, including those designed with inherent theoretical guarantees for stability and recursive feasibility.

References