CREDO | Conformalized Risk Estimation for Decision Optimization

Motivation

From recommended decisions to decision-risk assessment

Traditional predict-then-optimize pipelines often convert data into a single predicted outcome and then solve an optimization problem to produce or justify one decision. That workflow is useful, but it can leave an important question unanswered: how dependable is the prescribed decision when the uncertain outcome differs from the prediction?

CREDO focuses on decision-risk assessment for a prescribed decision. Instead of only asking which decision an optimizer returns, it evaluates whether a human or algorithmic decision remains near-optimal across plausible realizations of the uncertain outcome.

Comparison between conventional predict-then-optimize and CREDO decision-risk assessment. — Figure 1: From prediction to decision-risk assessment
Figure 1 contrasts conventional predict-then-optimize with CREDO's decision-risk assessment perspective: instead of only outputting a decision, CREDO evaluates the risk of a prescribed decision.

Core Question

Given a candidate decision \( z \), what is the probability that \( z \) is not near-optimal when the uncertain outcome \( Y \) is realized?

\[ P\{ z \notin \pi(Y) \} \]

Near-optimal: membership in \( \pi(Y) \), meaning optimal up to a user-specified tolerance for realized outcome \( Y \).

Decision risk: the probability \( P\{ z \notin \pi(Y) \} \) that a prescribed decision \( z \) is not near-optimal under the realized uncertainty.

Distribution-free: the conformal guarantee does not require specifying the full data distribution, under the paper's assumptions.

CREDO Framework

A two-step route to decision-risk assessment

1

Map the decision to an inverse feasible region

For a candidate decision \( z \), CREDO considers the set of uncertain outcomes under which \( z \) would be near-optimal. This region is written as \( \pi^{-1}(z) \).

2

Assess probability mass with conformal prediction

Generative conformal prediction is used to assess whether sufficient probability mass lies inside the inverse feasible region. Under exchangeability and the paper's assumptions, conformal calibration provides finite-sample coverage guarantees for auditing the prescribed decision.

CREDO workflow showing inverse feasible region construction and conformal risk estimation. — Figure 2: CREDO workflow
Figure 2 illustrates the two-step CREDO procedure: mapping a candidate decision to its inverse feasible region, then assessing decision risk using generative conformal prediction.

Why Generative Conformal Prediction?

Using multiple generated samples to avoid relying on a single point prediction

A single point prediction can be too narrow a basis for assessing the risk of a decision \( z \). By working with multiple generated samples, CREDO can assess whether sufficient probability mass is associated with the inverse feasible region \( \pi^{-1}(z) \) and avoid relying solely on one predicted outcome. The goal is a conservative risk assessment, while conformal calibration provides finite-sample coverage guarantees under exchangeability.

Results

What the framework is designed to support

Conservative risk estimation

CREDO is designed to estimate decision risk for prescribed decisions under uncertainty, with conformal calibration used to support conservative coverage statements.

Risk-aware comparison of candidate decisions

The estimated risk can help compare candidate human or algorithmic decisions when reliability is part of the audit or evaluation question.

Ablation insights

The paper evaluates modular framework components across several optimization settings, including how design choices affect the resulting risk estimates.

Interactive 2D CREDO Demo

Audit a candidate decision in a triangular linear program

Explore the LP \( \min_{z \in \mathcal{Z}} \langle y, z \rangle \), where \( \mathcal{Z} \) is the triangle with vertices \( (0,0) \), \( (1,0) \), and \( (0,1) \). The shaded outcome-space region approximates \( \pi^{-1}_{\epsilon}(z) \), the outcomes for which the selected decision remains near-optimal.

Decision space

\( \mathcal{Z} = \mathrm{conv}\{(0,0),(1,0),(0,1)\} \)

Selected \( z \) Candidate vertex

Outcome space

Samples are tested against \( \pi^{-1}_{\epsilon}(z) \).

Near-optimal Not near-optimal \( \pi^{-1}_{\epsilon}(z) \)

Risk estimate

Estimated probability that \( z \) is not near-optimal.

0.00 \( \widehat{\alpha}(z) \)

Lower risk means more generated outcomes keep the prescribed decision inside its inverse feasible region. Higher risk means the decision more often fails the near-optimality audit under the simulated uncertainty.

This is a simplified educational implementation of the CREDO workflow for a 2D linear program. It illustrates inverse feasible regions, generated samples, conformal-style balls, and estimated decision risk. It does not claim to reproduce the full theoretical guarantees or experiments from the paper.

Planned extension

CREDO paper assistant

This planned section would provide a chatbot-style assistant that helps readers navigate the paper, clarify notation, and connect the CREDO framework to decision-risk assessment concepts.

Code

Code repository

Repository link to be added after the project repository is confirmed.

Citation

BibTeX

@article{zhou2025credo,
  title  = {Conformalized Decision Risk Assessment},
  author = {Zhou, Wenbin and Orfanoudaki, Agni and Zhu, Shixiang},
  journal = {arXiv preprint arXiv:2505.13243},
  year   = {2025}
}