Risk Approval Simulation: Evaluating Policy Changes Before They Ship.
A simulation framework that turns proposed risk-model and policy changes into structured (population, policy, decomposition) triples — so stakeholders can compare approval, mix, and segment sensitivity before anything ships, with every assumption named as a first-class object.
Context
Risk-policy changes — a tweaked threshold, a new segment carve-out, a model swap — compound in non-obvious ways across approval rate, mix quality, and downstream loss. Teams were making these decisions with back-of-envelope estimates and partial historical snapshots, because the tooling to rigorously compare scenarios didn't exist.
The mandate was to build a simulation layer that could turn 'I think this change will do X' into 'here is the expected approval / mix / loss surface under this change, with the historical population it was evaluated against named explicitly.'
Business Problem
“The question was never whether the change would work — it was how much of the curve it would shift, and which segments would bear it.”
Before the framework existed, evaluating a policy change meant an analyst pulling ad-hoc SQL against a historical window, applying the new rule by hand, and presenting a single point estimate in a slide. That approach hid the sensitivity of the result to the window chosen, the population filters applied, and the composition of who-got-approved-before-vs-after. Stakeholders couldn't tell which part of the delta was the policy change and which was the analyst's implicit assumptions — so every review round re-litigated the setup instead of the decision.
- Assumption Opacity
- Mix-Shift Blindness
- Non-Reproducible Analysis
- Stakeholder Misalignment
The Senior Approach
The framework separates three things that had been tangled: the historical population you're evaluating against, the policy you're applying, and the decomposition of the result. Any scenario is a triple of (population, policy, decomposition) — and because each is explicit, a reviewer can challenge any one without re-running the whole analysis.
Population Module
Standardized the historical snapshots teams could run against — documented cutoffs, inclusion rules, and composition — so 'which window' stopped being an implicit analyst choice.
Policy Engine
Expressed proposed changes as structured policy objects the engine could apply uniformly, so a threshold tweak and a full model swap ran through the same code path.
Decomposition Layer
Output the expected shift as approval rate, mix breakdown, and segment-level sensitivity — so stakeholders could see not just the headline but the distribution underneath it.
Realistic System Architecture
Representative Simulation Surface
# A scenario is a named (population, policy) pair — the engine
# returns a structured delta that can be compared to baseline or
# to another scenario side-by-side.
def simulate(scenario: Scenario) -> ScenarioResult:
pop = load_population(scenario.population) # documented cutoff
base = apply_policy(pop, scenario.baseline)
cand = apply_policy(pop, scenario.candidate)
return ScenarioResult(
approval_delta=cand.approval_rate - base.approval_rate,
mix_shift=segment_breakdown(base, cand),
sensitivity=bootstrap_ci(pop, scenario.candidate),
population_manifest=pop.manifest, # explicit for review
)Outcome & Senior Judgment
The Impact
Key Takeaways
- 1
Make the population explicit.Most 'my simulation disagrees with yours' disputes are population disputes. Naming the historical window as a first-class object closes that gap faster than any chart does.
- 2
Policies are objects, not slides.When a proposed change is expressed as code the engine can execute, review shifts from arguing about point estimates to challenging specific assumptions in a specific object.
- 3
Decomposition is where trust gets built.Stakeholders don't need headline numbers; they need to see which segments moved. The framework's leverage came from always showing the underneath.