Canary Testing & Model Rollout
Gradually roll out new models or provider changes with traffic splitting, real-time metric comparison, and automatic rollback on regression.
Features
Traffic Splitting
Route a percentage of traffic to a canary model while the rest stays on the stable baseline. Adjust the split from 1% to 100% via dashboard or API.
Metric Comparison
Compare latency, error rate, cost per request, and custom quality scores between canary and baseline in real time. Statistical significance testing highlights meaningful differences.
Automatic Rollback
Define rollback conditions — e.g., error rate > 5% or p95 latency > 2s. CloudVera automatically shifts traffic back to the baseline if thresholds are breached.
Promotion Workflow
When canary metrics are healthy, promote with one click. The canary becomes the new baseline and the previous model is archived with full metric history.
Steps
Create Canary
Select a project and choose the canary model. Set initial traffic percentage (recommended: 5%).
Define Success Criteria
Set thresholds for latency, error rate, cost, and optional quality scores. These determine auto-rollback and promotion eligibility.
Monitor
Watch real-time dashboards comparing canary vs. baseline. CloudVera runs statistical tests and flags significant regressions.
Promote or Rollback
Promote the canary to become the new default, or roll back to the baseline. All decisions are logged in the audit trail.