Prestanda Consulting
Home › Insights › Part 2: Data Drift Is Silently Breaking Your AI Models — Here’s How We Caught It Early
MLOps

Part 2: Data Drift Is Silently Breaking Your AI Models — Here’s How We Caught It Early

How Prestanda Consulting wired MLflow, Great Expectations, and Evidently AI into CI/CD to detect and block data drift before it reached production.

PBy Prestanda Consulting
Sep 18, 2025 · 2 min read

In the first part of this series, we covered how schema drift threatened an AI pipeline and how it was made resilient. But solving schema drift was only the beginning.

Schema drift breaks pipelines loudly. Its quieter twin, data drift, changes the meaning inside otherwise stable schemas. Customer behavior evolves, distributions shift, and models become less aligned with the world they were trained on — often without an error message.

From Schema Drift to Data Drift: The Next Battle

CatBoost and PyTorch models were consuming stable schemas. BigQuery, Firestore, and Feast looked consistent. Yet predictions started slipping. The issue was data drift.

Schema drift breaks pipelines. Data drift breaks trust.

How We Fought Back: MLflow, Great Expectations, and Evidently AI

The engineering team connected three controls to catch drift before it poisoned production:

  • MLflow: logged every CatBoost and PyTorch inference run.
  • Great Expectations: validated bounds, nulls, and business logic with strict checks.
  • Evidently AI: provided drift dashboards comparing baseline and current predictions.

Together, they made drift detection visible and actionable.

Turning Drift Detection Into CI/CD

Drift detection was embedded into delivery pipelines so a compromised model could not pass through unchecked:

  1. MLflow logging wrapped every inference run.
  2. Great Expectations suites validated feature values and ordering logic.
  3. Evidently AI reports compared baseline and current distributions.
  4. CI/CD pipelines failed builds automatically when drift or mismatches were found.
Concrete example: For an MSP vendor value-score ranking, if the order changed unexpectedly, the build failed immediately. Drift was no longer silent; it was blocked at the source.

Why This Matters

  • Schema drift causes pipelines to fail loudly.
  • Data drift causes predictions to fail silently.

Silent failures are particularly dangerous because dashboards can remain green while business KPIs decline. Wiring MLflow, Great Expectations, and Evidently AI into CI/CD made drift detection a release gate rather than a retrospective diagnostic.

Key Takeaways

  • Schema drift breaks pipelines; data drift breaks trust.
  • Data drift is silent but consequential.
  • MLflow, Great Expectations, and Evidently AI made drift detection part of CI/CD.
  • Business rules, including vendor rankings, can be enforced in code.
  • Trust in AI requires monitoring data as closely as models.

Originally published by Prestanda Consulting on Medium. View the original Medium publication.