How to detect model drift and data drift in production prediction APIs using Evidently AI?

Detect model drift and data drift in production APIs by continuously feeding live request data into Evidently AI monitors and comparing the resulting metrics against a baseline reference dataset.

1. Collect a stable reference – export the feature set used for training (e.g., train_features.csv) and store it in a version‑controlled location (S3, GCS, or artifact store).
2. Stream current API payloads – write a lightweight logger in the inference service that appends each request/response row to a parquet file or a Kafka topic.
3. Define column mapping – tell Evidently which columns are numerical, categorical, and which column is the target.
4. Create a dashboard – use DataDriftTab, CatTargetDriftTab, and RegressionPerformanceTab (or ClassificationPerformanceTab) to compute drift scores and performance metrics.
5. Set thresholds – treat a feature as drifted when drift_score > 0.6 (equivalent to p‑value < 0.05). Flag model degradation when MAE or F1 changes by more than 20 % relative to the reference.
6. Automate alerts – schedule the script in Airflow/Kedro or run it as a K8s cronjob; push alerts to PagerDuty or Slack when any threshold is breached.

Example code

import pandas as pd
from evidently import ColumnMapping
from evidently.dashboard import Dashboard
from evidently.dashboard.tabs import DataDriftTab, RegressionPerformanceTab

reference = pd.read_csv('s3://bucket/train_features.csv')
current = pd.read_csv('s3://bucket/api_requests.csv')

column_mapping = ColumnMapping(
    numerical_features=['age','salary'],
    categorical_features=['city','device'],
    target='label'
)

dashboard = Dashboard(tabs=[DataDriftTab(), RegressionPerformanceTab()], column_mapping=column_mapping)
dashboard.calculate(reference, current)
dashboard.save('drift_report.html')

Quick comparison

| Metric | Baseline | Current | Drift? (threshold) |
|--------|----------|---------|--------------------|
| Age mean shift | 35.2 | 38.7 | Yes (|Δ| > 5 %) |
| City distribution (KL) | 0.02 | 0.18 | Yes (> 0.1) |
| MAE | 0.42 | 0.55 | Yes (+30 %) |

When any row in the table meets the drift condition, trigger a CI/CD rollback or model retraining pipeline.