A 2025 MIT report shows that, even after massive enterprise spending of billions of dollars on GenAI pilots, 95% of organizations are seeing no business returns. AI-related incidents and failures are one of the main things breaking consumer trust and hindering wider adoption of AI. Stanford Institute for Human-Centered Artificial Intelligence released the 2025 AI Index report, which shows that 233 AI-related failures were reported in 2024 alone across multiple organizations.
Imagine deploying a machine learning model that predicts equipment failures in a manufacturing company. During training and the first day of inferencing in production, the model is 95% accurate. Six months later, the accuracy degrades to 70%, and no one is aware. The model keeps making bad predictions, and the business continues trusting it, which finally results in massive failures and disruption of the business.
Now, imagine the damage if the same were to happen in the healthcare industry, where actual lives are at stake. These are not hypothetical situations, but the reality of trust-breaking failures in AI. These are the exact problems AI observability can solve.
What is AI Observability?
AI Observability is the concept of continuously monitoring, understanding, and explaining the behavior of AI systems. It covers three key areas: traditional software observability (i.e., metrics, logs, and traces), data engineering (i.e., data lineage and quality), and finally AI-specific concerns like drift, fairness, and explainability. Think of it as monitoring, debugging, and governance of AI, all encapsulated into one.
Traditional observability tools alone are simply not equipped to handle the several layers of complexity added by the AI systems. Traditional software is deterministic, i.e., the same input gives you the same output. If there is an issue, you would see exceptions, stack traces, etc. However, AI models are probabilistic in nature, i.e., the same input can give you different outputs as the data or context changes. They need a different kind of instrumentation to ensure the models don't silently degrade over time.
Three ways AI models degrade over time
- Data Drift
Data drift happens when the statistical distribution of input features deviates from what the model was originally trained on. A banking fraud detection model trained on pre-pandemic buying patterns might behave differently when the buying habits have permanently changed. In this case, the model hasn’t changed or isn’t broken, but it is just operating outside the scope of its training distribution.
- Concept Drift
Concept drift is more damaging as it is subtler and more difficult to identify. The input distribution might be the same, but the relationship between the model inputs and outputs changes. A credit scoring model might lose its predictive power in scoring applicant profiles as the economic conditions change, and therefore the default risk profile changes fundamentally.
- Label Drift
If the model outputs are used for deriving insights and making decisions, but at the same time are also generating data for the next round of training, you can run into a feedback loop where the model reinforces its own biases. This is very dangerous in recommendations and content moderation.
As you can see, none of the above failure modes produce traditional error signals like bad infrastructure metrics or an increase in errors/exceptions, or an increase in API response times, etc. But the model is still silently wrong.
Real World Impact of AI Model Degradation
The risks of deploying AI without observability are not hypothetical. The Epic Sepsis Model (ESM) is one of the most documented examples of undetected clinical AI degradation. ESM is a proprietary algorithm embedded in Epic’s EHR and deployed across hundreds of hospitals in the US.
ESM was trained on data between 2013 and 2015 from approximately 405,000 encounters. During development and internal testing at Epic, the model reported an AUC of 0.76 - 0.83. But when independently validated at Michigan Medicine in 2021, it achieved an AUC of just 0.63. This means it missed two-thirds of sepsis patients while generating false alerts on 18% of all hospitalizations. A follow-up study in NEJM AI showed that, in certain conditions, the AUC further dropped to 0.47.
Here are the failure modes that could have been caught if observability were in place:
- Feature Drift - The model had no monitoring in place for changes in medication codes, lab distributions, and ICD coding practices that had been updated since 2013
- Performance Proxy Degradation - There was no dashboard tracking the patients flagged as low-risk by the ESM, who were experiencing rising deterioration rates
- Subgroup fairness - No alerts on performance differences across age cohorts or patient demographics over time
- Lineage/Schema drift - Epic’s proprietary, opaque architecture meant upstream EHR schema changes had no visibility
Epic eventually released a V2 of the model with gradient boosting and local retraining capability. A proper observability stack would have surfaced these signals, which were hiding in plain sight.
Five Pillars of AI Observability
1. Data Quality Monitoring
Data Preprocessing is the first step in the Machine Learning lifecycle. Before worrying about model performance, one needs to know whether the data that is being used for training the model is trustworthy. This means monitoring for:
- Schema Violations i.e. missing fields, type changes etc
- Statistical distribution changes i.e. mean, variance, percentile shifts in features
- MIssing value rates i.e. sudden spikes in nulls etc
- Referential Integrity i.e. broken foreign keys and orphaned records etc
2. Model Performance Monitoring
How do we know if the model is still doing what it was trained to do? The challenge in many real world systems is, ground truth feedback arrives late or not at all. You cannot always know immediately whether the model prediction was right.Two strategies address this.
- Proxy Metrics - These are upstream signals correlated with model quality. For example, click-through rates for recommendation models, dispute rates for fraud models and return rates for demand forecasting etc
- Delayed Evaluation - This batches ground truth as it becomes available and continuously compares performance metrics against historical predictions and recomputes
3. Explainability and Interpretability
Knowing something went wrong is not enough. Understanding why it went wrong is true observability. Modern explainability techniques are as below
- SHAP - Shapley Additive Explanations, assigns each feature a contribution value for individual predictions
- LIME - Local Interpretable Model-agnostic Explanations, approximates a model locally with a simple interpretable one
- Attention - For transformer based model, Attention Visualization shows which input tokens the model focused on
- IG - Integrated Gradients, traces prediction back through a neural network to attribute importance to input features
In highly regulated industries like healthcare and finance, explainability is a regulatory requirement.
4. Fairness and Bias Monitoring
A model can be highly accurate on average but failing specific demographic subgroups systematically. For example, in healthcare, a model trained predominantly on data from one population can be inaccurate for under-represented groups, which could have dangerous consequences, even fatal sometimes.
Common fairness metrics include demographic parity (equal positive prediction rates across groups), equalized odds (equal true positive and false positive rates) and individual fairness (similar individuals treated similarly)
5. Lineage and Provenance Tracking
Not if, but when a model makes an unexpected prediction, you need the ability to trace back the entire chain i.e. which model version was used, what was the training data, what were the pre processing steps, which feature engineering pipeline provided the inputs etc. Without lineage, tracking and debugging capabilities, solving a bad prediction is impossible. You can't fix what you can't see.
Architecture of an AI Observability Stack
A production-grade AI observability stack typically has four layers:
- Layer 1: Instrumentation
All prediction requests should emit structured telemetry like input features, model version, prediction output, confidence score, latency and any contextual metadata. This is the raw data that feeds all the downstream layers
- Layer 2: Collection and Storage
Telemetry events flow into a streaming pipeline (Kafka is the typical standard) and are stored in a timeseries friendly format. A queryable lakehouse powered by Apache Iceberg is ideal for storing heterogeneous ML and telemetry due to Iceberg’s features like ACID transactions and schema evolution support.
- Layer 3: Analysis and Detection
This layer runs the statistical tests, trains anomaly detection models on the telemetry itself, and computes drift metrics. Population Stability Index (PSI) and Kullback-Leibler divergence are commonly used for drift detection. Evidently AI and WhyLabs provide pre-built drift detection pipelines.
- Layer 4: Alerting and Action
This is the final layer which translates detections into actions. PagerDuty alerts for severe drift and automatic rollbacks can be set up when performance drops below SLAs. Retraining triggers can be set up to kick off CI/CD pipelines to retrain and redeploy models. Publishing the insights on dashboards helps for human review.
LLM Observability: Beyond classic ML
Observability for Large Language Models (LLMs) further introduces additional complexity because LLM outputs are open-ended text. You cannot really compute accuracy against a ground truth label the same way as classic ML. Different concerns emerge, like below:
- Hallucination - Detecting hallucinations means identifying when the model confidently states something false
- Safety - Toxicity and safety monitoring i.e. detecting harmful, biased or policy violating outputs
- Security - Prompt injection detection i.e. finding malicious inputs designed to hijack model behavior
- Cost - LLM inference is expensive. Observability must track economics like latency and cost per token
- Context- Context window utilization i.e. monitoring how much context is being consumed and whether retrieval is effective
Conclusion
AI Observability is not a feature. It is a foundational layer for AI systems to avoid trust-breaking failures. As organizations move models that make critical decisions from pilots to production, the cost of a silent model failing grows from embarrassing to catastrophic.
The good news is that the AI observability tooling ecosystem has matured significantly. What used to take months or years to build from scratch can now be built with open source components in weeks. The patterns are better understood than before. The cost of not implementing them is far higher in terms of degraded model performance, regulatory risk and eroded trust.
The organizations that will succeed with AI are those that treat their models not as one-time deliverables, but as mission-critical systems that require continuous care, measurement, and understanding. AI observability is how you build that understanding and is fundamental for building trust in AI systems.
References
[1] MIT Technology Review, MIT Report Finds Most AI Business Investments Fail, Reveals 'GenAI Divide' (2025), Virtualization Review
[2] ComplexDiscovery, AI Index Report 2025: A Wake-Up Call for Cybersecurity and Legal Oversight (2025), ComplexDiscovery
[3] J. Harvey et al., A Scoping Review of Reporting Gaps in FDA-Approved AI Medical Devices (2024), npj Digital Medicine
[4] E. Lyons et al., External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients (2021), JAMA Internal Medicine