I spent three years under a strict, self-imposed constraint; listening exclusively to Kanye West (seriously; I wrote about it a few times). It was a bizarre yet highly effective exercise in focusing on one creative voice to unlock my own productivity. That same strategic application of constraint is what we now need to apply to the chaos of Generative AI.
The introduction of large-scale AI has been driven by hype and a shallow chase for engagement. We have deployed these powerful, probabilistic systems and then attempted to measure their success using the same old, deterministic metrics we applied to simple features. That is a strategic error, and it is costing us.
The conversation needs to move from volume to integrity.
The failure of the short-term engagement metric
For over a decade, we have been obsessed with the North Star Metric (NSM): Daily Active Users, Time to Resolution, or similar vanity metrics that capture short-term usage. This approach works for simple, deterministic products; if a user logs in, they are active.
However, an NSM like ‘AI-Generated Outputs Per Session’ becomes a dangerously misleading lagging indicator when applied to a generative system. Consider an AI that is supposed to summarise complex, real-time broadcast data for a client. If the model hallucinates a critical piece of information, or breaches compliance by leaking PII, it has technically succeeded under the NSM: an ‘output’ was generated and the user spent ‘time’ reviewing it.
The output was a catastrophic product failure, yet the metric registered a win. The NSM measures interaction; it utterly fails to quantify the reliability, safety, and strategic alignment that truly define value in an enterprise AI product. As a Head of Product, my focus must be on P&L protection and sustained value, not temporary spikes in user interaction.
Introducing the System Integrity Score (SIS)
To manage risk and deliver long-term value, we need an executive-level KPI that acts as a leading indicator of the system’s health. I call this the System Integrity Score (SIS).
The SIS is a single, composite metric designed for consumption by leadership and the board. It forces a strategic focus on the four critical vectors of responsible AI deployment. The score provides a clear, quantitative measure of our confidence in the system’s ability to maintain its defined utility over time.
Model alignment and drift velocity
The core of the SIS is the stability of the model’s value proposition. Model Alignment quantifies the current efficacy; how well the model’s outputs are meeting the intended purpose and user need, measured against a human-validated baseline. This is where we measure the quality of the ‘answer’.
Drift Velocity is the strategic measure of risk. It quantifies how quickly the model’s performance and alignment are degrading over time as it is exposed to new data and real-world prompts. A high drift velocity is a strategic alert; it means the system is rapidly becoming unpredictable and costly to maintain. We need to organise our resources around minimising this decay.
Compliance, safety, and business value
The other two pillars link system health directly to P&L and regulatory exposure. Compliance Adherence is a non-negotiable threshold measurement of the system’s safety guardrails. This includes PII masking, adherence to fairness metrics, and the speed of our mitigation response to novel security exploits. Failure in this category should instantly put the SIS into a critical state.
Finally, Business Value Realisation moves beyond usage to measure the actual strategic impact: cost reduction via automation, revenue uplift from new capabilities, and a clear link to a strategic objective. This ensures the technology is not just ‘doing a thing’, but ‘doing the right thing’ for the business.
Shifting the strategic focus
The System Integrity Score forces a complete shift in conversation. It moves us away from rewarding tactical product teams for driving high interaction volumes and towards rewarding strategic product leadership for delivering robust, compliant, and valuable AI systems.
AI is a chaotic, powerful force. Our job as leaders is to impose a rigorous framework of strategic constraint to ensure it serves the business, not just our users’ short-term curiosity. By measuring integrity first, we can ensure responsible deployment and build enterprise products that truly age well.
Integrity score is must have. If no score, how we trust the decision? It is too important for compliance now.
My concern is that the integrity score itself becomes the target for manipulation. You’re just outsourcing bias to a new metric. We need transparency, not a single opaque score.