Chapter 5: Defining Requirements, Constraints, and Success Metrics
"We need the recommendation engine to be 95% accurate." The business stakeholder delivered this requirement with the confidence of someone who'd done their homework. The data science team nodded politely, then spent the next hour explaining why accuracy isn't a meaningful metric for recommendation systems, why 95% might be impossible or unnecessary, and why the real question is what they're trying to optimise for.
This conversation happens in every data science project. Business stakeholders translate their needs into what they think are technical requirements. Data scientists translate technical possibilities into what they think are business requirements. And product managers get stuck in the middle, trying to bridge two worlds that often seem to speak different languages.
The solution isn't to become fluent in both languages, though that helps. The solution is to develop a systematic approach to requirements definition that starts with business outcomes, incorporates technical constraints, and results in success metrics that everyone can understand and commit to.
Many experienced practitioners learned this lesson through painful experience. Early in their careers, they would jump straight into model development without clearly defining what success looked like from the customer's or business perspective. The result was technically impressive solutions that created no business value because they solved the wrong problems or optimised for the wrong outcomes.
The proven framework for defining requirements starts with a deceptively simple question: "What should be achieved for the project to be a success?" But the key is describing success from the customer's or business point of view, not from a technical point of view.
This customer-centric approach to requirements definition changes everything. Instead of "build a recommendation engine with 95% accuracy," you might define success as "help customers discover products they wouldn't have found otherwise, leading to increased engagement and purchase satisfaction." The technical requirements flow from this business requirement, not the other way around.
But data science requirements definition goes beyond traditional product requirements in several important ways. You need to account for uncertainty, tradeoffs, and constraints that don't exist in traditional software development.
The first unique aspect is defining requirements in terms of outcomes rather than outputs. Traditional software requirements often focus on what the system should do. Data science requirements need to focus on what the system should achieve, recognising that there might be multiple ways to achieve those outcomes.
Consider fraud detection. An output-focused requirement might be "flag transactions that are likely to be fraudulent." An outcome-focused requirement might be "reduce fraud losses while minimizing customer friction from false positives." The difference is subtle but important. The first requirement assumes you know how to solve the problem. The second requirement acknowledges that the solution involves tradeoffs that need to be managed.
This outcome-focused approach helps you avoid the trap of over-specifying technical solutions before you understand the problem space. It also creates space for data scientists to propose solutions that you might not have considered.
The second unique aspect is defining requirements as constraints rather than specifications. Industry experts emphasise that "clearly defined constraints free us to do anything except breach those constraints, empowering us to innovate." In data science, constraints often drive more innovation than specifications.
Instead of specifying that your fraud detection system should achieve exactly 95% precision and 90% recall, you might constrain it to achieve at least 90% precision (because false positives are expensive) while maximising recall within that constraint. This gives data scientists freedom to explore different approaches while ensuring that the solution meets business needs.
Constraints can come from multiple sources. Business constraints might include budget limitations, timeline requirements, or performance thresholds. Technical constraints might include latency requirements, infrastructure limitations, or data availability. Regulatory constraints might include privacy requirements, fairness considerations, or explainability needs.
The key is identifying which constraints are hard (must be met for the solution to be viable) and which are soft (nice to have but not essential). Hard constraints become design parameters that shape your solution approach. Soft constraints become optimisation targets that you try to achieve within the hard constraints.
The third unique aspect is accounting for production requirements from the beginning. Experienced practitioners learned that considering production requirements early can fundamentally change your approach to the problem. If engineering has a requirement for sub-80ms response times, that might eliminate certain model architectures before you even start development.
Production requirements include performance requirements (latency, throughput, availability), operational requirements (monitoring, debugging, updating), and resource requirements (compute, storage, bandwidth). These requirements often involve tradeoffs with model sophistication and accuracy.
A complex deep learning model might be more accurate than a simple linear model, but it might also be too slow for real-time applications or too expensive to run at scale. Understanding these tradeoffs upfront helps you focus on solutions that can actually be deployed and sustained.
The fourth unique aspect is defining success metrics that reflect actual user experience rather than just technical performance. Leading practitioners give a perfect example: if you're building a recommendation system that suggests 100 products but the user interface only shows 5, your success metrics should focus on the top 5 recommendations, not all 100.
This requires "dogfooding" your own product to understand how customers actually experience it. You can't define meaningful success metrics from a conference room or a Jupyter notebook. You need to use your product the way customers use it and understand what matters for their experience.
Traditional machine learning metrics like accuracy, precision, and recall are important for model development, but they're often not sufficient for product success. You need business metrics that connect model performance to user outcomes and business outcomes.
For a recommendation system, technical metrics might include precision@5, recall@10, and diversity scores. Business metrics might include click-through rates, conversion rates, and user satisfaction scores. The key is understanding how the technical metrics relate to the business metrics and optimising for the metrics that actually matter for your use case.
The fifth unique aspect is defining requirements that account for data quality and availability. Your requirements need to be achievable with the data you actually have, not the data you wish you had.
This might require doing quick data exploration before finalising requirements. If your historical data only covers six months, you might not be able to detect seasonal patterns that require a full year of data. If your user behaviour data is sparse for certain user segments, your personalisation might not work well for those segments.
Industry experts recommend implementing a quick baseline as part of requirements definition. This baseline helps you understand what's possible with your current data and approach, which helps you set realistic expectations for more sophisticated solutions.
If stakeholders want 95% accuracy but your baseline only achieves 60%, you need to have an honest conversation about whether closing that gap is feasible and what it would take. This conversation might lead to adjusted requirements, additional data collection, or a phased approach that builds toward the ultimate goal over time.
The sixth unique aspect is defining requirements that enable learning and iteration. Data science projects often involve significant uncertainty about what's possible and what's valuable. Your requirements should create space for learning and adaptation rather than locking you into a specific approach.
This might mean defining learning milestones alongside delivery milestones. Instead of just committing to deliver a fraud detection system in six months, you might commit to understand what data predicts fraud in month one, build a baseline model in month two, and achieve target performance in month six.
These learning milestones help stakeholders understand progress even when you haven't delivered the final solution yet. They also create opportunities to adjust requirements based on what you learn along the way.
The seventh unique aspect is defining requirements that account for model lifecycle management. Machine learning models don't just get deployed and forgotten. They need to be monitored, updated, and potentially retrained as conditions change.
Your requirements should address how you'll detect when models are performing poorly, how you'll update them, and what happens when they fail. This might include requirements for model monitoring, automated retraining, or fallback systems that activate when primary models fail.
These operational requirements often affect your technical approach. A model that's easy to retrain might be preferable to a more accurate model that's difficult to update. A model that fails gracefully might be preferable to a more sophisticated model that fails catastrophically.
The final aspect is communicating requirements in ways that build stakeholder confidence while acknowledging uncertainty. Traditional requirements documents often imply certainty about what will be delivered and when. Data science requirements need to communicate your current best understanding while acknowledging what you don't know yet.
This doesn't mean being vague or uncommitted. It means being honest about what you know, what you don't know, and how you plan to learn. "Based on our current data exploration, we expect to achieve 85-90% precision in fraud detection. However, this estimate depends on data quality improvements that we'll complete in the first month. If data quality is worse than expected, we'll need to adjust our approach or timeline."
This level of transparency requires confidence and credibility. You need to demonstrate that you understand the challenges and have a thoughtful approach to managing them. But when you get this right, you build trust and create space for learning and adaptation.
Defining requirements, constraints, and success metrics for data science products isn't just about managing expectations. It's about creating alignment around what you're trying to achieve, what constraints you need to work within, and how you'll know when you've succeeded.
When you get this right, you don't just avoid misunderstandings and disappointments. You create a foundation for innovation within appropriate boundaries. You enable data scientists to explore creative solutions while ensuring that those solutions meet business needs. And you build stakeholder confidence in your approach even when individual experiments don't work out.
The next chapter will show you how to turn these well-defined requirements into effective collaboration with data scientists, engineers, and business stakeholders who each bring different perspectives and priorities to the project.