For successful AI projects, celebrate your graveyard and be prepared to fail fast

AI teams invest a lot of rigor in defining new project guidelines. But the same is not true for killing existing projects. In the absence of clear guidelines, teams let infeasible projects drag on for months.

They put up a dog and pony show during project review meetings for fear of becoming the messengers of bad news. By streamlining the process to fail fast on infeasible projects, teams can significantly increase their overall success with AI initiatives.

AI projects are different from traditional software projects. They have a lot more unknowns: availability of right datasets, model training to meet required accuracy threshold, fairness and robustness of recommendations in production, and many more.

In order to fail fast, AI initiatives should be managed as a conversion funnel analogous to marketing and sales funnels. Projects start at the top of the five-stage funnel and can drop off at any stage, either to be temporarily put on ice or permanently suspended and added to the AI graveyard. Each stage of the AI funnel defines a clear set of unknowns to be validated with a list of time-bound success criteria.

The AI project funnel has five stages:

Image Credits: Sandeep Uttamchandani

1. Problem definition: “If we build it, will they come?”

This is the top of the funnel. AI projects require significant investments not just during initial development but ongoing monitoring and refinement. This makes it important to verify that the problem being solved is truly worth solving with respect to potential business value compared to the effort to build. Even if the problem is worth solving, AI may not be required. There might be easier human-encoded heuristics to solve the problem.

Developing the AI solution is only half the battle. The other half is how the solution will actually be used and integrated. For instance, in developing an AI solution for predicting customer churn, there needs to be a clear understanding of incorporating attrition predictions in the customer support team workflow. A perfectly powerful AI project will fail to deliver business value without this level of integration clarity.

To successfully exit this stage, the following statements need to be true:

• The AI project will produce tangible business value if delivered successfully.
• There are no cheaper alternatives that can address the problem with the required accuracy threshold.
• There is a clear path to incorporate the AI recommendations within the existing flow to make an impact.

In my experience, the early stages of the project have a higher ratio of aspiration compared to ground realities. Killing an ill-formed project can avoid teams from building “solutions in search of problems.”

2. Data availability : “We have the data to build it.”

At this stage of the funnel, we have verified the problem is worth solving. We now need to confirm the data availability to build the perception, learning and reasoning capabilities required in the AI project. Data needs vary based on the type of AI project  —  the requirements for a project building classification intelligence will be different from one providing recommendations or ranking.

Data availability broadly translates to having the right quality, quantity and features. Right quality refers to the fact that the data samples are an accurate reflection of the phenomenon we are trying to model  and meet properties such as independent and identically distributed. Common quality checks involve uncovering data collection errors, inconsistent semantics and errors in labeled samples.

The right quantity refers to the amount of data that needs to be available. A common misconception is that a significant amount of data is required for training machine learning models. This is not always true. Using pre-built transfer learning models, it is possible to get started with very little data. Also, more data does not always mean useful data. For instance, historic data spanning 10 years may not be a true reflection of current customer behavior. Finally, the right features need to be available to build the model. This is typically iterative and involves ML model design.

To successfully exit this stage, the following statements need to be true: