4 Common Pitfalls In Putting A Machine Learning Model In Production

I spoke at a conference recently and one of the talks really resonated with me. It revolved around hosting, securing, and productionizing machine learning models.

The speaker asked the audience, “Who in this room has developed a machine learning or artificial intelligence model for their business?” Being a technology conference, 80–90% of the hands shot up.

“Now,” he continued, “how many of you have that code in production?” Nearly every hand went down. The demonstration was so simple, yet it was incredibly effective. Nearly everyone wants to inject ML into their business and understandably so, but these same people are running into a massive issue: it can be very, very difficult to make that model sustainable, especially with a cloud architecture.

In my experience, I’ve found a few main roadblocks that I’ll share with you, along with the ways we here at AiQ have gone about solving them.

Do you like this in-depth educational content on applied machine learning? Subscribe to our Enterprise AI mailing list to be alerted when we release new material. 

1. Don’t re-invent the wheel.

You’ve heard this a million times, and it’s not changing now. I’ve seen countless examples of projects that fail because they refuse to utilize solutions that already exist. Amazon Web Services (AWS) and Google Cloud have absolutely phenomenal machine learning suites and products for you to use, right out of the box.

Will they work for absolutely every use case? No, probably not. But they’re a great place to start, especially if you doesn’t have a wealth of machine learning expertise on staff.

Above is a sample of the information you can extract from Google Cloud’s Vision API. Say I had images or video feeds of my customers using my product (maybe in a focus group or similar setting) and I want to know how they’re reacting to the product based on facial cues. I could simply feed my images into Google Vision and receive the most likely emotions that each face is exhibiting.

When we’re building solutions for you, we want to build the best product. Period. We want to align with your cost goals and meet your technical requirements. Typically, offerings on AWS or Google Cloud allow for the best balance in terms of cost and results. Additionally, maintenance tends to be significantly easier, as the platform will handle any version updates, feature additions, or outages.

With that being said…

2. Usually solutions aren’t cookie cutter.

For smaller scale projects, an out-of-the-box solution may be enough, but typically for larger initiatives, cost will either become too prohibitive or too much custom functionality will be necessary.

These projects typically require custom solutions. Just as I’ve seen projects fail for trying to do too much, I’ve also seen projects fail by trying to do too little. I’m all for the “incremental benefit” mindset, attempting to extract as much short term value from my product without sacrificing long term goals, but sometimes this behavior can cannibalize a design.

In my career, I’ve attempted to walk this tightrope by doing 2 things: (1) make absolutely sure I understand the problem statement and expected business value and (2) do the necessary research.

For number 1, if the team is too bogged down in technical detail from the outset, it tends to miss the forest for the trees. Always keep in the back of your mind, “what am I really trying to accomplish?”

As for number 2, this one is a bit more complicated. My research usually starts on Google Scholar, combing through academic publications or blog posts to see how others have solved my problem. If that proves fruitless, I tend to look for similar problems (maybe in a different domain) until I find a promising lead. At that point, look for out-of-the-box solutions, determine if they meet each of my requirements, and move forward with implementation, if so. If not, something more custom must be built.

There are a ton of great machine learning frameworks and libraries out there, with more coming every day, simply begging to be used to meet your goals.

3. Not appropriately determining risk.

In the excitement of developing a cool new solution, many times we forget that these models are inherently risky. When people say “we don’t really understand how these models work”, that’s true to an extent. Explainable AI is an awesome growing field, dedicated to answering the exact question: “why is this model doing what it’s doing?”

Until the day comes where we can definitively explain why everything works exactly the way that it does, we’ll have to take the necessary precautions into consideration.

Understanding features and correlations amongst them.

Typically, we don’t want our models decisioning based on factors like race, gender, income levels, etc. so we don’t include them as inputs to our model. Everything should be fine, right? Not necessarily. We have to make absolutely sure that these factors aren’t bleeding in to other features that we are using. For instance, zip code could be a fairly strong indicator of demographics that live in a particular area. There must be a large effort invested in data exploration at the beginning of every project.

Will you allow your model to evolve in production?

When I say “machine learning” to someone, they usually assume that means that the model is changing in real-time as people are interacting with it. While there are models that do that (an article for another day), there are many that do not, and for good reason. Without the requisite checks and monitoring, there isn’t much stopping your model from spiraling out of control if the input data drastically starts changing.

Say you have a stock trader dynamically updating based on market trends. In a normal market, it might work excellently, but if something unpredictable happens (as they tend to, and usually at the worst possible time), the model may overcompensate to adjust to this new environment, throwing off the entire strategy it was trained to follow.

How often will you re-train or update the model?

There’s no one “right answer” to this question. It really depends on the problem and the modeling technique that you’re using, but it’s an important thing to figure out early in the process. There should be a standardized update approach and strategy for one simple reason: how do you know if your model is improving or getting worse?

Let’s say I have a 75% accurate model currently in production. How did I determine that 75%? Usually, I will use part of my historical data (maybe 20% or so) as a validation set.

Now let’s say I make an update a month later, and wow! My new accuracy is 85%! Happy with my results, I push the update to my platform, and all of sudden I’m seeing a major drop in results and my customers are complaining left and right. What happened?

If I don’t maintain my validation set (the original data I tested for accuracy), I’m not comparing apples to apples. I can’t definitively say that my update is performing better than my original model, which can cause major headaches down the road.

4. Not needing machine learning in the first place.

Okay, I cheated a little bit, but this is easily the biggest takeaway from this article. While I think ML is one of the coolest fields in computer science today, people tend to lose sight of the fact that it is a tool in the belt, not the belt itself.

You wouldn’t use a jackhammer to put in a nail, so don’t use machine learning to do something you can do with a basic Python script. There is going to be a massive temptation to use the cutting edge technology, and I understand that, but without the proper care and expertise, you could be setting yourself up for unnecessary failure.

I’ve seen so many instances where people will start their product brainstorming with phrases like “how can we use a chatbot?” or “what do you think we could use facial recognition for?” I haven’t seen many instances, on the other hand, where these projects cross the finish line.

Our goal at AiQ is to help clients understand and navigate these common pitfalls. If you’re interested in consulting, engineering, or just networking, connect with me on LinkedIn or reach out to us at contact@goaiq.com. Check out our website and thanks for reading!

This article was originally published on Medium and re-published to TOPBOTS with permission from the author.

Enjoy this article? Sign up for more updates on applied ML.

We’ll let you know when we release more technical education.