Getting practical, real-world results with ML and AI involves more than getting data, doing calculations, and building models. You can do everything else right, but if you don’t get this last step right, you’ll join the rapidly growing ranks of people who may have tried hard, but ended up accomplishing little in real-world terms.
The first part of this series laid out the issues and concentrated on the indispensable foundation of success, the data. The second part of this series dove into the analytic methods that can be used to generate value, with some advice about how to sequence the methods used.
In this post, we’ll concentrate on the relationship between the real world and the back office analytical work. What we’ll find is that an integrated, collaborative, closed-loop relationship between measuring, calculating and real world application is the path to success.
Loops, open and closed
Whether you run an operation closed loop or open loop is one of those absolutely key concepts, highly correlated with success, that is rarely discussed. Generation after generation discovers it by itself, or not, nearly always without fanfare. Who talks about the key role played by the invention of the governor in 1788 by James Watt – the invention that made his steam engine practical? In that case, the governor was the newly-minted part of a steam engine that kept the pressure of the steam reasonably constant. With a governor, steam engines no longer blew up, as they regularly did before its use.
It’s important to understand that the reason a governor works is that it’s an integral part of the steam engine. Steam goes into the governor, which then controls the throttle valve of the engine, slowing it down when it’s getting dangerously hot. This is closed-loop.
In more modern terms, running open loop means going on and on down a path without real-world feedback and testing of your work as it is being developed. It’s a little like trying to walk to a goal post at the opposite end of a football field with your eyes closed, using a carefully planned sequence of steps and turns. That’s open-loop, which essentially mean no feedback. But shockingly enough, a huge fraction of highly technical efforts in software and analytics operate in just this way! The people in charge insist they’re experienced, they’ve got a thoroughly vetted plan, and everyone should let them alone to get their work done.
There are many similarities between war-time software and running closed loop for analytics. Driving towards a goal, letting nothing get in the way. Optimizing for speed, not expectations. Leaping to a place that's better than today, and then cycling improvements.
The easiest way to see the difference is thinking about the previous posts in this series. Have you spent lots of time with data, and applied simple calculations to it? If not, you should. Once you have, … you should put your new understanding into practice! It may not be the very best solution that’s possible, but if it’s marginally better than what’s in place today, you should roll it out at least in a limited way and see how the world reacts to it. You’ll learn stuff! You may end up learning there are more variables you need to account for, different ways it needs to be applied, all sorts of things! In other words, don’t sit on the beach by the water for months – wade right in and see what it’s like. That’s when you’ll really start learning.
The World Responds and Changes
The key concept to understanding why running closed loop is so important is that the “world” is an incredibly complex, ever-changing set of actors. When you do something – almost anything – the world changes in response to what you did, if only in a small way. You have to run closed loop to respond appropriately as the world responds to your actions.
Oh, you may say, I’m just the genius in the back room who’s an expert in this or that branch of ML. I’m not acting on the world. I just need the time and support to get my amazing modeling work done.
That may be true. And that’s the problem! The whole point of doing ML/AI/etc. is to change something in the world – and it’s guaranteed that the world will change in response! Accounting for the responsive changes is just as important as whatever it is you first put out there. Even worse, the world constantly changes independent of anything you may do. So the solution you modeled for may not be valid, given the changes that happened.
Think about the carefully planned walk on the football field to the goal posts I described above, and how hard it would be to accomplish with your eyes closed, i.e., with no feedback. Now think about the same situation, except there's an opposing team on the field! You carefully study everything about the opposing team. You know who they are and where they are. Then the play starts and you start to execute your exquisitely planned march to the goal posts. Here's the trouble: opposing team members see what you're doing, and they change their positions! They move! Even worse, they run towards you and try to tackle you. And you are helpless, because you are carefully executing your wonderful plan with your eyes closed, unable to react to the other team's movements. Is that stupid or what? It's not just stupid, it's inconceivably stupid. That's why I spelled it out, because that's exactly how most ML and other analytics efforts are carried out. Open loop. Assuming that the world does not change in response to what you do.
Of course, the world is unlikely to be quite as single-minded and determined as members of an opposing football team. But you'd be surprised! You're making changes in the real world. Whatever you do, there are probably losers. Losers who won't be happy, and will change their behaviors so they become winners again. Or simply fail to act in the predicted ways.
Conclusion
Running closed-loop is absolutely indispensable to achieving success. Put something simple in the real world and then cycle, making it better and better, using increasingly sophisticated techniques. Whatever your final crowning technique is, whether it's ML, AI or something else, success will be yours, and you'll enjoy it all along, without the risk, anxiety and likely failures of the usual highly planned methods.