In this excellent post ‘What If You Can’t Afford AI?‘, Christopher C Penn explains how AI is only suited to very large problems and actually most people deal with human-size problems for which AI may not be very well suited.
“AI is good at three things: processing data faster (and thus being able to handle a lot of it), processing data more accurately, and processing data in routine ways.”
“But that presumes we have enough data to do all that labeling and processing. AI fails when we don’t have enough data. And therein lies the distinguishing factor, the real answer to the question. You need AI when you have machine-sized problems. You can use human solutions when you have human-sized problems.”
The point Christopher C Penn makes is that also a proper sampling of a small representative sample will also give quickly a proper estimate of trends and we don’t necessarily need a huge dataset and a trained AI to provide us guidance. In summary: be clever and don’t think AI will provide much more accurate responses than a clever sampling. “Find a way to reduce the data down to a human-sized problem and solve it with humans until you have enough resources – money, time, people – to work with the full-size dataset. Sampling data is a time-honored method to make big data smaller, and doesn’t require anything more sophisticated than a semester’s worth of statistics classes in university (assuming you did well in the class, of course). Make the data and the problem fit the resources you have to solve it as best as you can.”
Even if you are using AI, it is a good idea to check the robustness of the algorithm by some sampling anyway to check the main trends are adequately captured by the process
It is already possible to get good trends through proper data sampling and reduce problems to human-level problems before embarking on a complex and expensive AI solution. Don’t underestimate the power of this approach!
