An AI Engineer Walks Into A Data Shop... Posted on : Nov 09 - 2020

An AI-focused neural network software engineer walks into a data shop says hello to the shopkeeper. “I’ll have two data preparation functions, one testing and debugging toolset, a couple of application log tracking systems and a bag of potatoes,” asks the engineer.

Okay it’s not a great joke, there’s no punchline and the potatoes part is definitely just a ruse, but the way we might build the Artificial Intelligence (AI) functions of tomorrow has a kind of composabe, package-able feel. If it’s not quite off-the-shelf AI, then its composable AI that brings together some of the core functions that smart systems use regularly. It’s still down to our neural network engineer to know the recipe and peel the spuds, but we can start to shop for many of the individual components needed now.

Manual AI development is so last year

It might sound almost counter-intuitive to say out loud, but as AI is getting smarter at AI. But this is good, because AI is also getting far more complex. These are data science software systems with extremely complicated convoluted structures that need to juggle a myriad selection of algorithms, data manipulation strategies and ‘data pipeline’ steps (such as Extract, Transform & Load ETL tasks) alongside other feature selection, training, testing, deployment and monitoring functions. We have, perhaps unsurprisingly, gone beyond the point where we can handle all of those things manually

But (and this is more good news) many of those individual component tasks, functions and jobs are now well understood and documented.

Co-founder and CTO of Tel Aviv based data science platform company Iguazio is Yaron Haviv. Explaining that his firm is currently driving its technology towards delivering real-time AI functions, Haviv says that what we need now is a way to create, test, debug and publish popularized functions. Then, AI engineers can compose Machine Learning (ML) pipelines from those functions. He suggests that we can even store a simple or repetitive pipeline as a reusable component and ultimately build a bigger pipeline from it.

“One of the main challenges in Machine Learning deployments today is the transfer time it takes to get from prototyping and research to production. The various functions on an ML team (everyone from data scientists, data engineers, DevOps, application developers) often work in silos with minimal collaboration. Moving to a composable ML/AI architecture is an inevitable step as the complexity of ML pipelines increases. We must move to an architecture that enables collaboration and reuse: It’s time to transition to a composable ML architecture,” argues Haviv. View More