‘Octomize’ Your ML Code Posted on : Mar 31 - 2021

If you’re spending months hand-tuning your machine learning model to run well on a particular type of processor, you might be interested in a startup called OctoML, which recently raised $28 million to bring its innovative “Octomizer” to market.

Octomizer is the commercial version of Apache TVM, an open source compiler that was created in Professor Luiz Ceze’s research project in the Computer Science Department at the University of Washington. Datanami recently caught up with the professor–who is also the CEO of OctoML–to learn about the state of machine learning model compilation in a rapidly changing hardware world.

According to Ceze, there is big gap in the MLOps workflow between the completion of the machine learning model by the data scientist or machine learning engineer, and deployment of that model into the real world.

Quite often, the services of a software engineer are required to convert the ML model, which is often written in Python using one of the popular frameworks like TensorFlow or PyTorch, into highly optimized C or C++ that can run on a particular processor. However, the process of getting the code to run optimally is not easy, Ceze says.

“There’s really billions of ways in which you can compile a machine learning model into a specific hardware target. Picking the fastest one is a search process that today is done by human intuition,” he says.

“The way you lay out the data structures in memory matters a lot. And which instructions are you going to pick? Are you going to pick vector instruction? Are you going to run this on a CPU or a GPU?” he continues. “All of these choices lead to an exponential blowup of what are the ways in which we can run. Picking the right one is really hard. It’s done by hand tuning most of the time.” View More