Are Neural Nets the Next Big Thing in Search? Posted on : Feb 04 - 2022

Many, if not most, search engines in use today are based on keywords, in which the search engine attempts to find the best match for a word or set of words used as input. It’s a tried-and-true method that has been deployed millions of times over decades of use. But new search approaches based on deep learning, including vector search and neural search, have emerged recently, and early backers say they have the potential to shake up the search market.

Vector search uses a fundamentally different approach to finding the best fit between a term provided as input to the engine and the result that is presented to the user. Instead of powering the search by doing a direct one-to-one matching of keywords, in vector search, the engine attempts to match the input term to a vector, which is an array of features generated from objects in the catalog.

This approach relies on a machine learning (or deep learning) model to derive those features from the objects in the catalog. Once the ML or DL model converts the features of the objects into a two-dimensional vector with tens to hundreds of dimensions, the user can execute the search against that vector to generate the search results.

Vector search can potentially provide better search results because it’s providing richer context and more nuance than a basic keyword search can provide. In addition to generating vectors from groups of words, they can be generated from images, video, and audio too.

One early believer in vector search is Bob Wiederhold, the former Couchbase CEO who is now an independent advisor and investor in tech startups. Recently Wiederhold started working with a startup named Pinecone, which has developed a vector database that is being used to power search engines, among other use cases.

“Elastic is dominating the third-generation, keyword search-type approach. What the next-generation players are leaning towards is this more of this machine learning-driven approach,” Wiederhold says. “Basically, what Pinecone does is we use machine learning technology to take…that text-based input and convert that into vectors.” View More