Vector Databases

Databases and Generative.

How vector databases can revolutionize our relationship with generative AI

Rick Hao, Speedinvest  @hao_rick

April 30, 2023 8:20 AM

Person's face with swirling dots and lines. Vector database and AI concept.

Generative AI has received a lot of attention already this year in the tech world and beyond. Whether it’s ChatGPT’s prose or Stable Diffusion’s art, 2022 provided an insight into the potential for AI to disrupt creative industries.

Want must read news straight to your inbox?

But behind the headlines, 2022 brought an even more important development in AI: the rise of the vector database.

While their impacts are less immediately obvious, the adoption of vector databases could completely upend the way we interact with our devices, along with dramatically improving our productivity in a vast range of administrative and clerical tasks.

Ultimately, vector databases will be essential infrastructure in bringing about the societal and economic changes promised by AI.

EVENT

Transform 2023

Join us in San Francisco on July 11-12, where top executives will share how they have integrated and optimized AI investments for success and avoided common pitfalls.

But what is a vector database? To understand that, we have to make sense of the underlying problem it addresses: unstructured data.

The database dilemma

Databases are one of the software industry’s longest-lasting and most resilient verticals. The total spend on databases and database management solutions doubled from $38.6B in 2017 to $80B in 2021. And since 2020, databases have only further entrenched their position as one of the most rapidly growing software categories, owing to further digitization following mass shifts to remote working.

However, the modern database is still constrained by a problem that has persisted for decades: the problem of unstructured data. This is the up to 80% of data stored globally that has not been formatted, tagged or structured in a way that allows it to be rapidly searched or recalled. 

For a simple analogy of structured vs. unstructured data, think of a spreadsheet with multiple columns per row. In this case, a row of “structured data” has all the relevant columns filled in, whereas a row of “unstructured data” does not. In the case of the unstructured entry, it may be that the data has been automatically imported into the first column of the row; someone now needs to break up that cell and populate data into relevant columns.

Why is unstructured data a problem? In short, it makes it harder to sort, search, review and use information in a database. However, our understanding of unstructured data is relative to how data is usually structured. ... '