Looking Beyond LLM

Looking beyond, fascinating

Emerging Architectures for LLM Applications

by Matt Bornstein and Rajko Radovanovic

AI, machine & deep learning  enterprise & SaaS  AI  Generative AI  machine learning

TABLE OF CONTENTS

The stack

Design pattern: In-context learning

Data preprocessing / embedding

Prompt construction/ retrieval

Prompt execution / inference

What about agents?

Looking ahead

Explore more: AI + a16z

Large language models are a powerful new primitive for building software. But since they are so new—and behave so differently from normal computing resources—it’s not always obvious how to use them.

In this post, we’re sharing a reference architecture for the emerging LLM app stack. It shows the most common systems, tools, and design patterns we’ve seen used by AI startups and sophisticated tech companies. This stack is still very early and may change substantially as the underlying technology advances, but we hope it will be a useful reference for developers working with LLMs now.

This work is based on conversations with AI startup founders and engineers. We relied especially on input from: Ted Benson, Harrison Chase, Ben Firshman, Ali Ghodsi, Raza Habib, Andrej Karpathy, Greg Kogan, Jerry Liu, Moin Nadeem, Diego Oppenheimer, Shreya Rajpal, Ion Stoica, Dennis Xu, Matei Zaharia, and Jared Zoneraich. Thank you for your help!

The stack

Here’s our current view of the LLM app stack (click to enlarge): ... '