Semantic search and RAG¶
This section started with some slides and background on embeddings.
One of the most popular applications of LLMs is to build “ask questions of my own documents” systems.
You do not need to fine-tune a model for this. Instead, you can use a pattern called retrieval-augmented generation (RAG).
The key trick to RAG is simple: try to figure out the most relevant documents for the user’s question and stuff as many of them as possible into the prompt.
Long context models make this even more effective. “Reasoning” models may actually be less effective here.
Semantic search¶
If somebody asks “can my pup eat brassicas”, how can we ensure we pull in documents that talk about dogs eating sprouts?
One solution is to build semantic search on top of vector embeddings.
I wrote about those in Embeddings: What they are and why they matter.
Generating embeddings with LLM¶
LLM includes a suite of tools for working with embeddings, plus various plugins that add new embedding models.
We’ll start with the OpenAI hosted embedding model.
llm embed -m text-embedding-3-small-512 -c "can my pup eat brassicas?"
This returns a 256 long vector of floats.
More useful is if we store some information first. Let’s embed all the PEPs that start with a 3:
git clone https://github.com/python/peps
cd peps
llm embed-multi peps \
-m text-embedding-3-small-512 \
--files peps 'pep-3*.rst' \
-d peps.db \
--store
There’s a lot going on there. We’re using the 512 long text-embedding-3-small-512 model, saving embeddings to a peps.db SQLite database in a peps collection, scanning for peps/pep-3*.rst files. and storing the full documents along with their vectors.
I tried this… and it failed, because the documents were too long. Let’s create truncated documents first, like this:
llm install llm-cmd
llm cmd create a new folder peps-truncated which has every .rst file from peps/ in it but truncated to first 8000 characters
Which ran
mkdir -p peps-truncated && find peps/ -name '*.rst' -exec sh -c 'head -c 8000 "$1" > "peps-truncated/$(basename "$1")"' _ {} \;
Then I ran this:
llm embed-multi peps \
-m text-embedding-3-small-512 \
--files peps-truncated 'pep-3*.rst' \
-d peps.db \
--store
Confirmed with:
llm collections -d peps.db
And now we can search that collection for items similar to a term using:
llm similar -c 'string concatenation' -d peps.db peps | jq
Here’s a SQLite database with ALL of the PEPs, so that our workshop doesn’t burn through my API credits with everyone embedding the same data!
Answering questions against those PEPs¶
This time we’ll build a bash script:
llm '
Build me a bash script like this:
./pep-qa.sh "What do string templates look like?"
It should first run:
llm similar -c $question -d peps.db peps
Then it should pipe the output from that to:
llm -s "Answer the question: $question" -m gpt-4.1-mini
That last command should run so the output is visible as it runs.
' -x > pep-qa.sh
chmod +x pep-qa.sh
./pep-qa.sh "What do string templates look like?"
I got this.
(Running code that an LLM has generated without first reviewing it generally a terrible idea!)
If you want to port the above to Python you should consult the Working with collections section of LLM’s Python API documentation.
RAG does not have to use semantic search!¶
Many people associate RAG with embedding-based semantic search, but it’s not actually a requirement for the pattern.
A lot of the most successful RAG systems out there use traditional keyword search instead. Models are very good at taking a user’s question and turning it into one or more search queries.
Don’t get hung up on embeddings!
RAG is dead?¶
Every time a new long-context model comes out, someone will declare the death of RAG.
I think classic RAG is dead for a different reason: it turns out arming an LLM with search tools is a much better way to achieve the same goal.
Which brings us to our next topic: Tool usage.