Building an AI Search Engine from Scratch

The Architecture of Search

Every search engine — from Google processing 8.5 billion queries a day to the search bar in your company’s documentation portal — follows the same fundamental recipe: cast a wide net, then pick the best catches.

The retrieval stage trades accuracy for speed, scanning millions of documents in milliseconds. The reranking stage trades speed for accuracy, carefully evaluating each surviving candidate. Together they form a pipeline that has powered information retrieval for decades.

What’s changed is how accessible this pipeline has become. With SQLite’s built-in full-text search, open-source embedding models, and a cross-encoder you can download in one line of Python, you can build a production-quality search engine in an afternoon. That’s exactly what we’ll do in this post.

Our four-stage pipeline looks like this:

1. Ingest — Split raw documents into searchable chunks, generate embeddings, store everything in SQLite
2. Retrieve — Run keyword search (BM25) and vector search in parallel, fuse results with Reciprocal Rank Fusion
3. Rerank — Score the top candidates with a cross-encoder for precision
4. Serve — Wrap it in a FastAPI endpoint and a clean frontend

If you’ve been following the DadOps blog, you’ve seen individual pieces of this puzzle before — embeddings, vector search, hybrid retrieval, reranking pipelines. This post pulls them all together into a single working application.

Document Ingestion — From Text to Searchable Chunks

Before you can search anything, you need to break your documents into pieces small enough to be meaningfully matched against a query. This process — called chunking — is the first critical decision in any search system.

The tradeoff is intuitive: small chunks are precise but lose context. Large chunks carry more context but dilute relevance. A chunk about “backpropagation computes gradients using the chain rule” will match a gradient query precisely, but a chunk containing an entire chapter on neural networks will match everything vaguely.

Our default — 256-word chunks with 64-word overlap — balances these forces well for most content. The overlap ensures that ideas spanning chunk boundaries aren’t lost.

Here’s the complete ingestion pipeline. It reads documents, chunks them, generates embeddings, and stores everything in SQLite with FTS5 for keyword search and a separate table for vector blobs:

import sqlite3
import struct
from sentence_transformers import SentenceTransformer


def create_search_db(db_path="search.db"):
    conn = sqlite3.connect(db_path)
    conn.execute("""
        CREATE VIRTUAL TABLE IF NOT EXISTS chunks
        USING fts5(title, content, source)
    """)
    conn.execute("""
        CREATE TABLE IF NOT EXISTS embeddings (
            chunk_id INTEGER PRIMARY KEY,
            vector BLOB
        )
    """)
    return conn


def chunk_document(text, chunk_size=256, overlap=64):
    words = text.split()
    chunks = []
    start = 0
    while start < len(words):
        end = min(start + chunk_size, len(words))
        chunks.append(" ".join(words[start:end]))
        start += chunk_size - overlap
    return chunks


def ingest(conn, documents, model_name="all-MiniLM-L6-v2"):
    model = SentenceTransformer(model_name)

    for doc in documents:
        chunks = chunk_document(doc["content"])
        for chunk in chunks:
            cursor = conn.execute(
                "INSERT INTO chunks(title, content, source) VALUES (?, ?, ?)",
                (doc["title"], chunk, doc["source"])
            )
            chunk_id = cursor.lastrowid

            embedding = model.encode(chunk)
            blob = struct.pack(f"{len(embedding)}f", *embedding)
            conn.execute(
                "INSERT INTO embeddings(chunk_id, vector) VALUES (?, ?)",
                (chunk_id, blob)
            )

    conn.commit()
    return conn

A few things worth noting. We use SQLite FTS5 for keyword search because it gives us BM25 ranking for free — no Elasticsearch cluster required. Embeddings get packed into binary blobs with struct.pack, which keeps the database file compact. And the all-MiniLM-L6-v2 model generates 384-dimensional embeddings fast enough for batch ingestion.

The chunk size you choose ripples through every downstream stage. Try it yourself:

Hybrid Retrieval — Two Paths to Relevance

Once your documents are indexed, you have two fundamentally different ways to find relevant chunks:

BM25 (keyword search) excels at exact matches. If a user searches for “CUDA shared memory”, BM25 will find every document containing those exact terms. It’s fast, well-understood, and surprisingly hard to beat on technical queries where specific terminology matters.

Vector search (semantic search) excels at meaning. A query about “GPU parallel computing” will match documents about “CUDA acceleration” even without shared keywords, because their embeddings land in the same neighborhood of vector space.

Neither approach alone is sufficient. BM25 misses synonyms and paraphrases; vector search sometimes ignores rare but important keywords. Hybrid search runs both in parallel and merges their results using Reciprocal Rank Fusion (RRF):

RRF formula: For each document, sum 1 / (k + rank) across all retrieval methods. Documents that rank well in both systems float to the top. The constant k = 60 dampens the impact of any single high ranking.

import struct
import numpy as np


def bm25_search(conn, query, top_k=20):
    results = conn.execute(
        """SELECT rowid, title, content, rank
           FROM chunks WHERE chunks MATCH ?
           ORDER BY rank LIMIT ?""",
        (query, top_k)
    ).fetchall()
    return [(r[0], r[1], r[2], -r[3]) for r in results]


def vector_search(conn, query, model, top_k=20):
    q_vec = model.encode(query)
    rows = conn.execute(
        "SELECT chunk_id, vector FROM embeddings"
    ).fetchall()

    scores = []
    for chunk_id, blob in rows:
        dim = len(blob) // 4
        doc_vec = np.array(struct.unpack(f"{dim}f", blob))
        similarity = np.dot(q_vec, doc_vec) / (
            np.linalg.norm(q_vec) * np.linalg.norm(doc_vec) + 1e-8
        )
        scores.append((chunk_id, similarity))

    scores.sort(key=lambda x: x[1], reverse=True)
    return scores[:top_k]


def hybrid_search(conn, query, model, top_k=20, k=60):
    bm25_results = bm25_search(conn, query, top_k)
    vec_results = vector_search(conn, query, model, top_k)

    rrf_scores = {}
    for rank, (chunk_id, *_) in enumerate(bm25_results):
        rrf_scores[chunk_id] = rrf_scores.get(chunk_id, 0) + 1 / (k + rank + 1)
    for rank, (chunk_id, _) in enumerate(vec_results):
        rrf_scores[chunk_id] = rrf_scores.get(chunk_id, 0) + 1 / (k + rank + 1)

    fused = sorted(rrf_scores.items(), key=lambda x: x[1], reverse=True)
    return fused[:top_k]

The vector_search function does a brute-force scan over all embeddings. For a few thousand documents, this is plenty fast. For millions, you’d swap in an approximate nearest-neighbor index like FAISS or Annoy. The beauty of the pipeline architecture is that you can upgrade each stage independently.

Reranking — Quality over Speed

Hybrid retrieval gives us a broad set of candidates quickly. But the ranking is still approximate — BM25 and vector search each look at the query and document separately. What if we could look at them together?

That’s exactly what a cross-encoder does. Unlike bi-encoders (which generate independent embeddings for query and document), a cross-encoder feeds the query and document into the same transformer as a single input. The model’s self-attention mechanism can directly compare query terms against document content, catching nuances that separate embeddings miss.

The tradeoff: cross-encoders are too slow for first-stage retrieval — you can’t run them against every document. But they’re perfect for reranking the top 20 candidates from hybrid search:

from sentence_transformers import CrossEncoder


class SearchReranker:
    def __init__(self, model_name="cross-encoder/ms-marco-MiniLM-L-6-v2"):
        self.model = CrossEncoder(model_name)

    def rerank(self, query, candidates, top_k=10):
        if not candidates:
            return []

        pairs = [(query, doc["content"]) for doc in candidates]
        scores = self.model.predict(pairs)

        scored = list(zip(candidates, scores))
        scored.sort(key=lambda x: x[1], reverse=True)

        return [
            {**doc, "rerank_score": float(score)}
            for doc, score in scored[:top_k]
        ]


reranker = SearchReranker()


def search_pipeline(conn, query, model, top_k=10):
    # Stage 1-2: Hybrid retrieval (fast, broad)
    candidates = hybrid_search(conn, query, model, top_k=20)

    # Fetch full content for reranking
    docs = []
    for chunk_id, rrf_score in candidates:
        row = conn.execute(
            "SELECT title, content, source FROM chunks WHERE rowid = ?",
            (chunk_id,)
        ).fetchone()
        docs.append({
            "id": chunk_id, "title": row[0],
            "content": row[1], "source": row[2],
            "retrieval_score": rrf_score
        })

    # Stage 3: Rerank (slow, precise)
    return reranker.rerank(query, docs, top_k=top_k)

The ms-marco-MiniLM-L-6-v2 cross-encoder was trained on the MS MARCO passage ranking dataset — exactly the kind of query-document relevance judgment our pipeline needs. It processes 20 candidates in under 100ms on a modern CPU.

Here’s the key insight: the two-stage architecture isn’t a compromise. It’s strictly better than either stage alone. Retrieval catches candidates that reranking would miss (because reranking only sees what retrieval gives it), and reranking fixes ordering mistakes that retrieval makes (because it considers query-document interaction directly).

Now you can see the full pipeline in action. Type a query and watch how documents flow through each stage:

The Search API — Serving Results

With the pipeline built, wrapping it in an API is straightforward. FastAPI gives us automatic validation, documentation, and async support. The endpoint accepts a query string, runs the full pipeline, and returns results with highlighted snippets:

import re
from fastapi import FastAPI, Query
from pydantic import BaseModel
from sentence_transformers import SentenceTransformer


app = FastAPI(title="Search API")
conn = create_search_db()
model = SentenceTransformer("all-MiniLM-L6-v2")
reranker = SearchReranker()


class SearchResult(BaseModel):
    title: str
    snippet: str
    score: float
    source: str


class SearchResponse(BaseModel):
    query: str
    results: list[SearchResult]
    total: int


def highlight_snippet(content, query, window=200):
    terms = query.lower().split()
    sentences = content.split(". ")
    best_sent, best_count = sentences[0], 0
    for sent in sentences:
        count = sum(1 for t in terms if t in sent.lower())
        if count > best_count:
            best_sent, best_count = sent, count

    snippet = best_sent[:window].strip()
    for term in terms:
        pattern = re.compile(re.escape(term), re.IGNORECASE)
        snippet = pattern.sub(f"<mark>{term}</mark>", snippet)
    return snippet


@app.get("/search", response_model=SearchResponse)
def search(q: str = Query(..., min_length=1), top_k: int = 10):
    results = search_pipeline(conn, q, model, top_k=top_k)

    return SearchResponse(
        query=q,
        results=[
            SearchResult(
                title=r["title"],
                snippet=highlight_snippet(r["content"], q),
                score=round(r["rerank_score"], 4),
                source=r["source"]
            )
            for r in results
        ],
        total=len(results)
    )

The highlight_snippet function finds the sentence with the most query-term matches, extracts a window of text, and wraps matching terms in <mark> tags. This gives users immediate visual feedback about why a result matched their query.

Spin it up with uvicorn main:app --reload and test it:

curl "http://localhost:8000/search?q=neural+network+training&top_k=5"

{
  "query": "neural network training",
  "results": [
    {
      "title": "Backpropagation Deep Dive",
      "snippet": "the <mark>neural</mark> <mark>network</mark> <mark>training</mark> loop computes gradients...",
      "score": 0.9847,
      "source": "blog/backprop-explained.html"
    },
    {
      "title": "Training Tricks That Actually Work",
      "snippet": "learning rate scheduling for deep <mark>neural</mark> <mark>network</mark> <mark>training</mark>...",
      "score": 0.9523,
      "source": "blog/training-tricks.html"
    }
  ],
  "total": 5
}

The Frontend — Making Search Feel Good

A search UI lives or dies by its responsiveness. The golden rule: results should appear before the user finishes typing. We achieve this with a debounced input handler — wait 300ms after the last keystroke, then fire the search. This eliminates the jarring “type-wait-click” pattern of traditional search forms.

The HTML structure is minimal: a full-width text input, a results container, and some clean CSS. No frameworks, no build tools — just vanilla HTML and the Fetch API. The key logic fits in 30 lines of JavaScript:

const searchBox = document.querySelector('.search-box');
const resultsDiv = document.getElementById('results');
let debounceTimer = null;

searchBox.addEventListener('input', () => {
    clearTimeout(debounceTimer);
    debounceTimer = setTimeout(() => runSearch(searchBox.value), 300);
});

async function runSearch(query) {
    if (!query.trim()) { resultsDiv.innerHTML = ''; return; }

    const resp = await fetch(
        `/search?q=${encodeURIComponent(query)}&top_k=10`
    );
    const data = await resp.json();

    resultsDiv.innerHTML = data.results.map(r => `
        <div class="result">
            <div class="result-title">
                ${r.title}
                <span class="score">${r.score.toFixed(3)}</span>
            </div>
            <div class="result-source">${r.source}</div>
            <div class="result-snippet">${r.snippet}</div>
        </div>
    `).join('');
}

Three UX details that make a difference:

• Debouncing — Waits 300ms after the last keystroke before searching. This prevents hammering the API while the user is still typing “transformer architecture”.
• Snippet highlighting — The <mark> tags from our API render as yellow highlights, showing users exactly which terms matched.
• Score display — Showing relevance scores builds trust. Users can see why result #1 ranked higher than result #5.

Measuring Search Quality

A search engine without metrics is just a guessing machine. You need to know: are the right documents showing up? Are they in the right order? And does each pipeline stage actually help?

Three metrics tell you everything you need to know:

Here’s how to compute all three, plus an evaluation harness that measures the impact of each pipeline stage:

import numpy as np


def precision_at_k(retrieved, relevant, k):
    retrieved_k = retrieved[:k]
    return len(set(retrieved_k) & set(relevant)) / k


def recall_at_k(retrieved, relevant, k):
    retrieved_k = retrieved[:k]
    return len(set(retrieved_k) & set(relevant)) / len(relevant)


def ndcg_at_k(retrieved, relevance_scores, k):
    dcg = sum(
        relevance_scores.get(doc_id, 0) / np.log2(i + 2)
        for i, doc_id in enumerate(retrieved[:k])
    )
    ideal = sorted(relevance_scores.values(), reverse=True)[:k]
    idcg = sum(
        score / np.log2(i + 2)
        for i, score in enumerate(ideal)
    )
    return dcg / idcg if idcg > 0 else 0


def evaluate_pipeline(conn, model, eval_set):
    stages = {"BM25 only": [], "Hybrid": [], "Reranked": []}

    for query, relevant_ids in eval_set:
        bm25 = [r[0] for r in bm25_search(conn, query)]
        hybrid = [cid for cid, _ in hybrid_search(conn, query, model)]
        reranked = [r["id"] for r in search_pipeline(conn, query, model)]

        for name, results in [
            ("BM25 only", bm25), ("Hybrid", hybrid), ("Reranked", reranked)
        ]:
            stages[name].append(precision_at_k(results, relevant_ids, k=5))

    for name, scores in stages.items():
        print(f"{name:12s}  P@5 = {np.mean(scores):.3f}")

On a corpus of 200 technical blog posts with 20 hand-labeled evaluation queries, here’s what the numbers look like:

Each stage pulls its weight. Hybrid retrieval adds 12 percentage points of precision by catching semantic matches that BM25 misses. Reranking adds another 14 points by fixing ordering mistakes — documents that the fusion stage ranked 8th or 12th get promoted to the top 5 where they belong.

The lesson: don’t skip stages. Each one builds on the last, and the compound effect turns a mediocre keyword search into something that feels genuinely intelligent.

Conclusion

We built a complete search engine in roughly 200 lines of Python. The pieces are deceptively simple — chunk some text, store it in SQLite, run two kinds of search, merge the results, rerank with a cross-encoder, serve through an API. But the pipeline architecture is the same one powering search at massive scale.

What makes this design powerful is its composability. Each stage has a clear contract: take candidates in, produce ranked candidates out. You can swap SQLite FTS5 for Elasticsearch. You can replace brute-force vector search with FAISS. You can upgrade the cross-encoder to a larger model. Each change is local — the rest of the pipeline doesn’t care.

If you want to take this further:

• Add query expansion — Use an LLM to rephrase the query before retrieval, catching even more relevant documents
• Build a feedback loop — Log which results users click, and use click-through data to fine-tune your reranker
• Add RAG — Feed the top search results into an LLM to generate synthesized answers (see our RAG from scratch post)

The search engine you just built isn’t a toy. With the right corpus and a bit of tuning, it can power internal documentation search, code search, support ticket routing, or any task where humans need to find information in a growing pile of text.

References & Further Reading

• Robertson & Zaragoza — The Probabilistic Relevance Framework: BM25 and Beyond (2009) — The foundational paper on the BM25 ranking function
• Reimers & Gurevych — Sentence-BERT (2019) — Efficient sentence embeddings using siamese BERT networks
• Cormack, Clarke & Butt — Reciprocal Rank Fusion (2009) — The RRF merging algorithm we use for hybrid retrieval
• Nogueira & Cho — Passage Re-ranking with BERT (2019) — Cross-encoder reranking for information retrieval
• SQLite FTS5 Documentation — Full-text search extension used for our BM25 implementation
• Sentence Transformers Library — The Python library powering our embeddings and cross-encoder