Semantic search uses embeddings to match on meaning rather than keywords, so users find relevant content even when words do not overlap. Learn how it works and where to apply it.
More about Semantic Search
Semantic search is a retrieval technique that matches queries to documents by meaning rather than by exact words. Instead of looking for "pricing" in a corpus, it finds the page about costs, plans, and subscriptions because the underlying concepts are close, even if no keyword overlaps.
Semantic search became practical at scale with the rise of transformer-based embedding models, which convert text into high-dimensional vectors that encode semantic similarity. Combined with a vector database, it can search millions of documents in milliseconds and produce far better results than keyword search on natural-language queries.
How Semantic Search Works
The pipeline has three simple stages:
- Embedding: every document, or chunk of a document, is passed through an embedding model that produces a dense vector.
- Indexing: those vectors are stored in a vector database along with their source text and metadata.
- Querying: the user's query is embedded the same way, and the database returns the documents whose vectors are closest to the query vector, usually by cosine similarity.
The clever part is the embedding model. Modern models like OpenAI's text-embedding-3, Cohere Embed v3, or open-source options like BGE and E5 are trained on massive corpora so that semantically similar text lands near each other in vector space. "How do I cancel?" and "end my subscription" end up close, while "cancel my flight" lands elsewhere.
Semantic Search vs. Keyword Search
Both approaches have a place:
- Keyword search (often powered by BM25 or TF-IDF) excels at exact matches: product SKUs, error codes, names, and quoted phrases.
- Semantic search excels at natural-language queries where users paraphrase or use synonyms.
Teams often run both side by side and blend results, a pattern called hybrid search. The keyword results catch precise lookups and the semantic results catch meaning-based queries. Hybrid search is the current best practice for general-purpose retrieval in AI chatbots.
Why Semantic Search Matters for Chatbots
An AI chatbot is only as good as the passages it retrieves. Ask a customer support bot "how do I get a refund?" and the system has to find the refund policy even if the doc calls it "returns" or "money back guarantee". Keyword search misses that. Semantic search does not.
Semantic search powers:
- Retrieval augmented generation: retrieving the right chunks to ground the large language model on.
- FAQ matching: surfacing the closest existing answer rather than generating a new one.
- Routing and handoff: classifying incoming messages and picking the right downstream flow.
- Conversation recall: finding relevant prior messages from chat history when the context window is not big enough to hold them all.
SiteSpeak indexes every customer's site with a modern embedding model and a hosted vector store, so every user message triggers a semantic search across the full knowledge base in real time. The retrieved passages are then handed to the LLM, which writes the final answer grounded in the actual content.
Improving Semantic Search Quality
Retrieval quality is not just about the embedding model. Small changes produce large gains:
- Chunk size matters: chunks that are too long dilute the signal; too short lose context. 200 to 500 tokens per chunk is a common sweet spot.
- Metadata filtering: restrict searches by language, product area, or customer to cut down noise.
- Query rewriting: use an LLM to expand or clarify the user's question before embedding.
- Reranking: run the top-K results through a cross-encoder that scores each query-document pair together, which typically lifts precision 10 to 20%.
- Feedback loops: log what users thumb-up or thumb-down, then tune chunking, retrieval count, or reranker.
Limitations
Semantic search is strong but not a silver bullet:
- Acronyms and jargon: niche terms may not embed well unless the model has seen them during training.
- Exact matches: product IDs, phone numbers, and SKUs are better served by keyword or hybrid search.
- Noise vs. precision: returning 20 close results does not help if only 2 are actually relevant. Rerankers fix this.
- Embedding drift: if you swap embedding models, you have to reindex everything.
The right goal is not "use semantic search everywhere" but "use the right tool for each part of the query space".
