A practical guide to implementing semantic search for technical content
Have you ever spent hours writing a blog post only to have it disappear into the void? You're not alone. I spend a lot of time making detailed tutorials, explanations, and code examples, yet much of this content is hard to find.
90% of users never venture past the first page of search results, and most scan only the top 3-5 entries before reformulating their query or abandoning the search altogether. For technical blogs, this problem is even more acute - the specialized vocabulary and conceptual relationships between topics make keyword matching particularly ineffective.
Search solutions that rely solely on finding exact words will often have huge misses. For example my post about custom tags for FastHTML may be be highly relevant to someone looking to use web components but web components is a term never mentioned directly in the blog post!
This disconnect creates a frustrating experience for both content creators and consumers. Content creators have to spend time trying to jam in keywords to make things more discoverable, and readers struggle to find the solutions they're looking for.
The question becomes: how can we make technical content discoverable by meaning rather than just keywords?
In this tutorial, you'll learn how to implement a semantic search system using LanceDB. Instead of relying on exact keyword matches, you'll be able to find content based on meaning and conceptual relationships. We will go beyond the hello-world of "vector-search" and do a hybrid vector-search + keyword search approach and then re-rank final results with a cross-encoder. By the end of this tutorial, you'll know what that means and how to implement it.
By the end of this post, you'll have a complete solution that can:
Here's a quick preview of what we'll build:
# A simple semantic search query that finds related content
query = "How do I make web components?"
results = blog_search.search(query, limit=3)
for result in results:
print(f"Post: {result.metadata['title']}")
print(f"Relevance score: {result.score:.2f}")
# Output:
# Post: Basic Testing in Python
# Relevance score: 0.87
#
# Post: Python Tips and Tricks
# Relevance score: 0.72
This isn't just theory - you'll implement this system step-by-step using real blog posts, and I'll show you how to adapt it to your own content. Whether you have a personal tech blog, manage documentation for a larger project, or have anything else you want people to be able to search, this approach will help your valuable content reach the right audience
This is the foundation of a modern retrieval system and I am hard-pressed to thing of an example where you want good semantic search but would not want this as the foundation.
💡 Check out the companion repo for runnable code examples
At the heart of semantic search is the concept of vector embeddings. AI models cannot understand words directly so we have to convert everything to numbers to compare them programatically. Let's take a simple example, maybe we give 3 words vector embeddings
You can see how using just the numbers Cat and Dog are more similar to Animal than they are to Bog. What does each specific number mean? I don't know - they don't map to human language! Similar concepts end up close to each other in this space, even if they use different words. That's what semantic search is.
In a more targetted example, the phrases "how to test Python code" and "writing unit tests for Python functions" might use different words, but their vector embeddings would be very similar because they represent the same concept.
This leads to the question: "How do you pick the numbers for each word, sentence, or article?".
Modern language models like BERT, GPT, and their derivatives can generate these embeddings by processing vast amounts of text and learning the relationships between words and concepts. We can leverage those pre-existing models to generate these embeddings (training a new model to do this is out of the scope of this tutorial).
fh_tutorial = clean_post(read_url('https://isaac.up.railway.app/blog/blog_post?fpath=posts%2FFasthtmlTutorial.ipynb'))
ms_scratch = clean_post(read_url('https://isaac.up.railway.app/blog/blog_post?fpath=posts%2FMeanShiftFromScratch.ipynb'))
py_tips = clean_post(read_url('https://isaac.up.railway.app/blog/blog_post?fpath=posts%2FPython.ipynb'))
bas_test = clean_post(read_url('https://isaac.up.railway.app/blog/blog_post?fpath=posts%2FBasicTesting.ipynb'))We'll prepare our blog posts. For simplicity, we'll extract titles and content from our sample posts:
Credit: This post was inspired by Ben Clavie's excellent work on semantic search implementations. His talk on RAG systems at a conference provided many of the core concepts and techniques explored in this tutorial. I highly recommend checking out his original presentation for additional context and perspectives. You will be glad you did - it is well worth the time.
import pandas as pd
from sentence_transformers import SentenceTransformer
import lancedbposts = [fh_tutorial, ms_scratch, py_tips, bas_test]
blog_data = [{'title': post.split('\n')[0].replace('# ', ''), 'content': post} for post in posts]
df = pd.DataFrame(blog_data)
dfThe next thing we need is some sort of vector embedding for each post. As mentioned before, we can us a pretrained model for this.
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")💡 One place to look for embedding models to use is the MTEB Leaderboard
BEIR. Though be careful because many of the models are overfit to the leaderboard.
df['vector'] = df['content'].apply(lambda x: model.encode(x, normalize_embeddings=True))df.head()We can load all that into our vector database, lancedb.
db = lancedb.connect("./lancedb-tutorial")
table = db.create_table("blog_posts", data=df ,mode="overwrite")Now, we're ready to use LanceDB to search. There are a two main steps for this
1. Create an embedding of your query or question
query = "How do I make web components?" # Get a query
query_embedding = model.encode(query) # Create an embedding of it
query_embedding.shape, query_embedding[:3] # 384 numbers representing the query2. Compare Similarity
Now we can search the table to find most similar embeddings
results = table.search(query_embedding).metric('cosine').to_pandas()
results['_distance']💡 Different models use different distance metrics. This was trained with cosine similarity per the huggingface model card so I matched to that.
3. See Results
Now we can see which post is the most similar to the query based on those embeddings
for _, row in results.iterrows():
print(f"Post: {row['title']}")
print(f"Distance: {row['_distance']:.2f}")assert not 'web component' in fh_tutorial.lower()This basic implementation allows us to search our blog posts semantically. When we run a query like ""How do I make web components?", it will find the revelant post even though web component is not mentioned in the tutorial directly.
However, this approach has a lot of limitations.
Our simple semantic search implementation works, but it has several limitations that make it inadequate for serious technical content:
Full document embeddings lose detail: When we embed entire blog posts, we're compressing thousands of words into a single 384-dimensional vector. This means specific technical details get lost.
Code blocks get mixed with text: Technical blogs contain a mix of explanatory text and code examples. Our current approach treats both the same way, diluting the search quality.
No re-ranking: We're using a simple bi-encoder approach with cosine similarity, but as Ben Clavié explains, this misses the nuance that cross-encoders can provide.
To address the limitations of our initial approach, we need to implement a more sophisticated solution. Following Ben Clavié's recommendations, we'll improve our system by:
Let's start with the chunking strategy, which will help us preserve more context and detail:
There are a plethora of chunking strategies you can test, but lets use the most obvious one. Let's use the chunks defined by the author of the article, and split on markdown headers.
Note: There's lots of discussions on optimal chunk length, and overlap, and different ways to split it. I recommend starting with what makes sense, and then try out different ones.
def chunk_by_markdown_sections(markdown_text, min_length=250):
"""Split markdown text into chunks based on header sections."""
lines = markdown_text.split('\n')
chunks = []
current_chunk = []
current_title = "Introduction"
for line in lines:
if line.startswith('#'): # New header found
# Save previous chunk if it's substantial
if current_chunk and len('\n'.join(current_chunk)) >= min_length:
chunks.append({'title': current_title, 'content': '\n'.join(current_chunk)})
# Start new chunk
current_title = line.lstrip('# ')
current_chunk = [line]
else:
current_chunk.append(line)
# Add the final chunk if it exists and meets length requirement
if current_chunk and len('\n'.join(current_chunk)) >= min_length:
chunks.append({'title': current_title, 'content': '\n'.join(current_chunk)})
return chunkschunk_by_markdown_sections("""# Markdown\n# Chunked\n## Based on markdown\n## Headers for RAG""", min_length=1)Now let's apply this chunking function to our blog posts:
all_chunks = []
for i, row in df.iterrows():
for chunk in chunk_by_markdown_sections(row['content']):
all_chunks.append({
'post_title': row['title'],
'chunk_title': chunk['title'],
'content': chunk['content'],
'vector': model.encode(chunk['content'], normalize_embeddings=True)})
# Create DataFrame with all chunks
chunks_df = pd.DataFrame(all_chunks)The most important thing to learn from this entire guide is that when you do things, you should look at your data. Don't assume things were right. Don't assume your idea made sense. Print it out and look!
for i,r in chunks_df.iterrows():
print(r.content)
print("-----\n")
if i==2: breakNow we can create a new LanceDB table with our chunked content:
chunk_table = db.create_table("blog_chunks", data=chunks_df, mode="overwrite")We can then query to find the most relevant chunks instead of entire documents.
results = chunk_table.search(query_embedding).metric('cosine').to_pandas()
# Print results
for _, row in results.iterrows():
print(f"Distance: {row['_distance']:.2f}")
print(f"Post: {row['post_title']} : {row['chunk_title']}")
print('---')
if _==2: breakThis chunking approach gives us several advantages:
In the next section, we'll enhance our retrieval system by implementing a hybrid search approach that combines vector similarity with keyword matching for more accurate results.
Now that we've improved our system with chunking, let's implement the next key improvements from Ben Clavié's recommendations:
Let's start with implementing keyword search to complement our vector search:
import bm25s
import numpy as np
from sklearn.preprocessing import normalizeBM25 is a powerful keyword-based search algorithm that works by analyzing term frequency and document length. We'll use it to complement our vector search by capturing exact keyword matches that semantic search might miss.
tokenized_chunks = [doc.split() for doc in chunks_df['content']]def hybrid_search(query, top_k=5, vector_weight=0.7):
"""Perform hybrid search using both vector similarity and BM25 keyword matching."""
# Vector search
query_embedding = model.encode(query, normalize_embeddings=True)
vector_results = chunk_table.search(query_embedding).metric('cosine').limit(top_k*2).to_pandas()
vector_results['vector_score'] = 1 - vector_results['_distance']
# Keyword search with BM25s
# Create corpus from our chunks
corpus = chunks_df['content'].tolist()
# Tokenize the corpus and index it
corpus_tokens = bm25s.tokenize(corpus)
retriever = bm25s.BM25(corpus=corpus)
retriever.index(corpus_tokens)
# Tokenize the query and retrieve results
query_tokens = bm25s.tokenize(query)
docs, scores = retriever.retrieve(query_tokens, k=len(corpus)) # Get scores for all documents
# Map BM25 scores to our dataframe indices
bm25_scores = {i: scores[0, idx] for idx, i in enumerate(docs[0])}
vector_results['bm25_score'] = vector_results.index.map(
lambda x: bm25_scores.get(x, 0) if x in bm25_scores else 0)
# Normalize BM25 scores
if vector_results['bm25_score'].max() > 0:
vector_results['bm25_score'] = vector_results['bm25_score'] / vector_results['bm25_score'].max()
# Combine scores with weighting
vector_results['combined_score'] = (
vector_weight * vector_results['vector_score'] +
(1 - vector_weight) * vector_results['bm25_score'])
return vector_results.sort_values('combined_score', ascending=False).head(top_k)query = "How do I make web components?"
results = hybrid_search(query, 6)
for _, row in results.iterrows():
print(f"Post: {row['post_title']}")
print(f"Section: {row['chunk_title']}")
print(f"Relevance: {row['combined_score']:.2f}")
print("---")
if _==2: breakSo far with Vector Search a model takes in a piece of text and creates a representation of that output. This is really fast in practice because all of the documents you want to search against can have these vectors pre-calculated. Their vector representation doesn't change regardless of the user query so we calculated them up front and stored them in LanceDB.
However, cross-encoders make more powerful vector embeddings. It accomplished this by examining each query-document pair in context, which helps it understand nuanced relationships that might be missed by the initial retrieval step. For example, it can better understand when a document answers a question even if it uses different terminology.
The process works in two stages:
This approach gives us the best of both worlds - the efficiency of bi-encoders for initial retrieval and the accuracy of cross-encoders for final ranking.
from rerankers import Reranker
ranker = Reranker('cross-encoder', model='mixedbread-ai/mxbai-rerank-base-v1',verbose=False)💡 A good safe bet (today) is to use Cohere Rerank for an API for reranking. I'm not using this for this blog post because it requires an API key and it's not neccesary for this intro tutorial, but something you should look into!
def search_blog_posts(query, top_k=3):
"Search blog posts using hybrid search followed by cross-encoder reranking"
# Get candidates with hybrid search
candidates = hybrid_search(query, top_k=top_k*2)
# Rerank candidates
texts = candidates['content'].tolist()
doc_ids = candidates.index.tolist()
ranked = ranker.rank(query=query, docs=texts, doc_ids=doc_ids)
# Map scores back to candidates and return top results
candidates['rerank_score'] = candidates.index.map(
{r.document.doc_id: r.score for r in ranked.results}.get)
return candidates.sort_values('rerank_score', ascending=False).head(top_k)query = "How do I make web components?"
results = search_blog_posts(query)for _, row in results.iterrows():
print(f"Post: {row['post_title']}")
print(f"Section: {row['chunk_title']}")
print(f"Relevance: {row['rerank_score']:.2f}")
print("---")While our hybrid search with re-ranking is already a significant improvement, this are still lots of improvements that can be made.
Semantic search isn't magic - It's about transforming text into numbers that capture meaning. These numerical representation are flawed and cannot be relied on exclusively for every type of query.
Domain Knowledge is king - Understanding your specific domain and content type allows you to make intelligent decisions. Be a user of your own system and actually query and see responses. Do this A LOT. This allows you to start with a simple approach, identifying limitations, and systematically addressing them with targeted improvements.
Hybrid approaches outperform single methods - There is no magic answer that fixes all problems. A combination of vector search, keyword matching, and re-ranking provides significantly better results than any single approach alone. But this is just the basics and there are many more things to add based on use-case.
Chunking matters - How you divide your content has a huge impact on retrieval quality and user experience. The chunks should be meaningful units that preserve context while remaining focused enough to be useful. After chunking, look at them to see if they make sense!
The bi-encoder/cross-encoder pattern is widely applicable - The pattern of using a fast but less accurate method for initial retrieval, followed by a slower but more accurate method for refinement is super powerful
Evaluation is essential - Though we didn't cover it in this tutorial, having a way to measure search quality is critical for ongoing improvement. What gets measured getscan be improved.
If you've followed along to this point, you now have a powerful semantic search system for your blog posts. But there's massive room for growth and experimentation:
💡 Check out the companion repo to start experimenting!
Add Metadata Filtering: Enable users to narrow results by programming language, difficulty level, or post type (e.g., "Show me only Python tutorials for beginners").
Add Multi-Modal Search: Incorporate images, diagrams, and code snippets in your search index to find visual explanations alongside text (e.g., "Find me posts with architecture diagrams for microservices").
Add Evaluation Framework: Build a systematic way to measure search quality with metrics like precision and recall to continuously improve your system. Create implicit evaluation (link tracking) or user feedback systems.
Add Query Pre-processing: Identify and prioritize technical terms and entities in user queries to better match domain-specific content (e.g., recognizing "React hooks" as a specific technical concept).
Add Query Classification: Detect the intent behind searches to provide more tailored results (e.g., distinguishing between "how to" tutorials vs conceptual explanations).
Add Query Expansion: Automatically add related technical terms to queries to improve recall (e.g., expanding "web components" to include "custom elements" and "shadow DOM").
Experiment with different chunking strategies: Test different chunking strategies and sizes to find the optimal balance between context preservation and specificity for your content.
"More Like This" Functionality: Allow users to find similar content to a post they're already reading, creating a natural exploration path through your technical content.
I'd love to hear about your experiences implementing semantic search:
Share your implementation, ask questions, or suggest improvements in the comments below or reach out on Twitter/X @isaac_flath.
Remember, the field of semantic search and retrieval is evolving rapidly. The techniques we've covered provide a solid foundation, but staying curious and experimental will keep your system at the cutting edge.
Get notified about new posts on AI, web development, and tech insights.