Sarah Glasmacher
Sarah Glasmacher
my journey through ML and studies
⌘K

February: RAG Relevancy - Improving Relevance Filtering in RAG Search

For my February project, I’m diving into a challenge that has been holding back my local RAG system (see my page about this project started in January): retrieval relevance. Right now, my system always returns the top 3 results based on cosine similarity - even if they’re not actually relevant. This month, I want to explore better ways to filter and rank retrieved results, beyond just relying on similarity scores. One main task will be to figure out when the LLM is better off answering the question without any sources.

What I'm Aiming For: Project Goals

By the end of February, I want to:

  • Look at why some irrelevant results appear and identify patterns in bad matches.
  • Implement basic filtering strategies like cosine similarity thresholds, metadata filtering, or embedding distance ratios.
  • Explore LLM-based filtering where an AI model evaluates whether retrieved results truly answer the query.
  • Document findings and integrate at least one improvement into my retrieval pipeline.

Since I'm starting this month's project super late (in the last week of February like a true procrastinator), this is a smaller project scope and focuses mainly on exploration where every bit of knowledge gained is a win.

I’ll be sharing my experiments and code updates in the same repository as before:

➡️ https://github.com/GalaxyInfernoCodes/local-rag-system

Why I’m Doing This & Initial Plan

This project is a mix of exploration and practical improvements. While vector similarity is a great start, retrieval quality matters just as much as generation quality in a RAG system. I want to better understand how different techniques impact relevance so I can fine-tune my system.

Why Share This?

I believe learning out loud is the best way to grow! Sharing my thought process, experiments, and results helps me stay accountable and (hopefully) helps others working on similar problems.

Further Posts & Updates

I’ll be documenting my progress, roadblocks, and findings in the following blog posts:

  • Project Kickoff & Goals: Why I’m focusing on RAG retrieval relevance and my initial plan.
  • Filtering Strategies Explained: Quick tests on thresholding, metadata filtering, and more.
  • LLM-Assisted Relevance Filtering: Experimenting with AI-based verification of search results.