Q1: VectorVault - a Local RAG System from Scratch with Open-Source priority
This is the project overview page for my monthly project idea: creating my very own local RAG-system to search through my notes with natural language. I’ve been using Obsidian for everything - blog drafts, social media ideas, and work notes - but sometimes I'm too lazy to look for the right note. So, I’m building a system that lets me ask questions like, “What were my ideas for improving RAG workflows?” and instantly get the most relevant snippets.
I’ll start simple, using cloud APIs for the AI components, and gradually transition to open-source tools to make it all fully local (ideally).
What I'm aiming for: January Project Goals
By the end January, I want to have a working system that:
- Stores my notes as embeddings (fancy AI representations of text) in a local database
- Lets me type in questions or keywords and pulls up the most relevant snippets
- Generates summaries or answers using AI
Why I’m Doing This & Initial Plan
I wrote a whole blog post about starting this project, which you can find here: https://sarahglasmacher.com/starting-january-project-local-rag-system/
Read that blog post if you are interested in any of the following:
GitHub Link
There's not much to see at the time of initially posting this, but here is the repo:
https://github.com/GalaxyInfernoCodes/local-rag-system
February: RAG Relevancy - Improving Relevance Filtering in RAG Search
For my February project, I’m diving into a challenge that has been holding back my local RAG system: retrieval relevance. Right now, my system always returns the top 3 results based on cosine similarity - even if they’re not actually relevant. This month, I want to explore better ways to filter and rank retrieved results, beyond just relying on similarity scores. One main task will be to figure out when the LLM is better off answering the question without any sources.
What I'm Aiming For: February Project Goals
By the end of February, I want to:
- Look at why some irrelevant results appear and identify patterns in bad matches.
- Implement basic filtering strategies like cosine similarity thresholds, metadata filtering, or embedding distance ratios.
- Explore LLM-based filtering where an AI model evaluates whether retrieved results truly answer the query.
- Document findings and integrate at least one improvement into my retrieval pipeline.
Since I'm starting this month's project super late (in the last week of February like a true procrastinator), this is a smaller project scope and focuses mainly on exploration where every bit of knowledge gained is a win.
Further Posts and Updates about the project
Here's a list of all blog posts around the project:
- Starting blog post: Why I'm Doing and Sharing this & Initial Outline
- How to set up pgvector with Docker: Local Vector Database for Text Embeddings
- How to Use Cosine Similarity for Vector Search in pgvector
I’ll be documenting my progress, roadblocks, and findings in the following blog posts:
- Project Kickoff & Goals: Why I’m focusing on RAG retrieval relevance and my initial plan.
- Filtering Strategies Explained: Quick tests on thresholding, metadata filtering, and more.
- LLM-Assisted Relevance Filtering: Experimenting with AI-based verification of search results.