The challenge
User research found that 40–60% of users said their information was scattered and the reasoning behind decisions was lost. Early feedback was blunt: the product “looked like another Asana or ClickUp”, with the AI hidden behind conventional navigation.
There was also a hidden fault. Semantic search had been built on an external vector service that was never populated, so the feature appeared to work while finding nothing.
What I built
I rebuilt retrieval on pgvector inside the product’s own PostgreSQL database, with no external vendor, and scoped the vectors to the same user as the content they describe. I measured the relevance threshold against real data instead of inheriting it: the original setting rejected every genuine match while raising no error at all.
Then I moved the intelligence to where users already are. Search returns results by meaning with the matching passage shown. Answers carry their sources. Decisions are captured with their reasoning so they can be found months later. The dashboard opens with a briefing of what changed since the last visit instead of a row of counts. Memory of user preferences is visible and editable, and every action that changes data asks for confirmation before it runs.
The assistant runs on LangGraph and LangChain with OpenAI models, behind JWT and bcrypt authentication with role-based access. The backend has 2,711 Pytest tests and a 342-turn evaluation suite run against the live model; the frontend has 295 Jest tests.
Architecture and stack
- Frontend
- Next.js 15 · React 18 · TypeScript · MUI v6 · TanStack Query · React Hook Form + Zod · Framer Motion
- Backend
- Python · FastAPI · SQLAlchemy 2 · Alembic · Redis
- AI & retrieval
- PostgreSQL with pgvector · LangGraph · LangChain · OpenAI (GPT-4o-mini, text-embedding-3-small)
- Infrastructure
- Docker Compose · Caddy · AWS EC2 · AWS RDS · S3 · Sentry
- Testing
- Pytest (2,711 tests) · Jest + Testing Library (295 tests) · 342-turn live-model evaluation suite
Outcome
Semantic search now finds real matches, and every answer shows where it came from, which is what makes an AI assistant usable for decisions. The product no longer looks like another task tracker: it opens with what changed and answers questions about the work itself.
The lesson is one I apply to every RAG project: measure retrieval against real data before trusting it, because a search feature can look finished while returning nothing.
Last updated