"Store this in your library" works better than "remember this". here's why | Sam Partee

Sam Partee (CTO + co-founder of Arcade) sat down with Mateo to walk through the design choices behind Agent Library — the open-source local-first agent memory we released last week. The conversation starts where the project started.

This is less of a launch pitch and more of a design interview: why the word "library" outperforms "memory" with modern agents, why the canonical RAG pipeline (BM25 + vector + RRF + MMR) is actually the right floor for local memory, and the one place where doing the math correctly produced the wrong product behavior.

In this episode, we cover:

- Why naming it "library" (with sections, chapters, books, librarian) gets a higher success rate from agents than "memory"
- "Semantic saturation" — when fine-tuning collides with overloaded words like "memory" and tool-calling gets worse
- The Obsidian origin story: a typo that accidentally let the agent organize its own knowledge graph
- Why agent-organized libraries beat human-curated ones once the corpus is big
- The actual hybrid search recipe: BM25 + vector with reciprocal rank fusion, plus MMR for diversity
- Local-first by default: every model (text embed, code embed, CLIP, OCR, generation) running on a 32GB MacBook Pro
- Read-mostly by design: why deletes require a `--clobber` flag and why that matters for notes
- Local context (Agent Library) vs remote context (Arcade tools to Google Docs, Drive, etc.) — when to use which
- The hardest technical problem: weighting multimodal results when code, text, and images all "win" differently
- Why the code embedding ended up off by default in multimodal search
- What Sam wants from open-source contributors (vector DB swaps, SLM benchmarks, issues from real users)
- A tease for the next open-source release that uses Agent Library as its memory building block

⏰ TIMESTAMPS
00:38 – Welcome and intro
01:03 – Origin story: from an Obsidian tool to Agent Library
04:18 – Why "library" semantics work better than "memory"
07:13 – Hybrid search and the RAG pipeline (BM25, RRF, MMR)
09:54 – Daily use cases and personal workflow
13:18 – Local vs remote context: Agent Library vs Arcade
16:23 – Why SQLite and the fully local stack
19:24 – Open-source contributions and future roadmap
22:42 – Hardest technical challenge: multimodal search weighting
25:29 – What users find most useful, and wrap-up

🔗 LINKS & RESOURCES

Agent Library
- GitHub (Apache 2.0): https://github.com/ArcadeAI/agent-library
- Install: `uv tool install agent-library`

Referenced in this episode
- Thinking, Fast and Slow (the "tier one / tier two" book Sam alludes to): https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow
- Obsidian: https://obsidian.md/
- sqlite-vec: https://github.com/asg017/sqlite-vec
- SQLite FTS5: https://www.sqlite.org/fts5.html
- BM25 / TF-IDF: https://en.wikipedia.org/wiki/Okapi_BM25
- Reciprocal Rank Fusion (RRF) paper: https://doi.org/10.1145/1571941.1572114
- Maximal Marginal Relevance (MMR): https://www.cs.cmu.edu/~jgc/publication/The_Use_MMR_Diversity_Based_LTMIR_1998.pdf
- CodeBERT: https://huggingface.co/microsoft/codebert-base
- Tiangolo / FastAPI (the contributor model Sam aspires to): https://github.com/tiangolo/fastapi

Companion launch episode
- Agent Library: Own Your Agent's Memory in a Single SQLite File: https://www.youtube.com/watch?v=rHBaZkEt9rU

About the guest
Sam Partee, CTO + Co-founder at Arcade
- LinkedIn: https://linkedin.com/in/sampartee/
- GitHub: https://github.com/Spartee
- X: https://x.com/SamPartee

🚀 Try Arcade: https://account.arcade.dev/register?utm_source=youtube&utm_medium=description&utm_campaign=internal_interview&utm_content=agent_library

Looking for the fastest way to build MCP servers?
https://docs.arcade.dev/en/home/custom-mcp-server-quickstart?utm_source=youtube&utm_medium=description&utm_campaign=internal_interview&utm_content=agent_library

💬 Join the conversation: https://discord.gg/GUZEMpEZ9p
🔧 Build with Arcade: https://arcade.dev
📖 Arcade Docs: https://docs.arcade.dev

#AgentMemory #AgentLibrary #RAG #HybridSearch #SQLite #LocalFirst #AIAgents #ArcadeDev

Видео "Store this in your library" works better than "remember this". here's why | Sam Partee канала Arcade