The Memory Layer | Chris Latimer (CEO, Vectorize)

Listen to full episode:

Summary: This episode, hosts Anant and Ed are joined by Chris Latimer, Co-Founder of Vectorize, to unpack why context management matters, why simple vector search often falls short, and how agent memory can be architected to support real production systems. Chris breaks down his team's framework for thinking about memory as a mix of semantic, episodic, and working memory — tying product design, retrieval strategy, and model behavior together into a practical blueprint for building agents that actually get smarter over time.

Chapters:

00:00 - Podcast intro

02:05 - Introducing the episode topic: memory layers for agents

11:43 - Why memory is central to useful agents

13:24 - Introducing Chris Latimer and Vectorize

14:52 - The first wave of LLM magic and its blind spots

15:52 - Early RAG lessons and "rag hell"

17:12 - Why "RAG is dead" is usually just a misunderstanding

21:13 - Breaking memory into semantic, episodic, and working memory

24:26 - Building mental models instead of only raw recall

29:58 - Four retrieval strategies: vector, keyword, graph, and time

33:11 - Why re-ranking matters more than most teams realize

34:26 - Demo setup: an agent that learns by doing deliveries

54:35 - The counter bet: context windows versus memory systems

Sound Bites:

“Rag is a drag.”

“Why would I bother [shoving everything into the context window] if I don’t get significantly better results?”

“I’ve never worked anywhere... where our problem is that we don’t have enough work to do.”