A running log of what I'm building, why I'm building it, and what breaks along the way — the honest engineering diary of a local-first personal AI assistant, built one weekend at a time.
The Vision: Why a Local-First Personal AI Assistant
I'm building Amir — a personal AI assistant that runs entirely on my own hardware. No frontier model APIs. No cloud dependencies. No third party sitting between me and my own data. Just a local model, a local database, and code I understand line by line.
The end state isn't a chatbot. It's an embodied desktop robot — a Reachy Mini paired with a Strix Halo host — that recognizes me by face, hears me by voice, remembers what I've told it, and can act on my behalf across the tools I actually use (Asana, calendar, email, my WordPress sites). Something that sits on my desk, knows who I am, and gets more useful the longer I use it.
Why local-first? Because a personal assistant that ships my life to someone else's servers isn't personal. And because the interesting engineering problem — teaching a small local model to be genuinely useful with structured memory — is the one worth solving.
Why build it myself instead of stitching frameworks together? Because I'm using this project to actually learn Python. Copying framework code wouldn't teach me anything. Writing search_memories() by hand, watching it fail, and figuring out why — that teaches me.
The roadmap is eight phases: cleanup, structured memory, smart retrieval, auto-capture, voice, actions and integrations, desktop robot body, and vision. I'm somewhere between phase 2 and 3 as I write this — roughly 10 hours a week, alongside a full-time data analyst role at Lufthansa's Digital Hangar and consulting work on the side. This is the diary.
Phase 0 — Cleanup & Quick Wins
Before anything smart could happen, the project had to stop tripping over itself. Two databases floating around. Orphan files. Retrieval results coming back unsorted, so the model was getting the top 10 noisy matches instead of the top 3 relevant ones.
The fix was small but it mattered: one canonical database.db, one path through config.py, and a sort_results() call before slicing to the top 3. Boring work — but every phase after this one assumes the floor is stable, and it wasn't.
Phase 1 — Structured Memory
The original memory table was one blob field. Every fact went into the same shapeless column, and searching it meant running LIKE '%zurich%' across every row and hoping.
Phase 1 was the redesign: a typed schema with subject, type (identity, location, work, preference, relationship, activity), key, value, confidence, and status. And crucially, one function — save_fact() — as the only doorway into the memory store. Every write goes through it. No exceptions.
The single-doorway rule turned out to be one of the most important decisions in the project. When the LLM starts writing into memory, having exactly one function to guard is the difference between a validation problem and a chaos problem.
Phase 2 — Smart Retrieval & RoutingDone
This is the phase I just finished, and it was the hardest so far. The old retrieval ran LIKE queries across every column of every table. It returned matches, but they were disconnected fragments — a pile of shrapnel the model was expected to reason from.
The fix was search_memories(): a typed lookup that returns coherent row dicts — the whole row, all together. The unit of meaning is the row, not the cell. Then memories_retrival() on top as the orchestrator: it extracts terms, calls the search, deduplicates using the two-container pattern (a set of IDs for O(1) membership checks, a list of rows for ordered output), scores, sorts, and returns the top 3.
Two bugs worth remembering
- The missing comma. Two adjacent strings with no comma between them. Python silently concatenated them into one long useless string — no error, no warning. Valid memory questions were routing wrong and I couldn't figure out why. Traced it by walking through what the code actually did, line by line.
- A variable name mismatch in the dedup logic. Also silent. Also traced by reading, not guessing.
None — are the ones that eat weekends. Debug by tracing execution, not by re-reading intent.The chatbot now uses the new pipeline end-to-end. Direct recall works. Absence-checking works. Numeric facts come back correctly. Hybrid reasoning (memory plus general knowledge) works. It genuinely feels different to talk to.
Phase 3 — Auto-Capture (Next)
The phase I've been most excited about, and the one with the highest risk profile. Right now, saving a memory means typing a trigger phrase. Phase 3 removes that: when I say "Ahmad is my client and he lives in Dubai," Amir decomposes it and files each piece through save_fact() automatically.
The catch: this puts the LLM in the write path. Bad retrieval is ephemeral — the model says something wrong, I roll my eyes, next turn is fine. Bad extraction is persistent — it becomes a row in my database and pollutes every future query.
Before I touch auto-capture, there's non-negotiable pre-work: save_fact() currently inactivates any prior row on a type + key match. That's correct for preferences, but wrong for activities and events, where history should coexist as active rows. Then a Pydantic validation shield between the LLM's JSON output and save_fact() — shape and type validation, not truth validation. If the model returns something malformed, it never reaches the database. Roughly 60–90 hours over 8 weeks.
Phase 4 — Voice Loop
Speech-to-text in, text-to-speech out, a wake word — all local. This is where the ESP32-S3 starts earning its keep: a physical wake button on the desk before the wake word is reliable, then an LED ring that pulses when Amir is thinking. A robot with no feedback feels like a box. A box that lights up feels alive.
Phase 5 — Actions & Integrations
Read-only tool calling into Asana, calendar, and email. "What should I focus on today?" answered from real tasks, not general advice. This is where a bigger local model — 70B on the Strix Halo I'm targeting for end of 2027 — starts to matter. I'll probably rent cloud GPU time to validate 70B quality before committing to hardware. No real memory data going through rented machines, obviously.
Phase 6 — Desktop Robot
Reachy Mini by Pollen Robotics, paired with the Strix Halo host. A different discipline — electronics, motors, physical build — essentially a second project. But by the time I get here, the brain will exist. The body just gives it presence.
Phase 7 — Vision
Camera plus face recognition. Amir recognizes me and my family by sight: when I sit at the desk, he activates; when I leave, he goes idle. Presence detection first (is a face there, yes/no), recognition second (whose face). ESP-WHO on the ESP32-S3 can handle the first part locally, no cloud dependency.
Current Hardware Reality
Right now Amir runs on a laptop with an 8GB RTX 3080. The local model eats about 7.5GB of that, so anything else spills to CPU and reply times crawl to 3–4 minutes. Usable for development, painful for real conversation.
The planned upgrade — end of 2027 — is a 128GB AMD Strix Halo mini-PC. Unified memory, roughly 96GB usable for the GPU, enough headroom for 70B models. Compact enough to eventually live inside the robot, native Linux, and cheaper than the laptop equivalent.
Principles That Keep Proving Right
- Row as unit of meaning. Fetch coherent facts, not scattered cells.
- Pure specialists, clear orchestrators.
search_memories()does one thing;memories_retrival()composes. Don't conflate. - Sort before slice. Order matters before truncation. Obvious in retrospect.
- Single write doorway. Every fact goes through
save_fact(). No shortcuts, no bypasses. - Milestone before integration. Prove each piece works in isolation before wiring it in. Every time I skipped this, I paid for it.
- One focused change, then confirmation. Big diffs hide bugs. Small diffs surface them.
Why I'm Writing This Diary
Two reasons. One: I want a public record of the project as it actually happens — the bugs, the wrong turns, the small wins — not a polished post-hoc case study written after everything works. If someone else wants to build a local-first personal AI assistant, the missing-comma story is more useful than the architecture diagram.
Two: I'm using this project to learn Python. Writing the diary forces me to articulate what I actually learned each phase, in my own words, without pasting code. If I can't explain the two-container dedup pattern in a paragraph, I don't understand it yet.
Next entry: the pre-work for phase 3 — the type-aware save_fact() fix. The one that has to happen before the LLM writes anything.
Want the technical deep-dives on this local-first personal AI assistant? Each phase gets its own long-form post with the code, the bugs, and the reasoning. This diary is the through-line, and the full project lives on GitHub.
View the project on GitHub →
