Writings

Turning Quick Notes Into Long-Term AI Memory

September 7, 2026 / 10 min read

#AI#Automation#Hermes

How I turned a simple self-hosted notes app into a low-friction capture system that lets Hermes interpret, consolidate, and recall useful context over time.

I’ve been experimenting with a simple idea: what if taking a note was enough?

No filing it into the perfect folder. No deciding whether it belongs under “Projects,” “People,” or “Ideas.” No remembering to copy useful information into an AI assistant later.

I wanted to be able to write a quick thought somewhere convenient and let my agent take care of the rest.

For me, that starts with Memos, a lightweight self-hosted note-taking app that feels a little like a private Twitter timeline. I use Hermes Agent as my AI assistant, with OpenViking providing persistent long-term memory.

Connecting the three has created one of the more useful personal AI workflows I’ve built:

Memos is where thoughts enter the system.
Hermes decides what matters.
OpenViking remembers it.

flowchart TD
    Me["Me"]:::c
    Memos["Memos"]:::c
    Hermes["Hermes Agent"]:::c
    Memory["OpenViking"]:::c

    Me -->|"write notes"| Memos
    Hermes <-->|"shared notes"| Memos
    Hermes -->|"durable context"| Memory
    Memory -->|"semantic recall"| Hermes

    classDef c fill:#dbeafe;

The result is a loop where my notes aren’t just stored. They can become context my agent actually uses later.

Memos as an inbox for thoughts

The part I like most about Memos is how little commitment a note requires.

I don’t have to decide what a thought is before writing it. It could be something I want to remember, an idea for a project, a person I met, a task, a decision, or a random observation that turns out not to matter at all.

I just write it.

That makes Memos a good capture layer.

Traditional knowledge-management systems often encourage organization at the moment of capture. That works for some people, but it creates friction. A five-second thought can quickly become a filing exercise.

I wanted the opposite:

Capture first. Organize later.

And ideally, let the agent do most of the organizing.

Giving Hermes its own identity — and boundaries

One part of the setup I especially like is that Hermes doesn’t operate under my Memos account. It has its own account:

  • danbarbarito — my account
  • hermes — Hermes Agent

When Hermes creates a memo, it shows up as coming from Hermes Agent 🤖. When I create one, it comes from me.

That makes Hermes feel like another participant in the system rather than an invisible process impersonating me. It also creates a useful privacy boundary.

I use Memos’ visibility levels to control what the agent can see:

  • PROTECTED — visible to me and the authenticated Hermes account, so it can participate in the AI workflow.
  • PRIVATE — visible only to me. Hermes cannot access it.

If I write something I explicitly do not want an AI system to process — passwords, sensitive credentials, or simply something I want to keep to myself — I mark it PRIVATE.

Everything I’m comfortable sharing with Hermes can remain PROTECTED.

flowchart TD
    Note["I create a memo"]:::c
    Note --> Visibility{"Visibility"}:::c

    Visibility -->|"PROTECTED"| Shared["Available to Hermes"]:::c
    Visibility -->|"PRIVATE"| Private["Human-only"]:::c

    Shared --> Agent["Hermes may process it"]:::c
    Private --> Stop["Stops here"]:::c

    classDef c fill:#dcfce7;

I like that this boundary exists at the application layer itself. I don’t have to rely on a prompt telling Hermes to ignore certain notes. It simply doesn’t have access to them.

A two-way loop between me and the agent

There are two directions in the workflow.

Memos can send information to Hermes, and Hermes can write information back into Memos.

The first gives the agent access to my stream of thoughts. The second is what makes the setup feel much more interesting: Memos becomes a shared space where both of us participate.

Memos → Hermes

When I create or update a memo that Hermes is allowed to see, Memos sends it to a webhook handled by Hermes.

Hermes then decides what the note means and whether any of it is worth preserving beyond the original memo.

flowchart TD
    Write["I write a PROTECTED memo"]:::c
    Webhook["Memos sends it to Hermes"]:::c
    Interpret["Hermes interprets it"]:::c
    Decide{"Worth remembering?"}:::c
    Remember["Extract durable context"]:::c
    Ignore["Move on"]:::c
    OpenViking["OpenViking"]:::c

    Write --> Webhook --> Interpret --> Decide
    Decide -->|"yes"| Remember --> OpenViking
    Decide -->|"no"| Ignore

    classDef c fill:#f3e8ff;

Not every memo deserves to become long-term memory.

If I write:

Pick up paper towels

there probably isn’t much reason for Hermes to remember that six months from now.

But something like:

Dinner next Thursday with a friend I’ve been spending more time with.

contains potentially useful context: there’s a person I’m interacting with regularly, we have plans, and that relationship may become relevant in a future conversation.

The memo is the raw observation. Hermes decides what, if anything, should survive beyond it.

Hermes → Memos

This is the part of the workflow I find more interesting.

Hermes authenticates to Memos through its own account and can create memos through MCP. So Memos isn’t simply a place where I feed information into an AI system. The agent participates in the same space.

Sometimes that happens ad hoc. If Hermes reaches an important conclusion, discovers something useful, or makes a decision worth preserving, it can leave a memo of its own.

That creates a lightweight “show your work” timeline. Instead of the agent disappearing into a black box after I give it a task, I can open Memos and see what it has been doing.

Hermes also runs a scheduled review every six hours.

During that pass, it looks at recent activity, checks what it wrote previously to avoid repetition, and considers the broader context rather than treating every memo as an isolated event.

It can ask questions like:

  • Is this actually new information?
  • Does it reinforce or contradict something already known?
  • Is it part of an ongoing project or idea?
  • Does it reveal a preference or recurring pattern?
  • Is it temporary noise?
  • Should several small observations become one stronger memory?

That review gives Hermes a chance to consolidate what has happened rather than simply accumulating one-off memories. If there’s something meaningful to report, it writes a short update back into Memos.

So the timeline ends up containing two perspectives:

Me

  • thoughts
  • observations
  • reminders
  • ideas

Hermes

  • conclusions
  • progress
  • useful discoveries
  • periodic summaries

Hermes-created memos are also PROTECTED, keeping the whole exchange inside the shared workspace.

And there’s an even simpler interaction I use constantly: Hermes can reply directly to a memo.

If I jot down a question, Hermes can answer it on the note. If I capture an idea, it can add context, point out something I missed, or push back on an assumption. I can respond again, and the memo becomes a conversation without me ever leaving the place where I originally captured the thought.

flowchart TD
    Note["I write a PROTECTED memo"]:::c
    Hermes["Hermes reads it"]:::c
    Reply["Hermes replies"]:::c
    Me["I respond"]:::c
    Thread["A note becomes a conversation"]:::c

    Note --> Hermes --> Reply --> Me --> Thread

    classDef c fill:#fef9c3;

That’s what makes Memos feel less like a normal notes app and more like a shared activity feed between me and my agent.

Notes and memory are not the same thing

This became the most important conceptual distinction in the setup.

A note is something I wrote.

A memory is context the agent extracted from what I wrote.

Those shouldn’t necessarily be identical.

Imagine I write several memos:

Thinking about building a personal AI memory system.

I like Memos because I don’t have to organize anything when I write.

Agent memory should probably separate recent activity from long-term knowledge.

Those are three separate entries in my timeline.

But together, Hermes might infer something more useful:

The user prefers a low-friction capture system where raw notes remain separate from agent-maintained long-term knowledge.

That synthesized memory is much more valuable later than retrieving three disconnected notes.

And because Hermes periodically revisits recent activity, the memory doesn’t have to be perfect the first time. It can evolve as more context appears.

OpenViking as durable memory

When Hermes decides something should persist, it sends that context into OpenViking.

OpenViking gives Hermes a semantic memory layer for things like entities, events, preferences, and recurring patterns. Retrieval can happen by meaning, not just by matching the exact words I originally used.

Months later, I might ask:

What do you remember about that person I was seeing earlier this year?

Hermes doesn’t need me to remember which memo mentioned them or how I phrased it. It can search long-term memory and retrieve the accumulated context.

flowchart LR
    Raw["Raw Memos"]:::c --> Agent["Hermes"]:::c

    Agent --> Facts["Facts"]:::c
    Agent --> People["People"]:::c
    Agent --> Events["Events"]:::c
    Agent --> Preferences["Preferences"]:::c
    Agent --> Patterns["Patterns"]:::c

    Facts --> KB["OpenViking"]:::c
    People --> KB
    Events --> KB
    Preferences --> KB
    Patterns --> KB

    KB -->|"semantic recall"| Agent

    classDef c fill:#ccfbf1;

I don’t want to maintain a giant prompt containing everything Hermes should know about me.

I want useful context to accumulate naturally through normal interactions.

Why this workflow works for me

The biggest benefit isn’t really technical. It’s behavioral.

The cheaper it is to capture information, the more likely I am to do it.

If every useful thought requires opening a knowledge base, finding the right section, deciding where it belongs, formatting it, and determining whether it matters enough to keep, I’ll simply write down fewer things.

With this setup, my responsibility ends much earlier:

Write the thought.

Hermes handles the interpretation and consolidation later.

There are a few other benefits I’ve found useful.

The original record stays intact. Memos remains the chronological record of what I actually wrote and what Hermes actually said. OpenViking can hold a more abstract interpretation without replacing the source material.

The agent has boundaries. PROTECTED and PRIVATE notes let me integrate Hermes deeply without giving it blanket access to everything.

The system improves with use. An individual memo might be trivial, but enough small observations can eventually reveal a project, preference, relationship, or pattern that becomes useful context later.

And because Hermes writes back into Memos, I can also see that process happening instead of treating long-term memory as an invisible database somewhere in the background.

A small webhook wrinkle

The implementation wasn’t completely frictionless.

Memos and Hermes disagreed on how webhook signatures were presented, so I had to use a local-only workaround to connect the two. I filed a bug report documenting the compatibility issue.

Because the services run locally on the same host, the workaround was acceptable for my setup.

Edit: Seems like this was fixed in a recent version!

Final thoughts

What I like about this setup is that none of the individual pieces need to do everything.

Memos stays a fast, low-friction place to capture whatever is on my mind.

OpenViking focuses on durable context that might matter later.

Hermes sits between them, doing the work I’m least interested in doing manually: interpreting, filtering, connecting, consolidating, and recalling.

That separation lets me keep the convenience of a messy stream of thoughts without giving up the benefits of organized long-term memory.

And because Hermes has its own identity inside Memos, the system isn’t entirely happening behind the scenes. I can see what the agent is thinking, respond to it directly, and maintain a clear boundary around what it is allowed to know.

A lot of personal knowledge-management systems ask me to maintain the knowledge base.

This workflow flips that around.

I maintain the stream of thoughts. The agent maintains the understanding.

And that might just be the workflow I’ve been trying to find for a long time.