Sign InSign Up

🔮 Hermes Agent 🤖 — Deep Dive & Build-Your-Own Guide 📘

A practical, end-to-end walkthrough of Nous Research's Hermes Agent: the principles it's built on, the architecture that makes it work, and a concrete checklist for building a similar self-improving agent yourself.

Companion Reads: 🏗️ Building High-Quality AI Agents 🤖 — A Comprehensive, Actionable Field Guide 📚, 📎 Paperclip Deep Dive 🤖 — A Build Guide for an "AI Company" 🏢 Control Plane, 🤖 Multica Deep Dive — How to Build a Managed-Agents Platform 🌐.

📋 Table of Contents

🤖 1. What Hermes Actually Is (in one paragraph)

Hermes is a model-agnostic, self-improving conversational agent that runs locally as a CLI/TUI, on a server as a messaging gateway (Telegram/Discord/Slack/WhatsApp/Signal), or as a scheduled cron worker. Its key differentiator is a closed learning loop: while solving problems with tools, it writes reusable "skill" documents and curates a persistent memory file so the agent quite literally gets more capable the longer it runs. Everything — model, tools, skills, memory backend, execution environment, UI — is pluggable.

Two ideas to internalize before you build anything:

  1. One agent, many surfaces. A single AIAgent class powers every interface. Surfaces (CLI, gateway, cron, batch, API) are thin entry points that construct an agent and call run_conversation().
  2. Procedural memory > clever prompting. Most "smart agent" behavior comes not from prompt engineering but from the agent owning a folder of markdown documents (skills + memory + persona) it can read, write, and grow over time.

🧭 2. Core Principles

These are the design rules Hermes follows. Keep them in mind for your own build — most "weird" decisions in the codebase trace back to one of these.

2.1 🌐 Platform-agnostic core

The agent doesn't know whether it's running in a terminal, a Telegram chat, or a cron job. All platform specifics live in adapters that translate platform events → agent.run_conversation(...) and translate the response back. If you find yourself adding a Telegram-specific if branch inside core agent code, you've drifted from the architecture.

2.2 🔒 Prompt stability (cache-friendly)

The system prompt is assembled once at session start and does not mutate mid-conversation. This isn't aesthetic — it's economic. Anthropic and OpenAI prompt caches require a stable prefix to get hits. Mid-conversation toolset changes, memory reloads, or skill swaps invalidate the cache and 10× your cost. Defer changes to "next session" by default.

2.3 🔍 Progressive disclosure

Don't load every skill, every memory, every tool's full docs into the system prompt. Load descriptions (Level 0). Let the agent pull in full content (Level 1) only when it actually needs that skill. Load referenced files (Level 2) only when the skill itself requests them. This is how Hermes can ship 47 tools and dozens of skills while staying under context limits.

2.4 📝 Self-registration over central lists

Tools and plugins should register themselves at import time (registry.register(...)) rather than being added to a hand-maintained __all__ list. New tool = one new file, no edits elsewhere.

2.5 🧱 Profile isolation

Multiple independent agent instances coexist by each owning a HERMES_HOME directory (default ~/.hermes/, override via env var). Every filesystem path in the codebase goes through get_hermes_home() — never hard-code ~/.hermes.

2.6 🎒 The agent owns its own learning artifacts

Skills are not added by humans editing source code. The agent writes them via a tool called skill_manage after solving a non-trivial task. Memory is not curated by humans — the agent edits MEMORY.md and USER.md between turns. This is the loop.

🏗️ 3. High-Level Architecture 

┌──────────────────────────────────────────────────────────────────┐
│                          ENTRY POINTS                            │
│  CLI / TUI / Gateway (TG, Discord, Slack) / Cron / Batch / API   │
└──────────────────┬───────────────────────────────────────────────┘
                   │   each entry point builds an AIAgent
                   ▼
┌──────────────────────────────────────────────────────────────────┐
│                       AIAgent (core loop)                        │
│  build_system_prompt → call model → dispatch tool calls → repeat │
└─────┬─────────────┬────────────────┬────────────────┬────────────┘
      │             │                │                │
      ▼             ▼                ▼                ▼
┌──────────┐  ┌──────────┐    ┌────────────┐   ┌─────────────┐
│  Tools   │  │  Skills  │    │   Memory   │   │  Providers  │
│ Registry │  │  Loader  │    │  Manager   │   │ (model API) │
└──────────┘  └──────────┘    └────────────┘   └─────────────┘
      │
      ▼
┌──────────────────────────────────────────────────────────────────┐
│  Execution Environments: local / Docker / SSH / Modal / Daytona  │
└──────────────────────────────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Three tiers, in plain English:

  • Tier 1 — Surfaces: how a human or system talks to the agent (CLI, chat platforms, cron).
  • Tier 2 — Agent core: the loop, plus the four pluggable subsystems (tools, skills, memory, model).
  • Tier 3 — Execution backends: where shell/code-running tools actually run. Local laptop today, sandboxed Docker tomorrow, Modal cloud in production.

🔄 4. The Agent Loop (the heart of everything)

This is the single most important piece. The whole AIAgent class is essentially this loop:

1. Receive input            → from CLI / gateway / cron / ACP / web
2. Build system prompt      → persona + memory + skills + tools (ONCE per session)
3. Resolve provider         → which API key + endpoint for the chosen model
4. Call model               → one of FOUR API modes (auto-detected by endpoint/model):
                              chat_completions | codex_responses |
                              anthropic_messages | bedrock_converse
5. Parse response
   ├─ if tool calls present → dispatch each via registry → append results → GOTO 4
   └─ else                  → final assistant message → display → persist → done
6. Persist                  → SQLite SessionDB (WAL mode + FTS5 index)
Enter fullscreen mode Exit fullscreen mode

A few non-obvious details that matter:

  • Iteration budget — more nuanced than a simple counter. A thread-safe IterationBudget is shared across the parent agent and any subagents it spawns. execute_code refunds iterations on completion so a programmatic tool-loop doesn't drain the budget. On exhaustion: one warning message is injected (_budget_exhausted_injected), exactly one final API call is allowed (_budget_grace_call), then summarization is forced. No intermediate warnings — deliberate, to prevent the model from giving up early.
  • Reasoning content is stored separately from the visible assistant message (OpenAI o-series and Anthropic extended thinking both produce hidden "reasoning" tokens). Keep them in their own field; they're needed for cache validity but shouldn't be displayed. Callbacks: stream_delta_callback, interim_assistant_callback, thinking_callback, reasoning_callback.
  • Streaming with stateful scrubbing. A _stream_context_scrubber strips <memory-context> spans even when they're split across chunks — don't underestimate how fiddly this gets when tags straddle network boundaries.
  • Compression, not truncation. When context fills, a context_compressor summarizes middle turns rather than dropping them. The summary itself becomes a message. Lossy is fine; lossless will OOM.
  • Interrupts. Ctrl-C mid-tool-call must cleanly cancel the in-flight tool, append a "user interrupted" tool result to history, and return control. Don't kill the whole loop — let the agent see the interruption and respond.
  • Session resumption. --continue / --resume flags load prior history via SessionDB.get_messages(). SQLite WAL mode + a custom retry layer (20–150 ms jitter, BEGIN IMMEDIATE) handle multi-process write contention. A recap is shown to the user before continuing.

🧩 5. System Prompt Assembly

A prompt_builder.build_system_prompt() function concatenates these sections, in this order:

  1. Persona — SOUL.md / DEFAULT_AGENT_IDENTITY. Identity, voice, values.
  2. Platform hints — PLATFORM_HINTS. Tells the model whether it's running in CLI, Telegram, Slack, etc. — this changes formatting rules (no MarkdownV2 in CLI, no nested code blocks in Telegram, …).
  3. Memory guidance — MEMORY_GUIDANCE. Embeds a frozen snapshot of MEMORY.md + USER.md as a single block (separated by a § delimiter). Size-capped (~2200 chars MEMORY, ~1375 chars USER).
  4. Session search guidance — SESSION_SEARCH_GUIDANCE. Tells the agent it can search prior sessions via FTS5, with a small example.
  5. Skills guidance — SKILLS_GUIDANCE. The Level-0 skills index plus the heuristic prose nudging the agent to create skills after solving hard tasks.
  6. Context files — AGENTS.md and .hermes.md from the working directory.
  7. Tool-use enforcement — TOOL_USE_ENFORCEMENT_GUIDANCE. Hard rules about parallel calls, error recovery, etc.
  8. Tool schemas — JSON schemas for all enabled tools.

Then prompt_caching.py inserts cache breakpoints (Anthropic cache_control: {type: ephemeral}; equivalents for other providers). The whole assembled prefix becomes the cacheable region.

The frozen-snapshot pattern (this is the trick). MEMORY.md and USER.md are read once at session start and embedded immutably in the system prompt for the rest of the session. The agent can still write to those files on disk during the session — but the system prompt does not change. Result: cache stays valid across the whole conversation, and the new memory takes effect next session. Skip this and you destroy your prefix cache.

Memory security scan. Before injection, MEMORY/USER content is scanned for prompt-injection patterns, exfiltration attempts (curl/wget referencing env vars), persistence backdoors, and invisible Unicode. A poisoned memory file is the agent's prion disease — scan defensively.

Key rule: sections 1–8 are frozen for the session. User messages and tool results are appended to history; they don't go into the system prompt.

🛠️ 6. Tools System

6.1 📦 Self-registering registry

A central tools/registry.py exposes:

registry.register(
    name="read_file",
    toolset="filesystem",
    schema={...JSON schema...},
    handler=read_file_handler,
    available=lambda ctx: True,   # gating predicate
)
Enter fullscreen mode Exit fullscreen mode

Every tool file calls this at module import. The registry handles:

  • Schema collection for the system prompt.
  • Dispatch by name when the model emits a tool call.
  • Availability filtering (per-user, per-platform, per-toolset).
  • Error wrapping — any exception in a handler is converted into a tool result the model can see and react to. Never let a tool crash the loop.

All handlers return JSON strings, not Python objects. The model only ever sees text.

6.2 🗂️ Toolsets

Tools group into logical sets (filesystem, web, browser, code, mcp, vision, audio, …) — Hermes ships ~40+ tools (the docs say "47 built-in" in some places, "40+" in others; AGENTS.md says the filesystem is the canonical source because counts shift constantly — don't hard-code numbers in your own version). Users enable/disable by toolset rather than tool-by-tool. Disabled toolsets are completely absent from the system prompt — saves tokens and prevents the model from even knowing about them.

6.3 🖥️ Execution environments

Tools that run shell commands or code go through an environment abstraction (tools/environments/):

Backend Use case
local Dev on your laptop. Fastest. Zero isolation.
docker Shared dev box. One container per session.
ssh Remote VM. Treat the VM as the agent's "computer".
daytona / modal Serverless sandboxes for production. Auto-spin-up.
singularity HPC clusters.

Same tool, different blast radius. The agent doesn't know — only the environment changes.

6.4 🤖 Agent-level tools

A few tools (todo_*, memory_*, skill_manage, skills_list, skill_view) are intercepted before the generic tool dispatch and handled by the agent itself, because they mutate agent state (memory, skills, todo list) rather than the outside world. Keep this category small and explicit.

6.5 🔗 MCP integration

Model Context Protocol servers can be plugged in as additional tool sources. Hermes treats each MCP server as a virtual toolset, lets users filter individual tools, and dispatches calls through the same registry. This is how you get a long tail of integrations (GitHub, Slack, Linear, ...) without writing them yourself.

6.6 🛡️ Tool approval & safety (the layered defense)

Shell tools are dangerous. Hermes layers four mechanisms:

  1. Tirith — an external Rust-based scanner with auto-install + SHA-256 verification. Detects homograph URLs, terminal-injection attacks (ANSI escapes that hide commands), and known dangerous patterns.
  2. Regex dangerous-command detection — runs on a normalized command string (case-insensitive, whitespace-collapsed) so attackers can't bypass via RM -RF.
  3. Smart Approval — an LLM risk-rates each command. Low-risk auto-approves; medium/high blocks for human approval.
  4. Approval scopes — when a human approves, they pick Once / Session / Permanent. Trust accumulates instead of asking on every call.

When the agent is running on a messaging gateway and needs approval, it uses a threading.Event to block until the human responds in chat. A /yolo command bypasses approval entirely for trusted sessions. Sandboxed backends auto-bypass approval (the Docker/Modal sandbox is the safety boundary; double-prompting is just friction).

🧠 7. Skills System (the killer feature)

7.1 📄 What a skill is

A skill is a markdown document with YAML frontmatter that teaches the agent how to do one thing well. Not code. Not a config. A runbook the agent reads.

---
name: deploy-staging
description: Push current branch to staging via Vercel and verify health.
version: 1.2.0
platforms: [macos, linux]
requires_toolsets: [shell, web]
fallback_for_toolsets: []
required_environment_variables: [VERCEL_TOKEN]
tags: [deploy, vercel]
category: devops
---

## When to Use
The user asks to "ship", "deploy to staging", or "preview this branch".

## Procedure
1. Run `git status` — abort if dirty.
2. Run `vercel --token=$VERCEL_TOKEN`.
3. Poll `/healthz` until 200 OR 60s timeout.
4. Report the preview URL.

## Pitfalls
- Don't deploy from `main` — only feature branches.
- If the build fails, fetch logs via `vercel logs <deployment>`.

## Verification
The healthz endpoint returns `{"status":"ok"}`.
Enter fullscreen mode Exit fullscreen mode

7.2 📁 Where skills live 

~/.hermes/skills/
├── devops/deploy-staging/
│   ├── SKILL.md              ← the file above
│   ├── references/           ← extra docs the skill can pull in
│   ├── templates/            ← file templates
│   ├── scripts/              ← helper scripts the agent can run
│   └── assets/               ← images, etc.
├── .hub/                     ← installed from skills hub
└── .bundled_manifest         ← what shipped with Hermes
Enter fullscreen mode Exit fullscreen mode

7.3 🔍 Progressive disclosure (3 levels)

This is what keeps token usage sane:

Level What loads When
0 name, description, category Always — in system prompt
1 full SKILL.md content When agent decides to use the skill
2 files in references/, scripts/ When the skill body says "see references/foo.md"

The agent calls a read_skill (or equivalent) tool to escalate from L0 to L1 to L2.

7.4 ⚡ Triggering

Three ways a skill activates:

  1. Slash command — user types /deploy-staging please ship #123.
  2. Natural language — "deploy this to staging"; the agent matches against L0 descriptions and pulls in L1.
  3. Programmatic — cron jobs explicitly attach skills.

7.5 🎛️ Conditional activation

Frontmatter fields gate visibility:

  • platforms: [linux] — hidden on macOS.
  • fallback_for_toolsets: [web] — only visible if no premium web tool is enabled (e.g., a DuckDuckGo skill that fills in when Brave Search isn't configured).
  • requires_toolsets: [shell] — hidden if shell tool disabled.

This makes the skill catalog adapt to the deployment.

7.6 🔁 Self-improvement: the skill_manage tool

The agent uses two complementary tools:

  • Read path: skills_list (browse Level-0 index) and skill_view (escalate to Level-1/2 content).
  • Write path: skill_manage, a meta-tool with sub-operations:
Action Effect
create New skill from scratch
patch Surgical text replacement (preferred for updates)
edit Full rewrite
delete Remove skill (restricted to user/agent-created skills — can't delete bundled ones)

Note: file management within a skill (references/, scripts/) goes through generic write_file / remove_file tools scoped to the skill's directory.

The SKILLS_GUIDANCE block in the system prompt explicitly nudges the agent to create a skill after:

  • Solving a task that took 5+ tool calls.
  • Finding a non-obvious workaround.
  • Discovering a workflow it might repeat.

Skill installation from the hub is user-driven only (security). The agent never installs untrusted skills on its own — it can only skill_manage create from its own experience.

This is the closed learning loop. The agent writes its own playbooks while it works.

7.7 🌐 Skills hub & sharing

Skills are portable markdown — they're trivially shareable. Hermes integrates with multiple sources (official/, skills-sh/, github/, well-known/, url, clawhub, lobehub). On install, each skill is security-scanned for prompt injection, data exfiltration, and destructive commands before being trusted. Trust tiers: builtin > official > community.

The format is the open agentskills.io standard — meaning skills written for Hermes work in other compatible agents.

💾 8. Memory System

Three independent mechanisms working together (the "3-layer" framing is a teaching simplification — in the code they're orthogonal):

🧊 Mechanism 1 — Frozen-snapshot persistent memory

Two markdown files, both agent-curated:

  • MEMORY.md — facts. "Project ships every Tuesday." "Test DB password is in 1Password vault X." (~2200 char cap)
  • USER.md — user model. "Prefers terse answers." "Senior Go engineer, new to React." (~1375 char cap)

A MemoryStore reads them once at session start and embeds them in the system prompt as a single immutable block (delimited by §). The agent can write to those files mid-session (and the writes go to disk), but the system prompt's copy doesn't change until next session. This is what keeps the prefix cache valid.

🗃️ Mechanism 2 — Cross-session recall via SessionDB

A SessionDB (SQLite, WAL mode, FTS5 full-text index) stores every prior conversation turn. On demand, the agent uses a session_search tool to query it; an LLM summarizer condenses hits into a paragraph that fits in context. Multi-process write contention is handled with BEGIN IMMEDIATE + a custom retry loop (20–150 ms jitter).

🔌 Mechanism 3 — Pluggable provider (Honcho / mem0 / supermemory)

This is a swap-in, not an additional layer. A single MemoryProvider ABC (agent/memory_provider.py); orchestration via agent/memory_manager.py. Lifecycle hooks: prefetch() (before model call), sync_turn() (after turn), shutdown().

Provider knobs that matter:

  • Recall mode: hybrid / context / tools. Tools-mode lets the model decide when to query; context-mode just injects relevant memories every turn.
  • Write frequency: async / turn / session / numeric (every N turns).

Honcho's "dialectic" deserves a note because it sounds mystical and isn't: it runs three sequential reasoning passes — Initial Assessment → Self-Audit → Reconciliation — depth controlled by dialecticDepth (1–3). It's effectively self-critique chained for higher-quality user modeling.

You only have one active provider at a time. Pick the right abstraction for your use case (Honcho for deep user modeling, mem0/supermemory for vector recall, none if files+FTS5 are enough).

🔌 9. Plugin System

A PluginManager discovers plugins from three places:

  1. ~/.hermes/plugins/ (user-level)
  2. ./.hermes/plugins/ (project-level)
  3. pip entry points (hermes.plugins)

Each plugin defines a register(ctx) function and can hook into lifecycle events:

  • pre_tool / post_tool
  • pre_llm / post_llm
  • session_start / session_end

…and can register new tools, new CLI commands, or replace memory providers.

Iron rule: plugins must NEVER modify core files. If a plugin needs something the framework doesn't expose, the framework grows a generic hook — not a special-case import. This keeps the plugin surface stable.

📋 9b. The COMMAND_REGISTRY Pattern (worth stealing)

A single COMMAND_REGISTRY constant in hermes_cli/commands.py is the source of truth for every slash command. From this one structure, the codebase auto-derives:

  • CLI dispatch
  • Gateway hooks (so /skill foo works in Telegram)
  • Telegram inline menu entries
  • Slack slash subcommands
  • prompt_toolkit autocomplete
  • /help text

Adding a new slash command is one new CommandDef entry plus a handler. Zero scattered edits. This is the same pattern as the tools registry, applied to UI commands. Steal it for your own build — it's how Hermes scales surface area without scaling maintenance.

🎨 9c. Skin Engine (theming as data)

YAML files in ~/.hermes/skins/ (with inheritance from default). One YAML controls 18 named colors, spinner faces and verbs, agent name and greeting/farewell, prompt symbols, tool emojis, ASCII banners with Rich markup. Ten built-in skins (default, daylight, mono, poseidon, charizard, …). Hermes Mod ships a web editor with live preview and image→ASCII conversion.

The takeaway is architectural: branding lives in YAML, not code. A user can fork the look without touching Python. This matters more than you'd think for an agent users live inside for hours.

📡 9d. Multimodal & Streaming

  • Vision: vision_analyze tool. Anthropic image-to-text fallback caching via _anthropic_image_fallback_cache (when a model can't see images natively, the cache

Visit Website