Personal Brain: My Journey with ACP in Obsidian

In my previous article, From MCP to ACP - A New Standard in AI Agent Interoperability, I explored the evolving landscape of AI agent protocols. This month, I put theory into practice by integrating the Gemini-CLI agent into my favorite note-taking tool, Obsidian.md. The result is a working MVP of a plugin I’m calling Personal Brain, which seamlessly embeds an AI agent into your second brain.

This article is divided into two parts. First, I’ll introduce the Personal Brain plugin. Second, I’ll share the lessons I learned while working with the Agent Client Protocol (ACP) in a real-world application.

Introducing Personal Brain for Obsidian #

Personal Brain brings the power of a dedicated AI agent directly into your Obsidian vault. My goal was to create a tool that moves beyond simple text generation and acts as a true assistant for knowledge work. Unlike many AI tools that act as simple chatbots, Personal Brain provides a full-fledged agent that can perform complex tasks autonomously.

Obsidian’s philosophy was a perfect match for this project. It is a privacy-focused, local-first application where your notes are stored as Markdown files on your machine. This aligns perfectly with the goal of creating a secure and user-controlled AI assistant. While the agent sends data to Gemini, users with a Google Workspace account benefit from data governance policies that ensure their prompts and generated text are not used for model training, as detailed in their data governance documentation. This makes the plugin compatible with many internal company data policies.

Here’s how Personal Brain can enhance your workflow:

• Focus on Ideas, Not Just Words: Stop staring at a blank page. You can create rough drafts and then ask the agent to refine them, using the full context of your notes and attached files.
• Rich Contextual Understanding: The agent isn’t limited to just the text in your current note. You can attach a wide range of materials to your chats, including local files (like PDFs and images), web search results, and content from any website. This allows the agent to generate comprehensive and well-informed content.
• Harnessing Powerful Tools: The power of AI agents originally designed for software engineers—like Gemini-CLI, Claude Code, and others—can now be applied to knowledge management. You can automate complex creative and organizational tasks, streamlining your workflow without needing any technical expertise.
• A Home for Your Prompts: Keep your tested and refined prompts directly within your Obsidian vault, ready to be deployed whenever you need them.

For example, you can ask the agent to summarize a lengthy PDF document or analyze the content of an image, all from within Obsidian.

Under the hood, the plugin is built with TypeScript and Svelte 5, and it uses the official Agent Client Protocol (ACP) TypeScript SDK to communicate with the Gemini-CLI agent. Interestingly, the Gemini-CLI agent itself was a valuable assistant throughout the development process.

Join the Closed Beta #

I am now preparing to launch a closed beta for the Personal Brain plugin to gather feedback and refine its features. If you are an Obsidian user and interested in exploring the future of AI-assisted knowledge management, I invite you to register for the beta.

Join Closed Beta Testing

Lessons Learned from Implementing ACP in Obsidian #

Choosing Obsidian for this project was a deliberate decision. Over the past few years, I’ve shifted nearly all of my project documentation to Markdown. It’s a versatile format that can be easily converted to Word documents or uploaded to Confluence, and with tools like Mermaid, I can even embed complex charts directly into my notes. This workflow evolved until Obsidian became the natural environment for my second brain.

At the same time, I was exploring the benefits of AI agents. I discovered that even though tools like Gemini-CLI and Claude-Code are presented as “coding agents,” they work exceptionally well with Markdown. An Obsidian vault is a natural environment for them, which opened up the possibility of empowering my second brain with agentic AI. However, this created a clumsy workflow. I was constantly switching between Obsidian and a terminal, manually ensuring files were saved and reloaded, and processing commands in a separate window. I never was sure which version of document agent works on. It was powerful, but far from seamless.

From my research for my previous blog post, I knew a better way had to be possible. What better way to get to know a new technology like ACP than by trying it yourself? Since Obsidian is an Electron-based application that can be easily programmed with TypeScript, and the Agent Client Protocol offers an official TypeScript SDK, I decided to build this plugin.

Here’s a breakdown of my experience working with ACP.

What Went Well #

• Accelerated Development: ACP provided a solid foundation, allowing me to focus on user experience rather than reinventing the wheel. The protocol handles the complex back-and-forth communication between the client and the agent.
• Simplified Authentication and Privacy: For users with Google Workspace accounts, authentication is seamless. There’s no need to manage API keys, and usage is covered under the existing subscription. Furthermore, the Gemini-CLI agent adheres to Google Workspace privacy policies, which is a critical consideration for any tool that handles professional data. This makes the plugin compatible with many internal company data policies.
• Agent Interoperability: As I hoped, the protocol-based approach means you aren’t locked into a single agent. It opens up a future where users can choose from a marketplace of agents to find the one that best suits their workflow. While most agents currently focus on code, this opens the door for new agents that could specialize in other areas, such as analyzing and improving written text like this very blog post.

The Gray Areas #

While the experience was largely positive, I encountered some areas where the protocol felt immature or restrictive, particularly around file system interactions.

• Restrictive File System Model: ACP’s file system capabilities are built on a foundation of absolute paths. This model is limiting, making it difficult to integrate with applications that store files in the cloud (e.g., Google Drive) and effectively preventing the use of a remote AI agent that doesn’t have direct access to your local machine.
• Lack of File Management Tools: The protocol currently lacks a documented, standardized way for an agent to perform basic file management tasks. For example, there is no clear way to inform an agent that a requested file doesn’t exist (the current workaround is to return an empty string), nor are there tools for moving, renaming, or deleting files, which are often performed by the agent executing terminal commands.
• Limited Binary File Support: While the protocol can attach binary files to a prompt, it doesn’t support reading or writing them as part of its direct file system operations.
• Underdeveloped Permissions Model: The permissions system still feels underdeveloped. A more robust model is needed to give the user clear control over the agent’s access.

Most of these issues could be addressed by introducing the concept of a client-side virtual file system. The idea is to allow ACP clients to register their own URI schemes (e.g., vault://) to signal that they will be handling all file operations. With the client acting as the gatekeeper, a much more granular and secure permissions model could be built, giving users full control over what the agent can see and modify. This approach would also provide abstract paths (solving the remote agent problem and allowing for creative file structures based on tags or categories) and create a natural home for a full suite of file management tools.

What Didn’t Work #

• Custom Tool Integration: I was initially optimistic about providing new tools to the agent directly from the ACP client, thinking the client could act as a relay for an MCP server. Unfortunately, the current implementation is limited to providing an MCP servers configuration when the agent session is created, and only if the agent supports it. For Gemini-CLI, the only solution is to provide MCP services directly in its configuration files.
• Inefficient Environment-Specific Tools: A direct consequence of the tool integration issue is inefficiency. Agents often resort to running terminal commands to get information that the client application already has. For example, an IDE client knows the project’s build status from the LSP, and Obsidian knows which notes are related through its graph. An agent that could use tools provided by the client would be faster, more efficient, and more deeply integrated.
• Flaky Audio Compatibility: While the Gemini-CLI agent reports that it has audio compatibility, this feature proved to be unreliable in practice. Across many attempts, I was only able to get it to successfully transcribe an audio file once. This suggests the issue likely lies with the agent’s specific implementation rather than a failure of the protocol itself.
• Agent “Personality”: There is no way to provide the agent with a “personality” or system prompt that sets the context for its environment. I would love to be able to inform the agent that it is operating within Obsidian and should, for example, use [[]] to link to other notes.

The Road Ahead for Personal Brain #

My immediate focus is on preparing for the closed beta and gathering user feedback. Looking further ahead, I have several goals for the Personal Brain plugin:

• Integrate MCP Servers: Allow users to configure and connect to their own MCP servers from within the plugin’s UI.
• Support for More Agents: Enable compatibility with other ACP-compliant agents as they become available.
• Streamlined Onboarding: Create a user-friendly setup process that makes it easy for anyone to get started with an AI agent, regardless of their technical background.

This journey has been both challenging and rewarding. I’m excited about the potential of Personal Brain to transform how we interact with our knowledge bases, and I look forward to shaping its future with the help of the community.