Skip to content

nv-mcp

View as Markdown

@niivue/nv-mcp lets an AI agent work the NiiVue viewer you have open in your browser. Ask Claude, Cursor or another agent to “go to the left insula” or “cut an axial plane and send me a screenshot”, and the viewer does it while you watch.

It speaks the Model Context Protocol (MCP), the standard way agents call outside tools. Use it to let people explore images by asking, to let an agent look at the viewer to check its own work, or to drive a viewer from a chat.

How it works

There are two parts:

  • A small server that runs on your machine under Bun. Agents connect to it at http://127.0.0.1:4242/mcp.
  • Your NiiVue page, which keeps a connection open to that server. Each request from the agent goes to the page, which changes the scene and answers.

The server keeps no images or scene of its own, so what the agent sees is exactly what the person sees. It needs no login, and by default it listens on your own machine only.

Install

Terminal window
npm install @niivue/nv-mcp@next @niivue/niivue@next

Whichever package manager you use, the server runs under Bun.

Your first agent-ready page

Start the server:

server.ts
import { startServer } from '@niivue/nv-mcp/server'
startServer({ name: 'my-viewer' })

Run it with bun server.ts. Then connect your page to it, next to the NiiVue code you already have:

import NiiVue from '@niivue/niivue'
import { AgentClient, coreHandlers, type NiiVueHost, sceneState } from '@niivue/nv-mcp/browser'
const nv = new NiiVue()
await nv.attachTo('gl1')
await nv.loadVolumes([{ url: '/volumes/mni152.nii.gz' }])
const host: NiiVueHost = { view: nv }
new AgentClient(coreHandlers(host), { state: () => sceneState(host) }).attach()

host describes your page to the package. The NiiVue instance is all it needs; Building your app shows what else you can give it. The page finds the server by itself, and keeps trying if you start the server second.

Finally, tell your agent where the server is. For Claude Code:

Terminal window
claude mcp add --transport http niivue http://127.0.0.1:4242/mcp

Now ask it to “show the three slices and the 3D render”, “use the hot colormap” or “cut away the top half and send me a screenshot”. Keep the page’s tab in front while it works: a hidden tab cannot draw.

Other agents connect the same way. See Connecting agents.

Trying the demo

The repository has a ready-made page and server. From the repository root:

Terminal window
bunx nx dev demo-mcp

It opens the MNI152 brain template and knows the AAL atlas, so you can ask for regions by name. Its source is in apps/demo-mcp.

What an agent can do

An agent can do most of what a person at the viewer can, and a few things they cannot:

  • Find its way around. Ask where the crosshair is, move it to a point, or step it one voxel at a time. When the page has an atlas, it can also list the regions and go to one by name.
  • Load images. Volumes, overlays, label maps with their own names and colours, meshes, tracts, connectomes, signals and saved NiiVue documents, from any address the page can fetch. OME-Zarr stores too, if the page offers them.
  • Change how things look. Colormaps, display ranges, opacity, the slice layout, the colorbar, or any other NiiVue setting by name.
  • Cut and turn. Clip planes named for the side they take away (“cut off the left side”), the camera angle, and smooth animated moves from one view to the next.
  • Look at the result. A screenshot of the canvas, so it can check its own work. It can also find the point under any pixel it sees.
  • Mark things up. Distance measurements, outlines and annotations on the slices, boxes on the 3D render, and text labels pinned to regions or points if your page draws them.
  • Measure and edit. Statistics of a volume, over the whole image or inside a mask or the drawing, and its histogram. It can also run NiiVue’s volume transforms or write new voxel values.
  • Build a control panel. Buttons, sliders, menus and other controls on the canvas, tied to settings, images or actions your page offers. The page has to host them.
  • Work with whole-slide images. Switch levels, focal planes and channel colours, and read pixel values.
  • Save. Have the browser download the scene, a volume, a mesh, the drawing or a picture.

An agent cannot open a file on your computer from its path. Instead it can put an “Open file” button on the canvas for you to click, if your page offers an action that opens files.

If you have the page open in several tabs, the agent can list them and pick the one to work in. It can also give you a link that opens the page in a new tab with an id of its own, and keep talking to that tab.

Where to go next

  • Connecting agents: Claude desktop, Codex, Gemini CLI, VS Code, Cursor, Windsurf or your own program, and what to check when a call fails.
  • Building your app: an atlas, labels, controls on the canvas, sample data, tools of your own, and serving the page from another address.
  • Source on GitHub.