openpencil/packages/docs/programmable/ai-chat.md

107 lines
10 KiB
Markdown
Raw Permalink Normal View History

---
title: AI Chat
2026-03-09 14:43:11 +00:00
description: Built-in AI assistant with 90+ tools for creating and modifying designs.
---
# AI Chat
Press <kbd>⌘</kbd><kbd>J</kbd> (<kbd>Ctrl</kbd> + <kbd>J</kbd>) to open the AI assistant. Describe what you want — it creates shapes, sets styles, manages layout, works with components, and analyzes your design.
## Setup
feat(ai): add guided AI setup for providers, coding agents, and Pi (#916) * feat(storybook): prototype guided AI setup and task assignments * refactor(storybook): adopt shared control foundations * refactor(storybook): build AI setup on current settings foundations Move the prototype to settings/ai-setup and compose SettingsSection, SettingsGroup, SettingsRow, AppAlert, AppBadge, and AppCheckbox instead of local section, status, and badge markup. Replace the nonexistent danger color and raw amber with the error and warning tokens. * feat(ui): add a shared radio group AppRadioGroup wraps the Reka radio group with typed options, labels each radio by its option text with any description as its accessible description, and supports all arrow keys unless an orientation is set. The AI setup wizard uses it for the spending choice, named by the step heading, and shares the choice card style with its checkboxes. * fix(ui): draw unchecked checkboxes on the field background AppCheckbox filled its box with the surface (text) color, so unchecked boxes were nearly black in the light theme and nearly white in the dark theme. Use the panel field background and accent hover border shared with the radio group and switch. * refactor(storybook): drop the simplified AI connections panel AI setup has two modes: skippable guided onboarding for most people and the existing advanced settings for power users, both editing the same model settings. Remove the third, simplified connections and tasks panel. The wizard's Advanced settings action and the returning-user screen now stand in for ModelsPanel, which offers Run guided setup. Removing Gateway's own Back button also fixes the blank screen it led to. * feat(ai): plan guided AI setup from the model catalog planOnboarding proposes design and vision models from the access a person already has, using the real provider and agent catalog, and falls back to OpenRouter only when pay-as-you-go is allowed. applyOnboardingPlan merges a confirmed plan into the model settings, reusing matching connections and profiles, keeping roles it was not asked about, and dropping only the empty fresh-install profile. * refactor(ai): share model provider display names Move the provider, agent, and Pi display-name lookup out of the model settings workflow so guided setup can reuse it. * feat(ai): offer guided AI setup over the real model settings Guided setup asks what AI should help with, what access the person already has, and whether pay-as-you-go models are allowed, then proposes design and vision models from the provider and agent catalog. Connections reuse the provider key field and connection test, keys are saved through the credential manager, and the confirmed plan is merged into the model settings, keeping anything configured by hand. Saving reports saved, partial, or failed like the profile editor. A fresh install whose model settings are still the empty placeholder is offered setup once with a skippable welcome; existing setups never see it. Settings → AI & agents can run it again, and Advanced settings hands off to the model editor. The Storybook fixtures for agents, OpenRouter sign-in, Vercel AI Gateway, and the local server are replaced by the real flow, with all copy translated. Browser tests start with the offer dismissed through the shared Playwright storage state; the first-run spec clears it. * fix(ai): keep configured models and credentials safe in guided setup Running guided setup again planned from the catalog defaults, so it replaced hand-configured design and vision models and dropped their settings; it now keeps a configured model while its access is still selected or onboarding cannot offer that provider, and keeps vision when nothing new covers it. Reused profiles must have the capabilities the plan relies on, and an explicit vision assignment without image input is cleared. A server's saved key and its status now apply only to the connection at the address being entered, and its connection test uses that connection's API type. Entered keys are copied before saving, so closing setup mid-save no longer drops them, and the Models list refreshes key status after setup saves a key. Servers that do not check keys get a hint to enter any value, and a step that only keeps configured models says so instead of showing nothing. * test(ai): check key status right after guided setup The Models list must show a key saved by guided setup as connected without reopening Settings. * feat(ai): sign in to OpenRouter from guided setup OpenRouter can now be connected with its OAuth PKCE flow instead of a pasted key. In the browser, sign-in opens in a popup that returns to a static callback page on the app's origin, which relays the redirect to the editor over a BroadcastChannel, so the editor never navigates away. The desktop app opens the system browser and receives the redirect on a one-shot 127.0.0.1 listener, the localhost callback OpenRouter documents. Either way the editor checks the state, exchanges the code for a key directly with OpenRouter, fills it in, and runs the connection test. Waiting, blocked pop-ups, cancellation, expiry, and failures are reported in the step, which keeps the pasted-key path. Setup no longer offers Clear for a saved key, since removing keys belongs to the advanced settings, and the service worker leaves /oauth/ pages to the network. * fix(settings): report unreadable model keys as unavailable A saved key the browser credential store could not read, for example one left from an older session on the same origin, rejected the model status refresh and the startup credential check, which surfaced as a global error toast. Each read failure now marks only that connection as unavailable. * feat(ai): map every role in guided setup and verify OpenRouter sign-in Guided setup now proposes a model for design, vision, review, and fast work, and the review step is a map of those roles with every suitable model from the connected providers and a "Use recommended setup" shortcut. Fast work defaults to the provider's catalog model tagged as fast; behind an agent, review and fast work use the API model chosen for vision. Review and fast work are never asked about, so a configured choice, including none, stays unless it follows a design model it can no longer follow. The pay-as-you-go question only appears when the access already selected leaves a requested role without a model, so choosing OpenRouter or another account no longer asks it. After signing in with OpenRouter, setup checks the key with OpenRouter's key endpoint, which costs no credits, instead of a text generation test. The step then says it is signed in, names the key, warns when the account has no credits yet, and offers another account in place of the key field and test button. * feat(ai): offer OpenRouter only for goals nothing selected covers The separate pay-as-you-go step asked an abstract question even when the selected access already covered every goal. The connect step now names a goal nothing selected can cover, such as visual review behind a coding agent or a local server, and offers to add OpenRouter for it; once added, it says OpenRouter fills the gap and can be removed again. Setup can finish without visual review, but not without a design model. A local or company server can be marked as able to read images, which lets it cover visual review and makes it the preferred vision model over a paid account. * feat(ai): show provider logos and more providers in guided setup Guided setup shows monochrome logos for coding agents, API accounts, and local servers, from LobeHub's MIT-licensed static SVG set loaded as an `ai` icon collection, so they follow the theme like Lucide icons. DeepSeek, Z.ai, and MiniMax are offered under "More providers", and a local server can start from the Ollama or LM Studio address instead of typing it. * feat(ai): guide coding agent setup in guided setup Choosing Claude Code, Codex, or Gemini CLI in the desktop app now checks whether the agent's ACP program and OpenPencil's MCP server, which agents use to reach the canvas, are installed. An allowlisted agent_lookup command finds the program on the same widened PATH as the MCP lookup. The card shows install commands only for what is missing, checks again on request, links a new setup guide, and copies a prompt that asks an agent the person already uses to install both, confirm they are on PATH, and sign in. In the browser, the agent section links to the desktop app. * fix(ai): space the More providers toggle like a group heading The toggle sat flush against the account cards above and below it; it now reads as a group heading with the same rhythm as the other sections. * feat(ai): detect and install coding agents in guided setup Adopt the local agent discovery from #847. A desktop agent_lookup command reports each agent's own CLI, its ACP adapter, npm, and OpenPencil's MCP server on the widened PATH without starting any of them, and the app can install a missing adapter or the MCP server with npm, limited by the shell capability to those exact packages and the MCP version that matches the app. Guided setup now tells "installed but the OpenPencil adapter is missing" apart from "not installed", offers one-click installs, links each vendor's own setup guide, and keeps the manual commands and setup prompt for the browser, missing npm, or a failed install. Codex install instructions move to @agentclientprotocol/codex-acp, which replaces @zed-industries/codex-acp. Kiro CLI support from the same pull request is left for a separate change, since it needs ACP transport work. Co-authored-by: GitttHomie <134371845+GitttHomie@users.noreply.github.com> * build(app): resolve LobeHub icons with import.meta.resolve The architecture lint forbids createRequire in ESM build code. * test(app): seed AI setup specs through storageState Follows the storage seeding used by other browser specs and the import type rule. * feat(ai): set up Pi with its own sign-ins in guided setup Pi now runs with the providers signed in to in the Pi CLI and Pi's default model. The Harness companion reuses ~/.pi/agent; the app reads only Pi's settings.json for the default model and never auth.json. A saved key is still used as an AI Gateway key. Guided setup offers Pi next to the other coding agents. On the desktop it checks the Harness companion, the MCP server, and Pi's default model, and installs the companion with one click through npm. Agent discovery now reads the installed versions of the MCP server and the Harness companion from their package.json without starting them. Setup flags a version that does not match the app and shows the update command for the package manager that installed it, instead of reporting the server as installed and failing at the first message. * feat(ai): check agent companions before a chat starts A Pi chat without the Harness companion, or any agent chat whose companion or MCP server version does not match the app, failed when the process started and showed only the generic request error. The chat now checks the companions through agent discovery first and names the fix, with an action that opens guided setup. Pi sign-in and model problems use the same path. The Pi model editor shows the same companion, MCP server, and default model status as guided setup, and no longer requires a model ID, since Pi falls back to the default model set in Pi. Supersedes the companion detection in #566, which ran the companion to read its version and required an exact version match. * fix(harness): start Pi sessions with MCP tools and keep unsent messages Pi chats in the desktop app always configure OpenPencil's MCP server, and three companion problems stopped them: - @ai-sdk/harness-pi imports pi-mcp-adapter, which publishes TypeScript sources. Node refuses to strip types under node_modules, so the companion now strips them through a module load hook limited to TypeScript dependencies. Bun runs them as is. - pi-mcp-adapter imports @earendil-works/pi-tui, declared only as an optional peer, so npm left it out. The companion depends on it at the version pi-coding-agent uses. - Pi reports live-process resume, yet the service handed it state saved by an earlier session, and the just-bash sandbox cannot resume, so every later session with that ID failed. Live-process backends now start fresh and drop saved state. When a chat cannot start, the composer now keeps the typed message instead of discarding it. * fix(harness): keep companion stdout for protocol messages Pi prepares the packages listed in a person's Pi settings with npm, which inherits the companion's stdout, and libraries log through console.log. Both landed in the JSONL protocol stream, where the app discarded them with warnings. The companion now keeps the real stdout for protocol messages, sends other stdout writes to stderr, and quiets npm on success through its environment. The type-stripping hook no longer prints Node's experimental warning, and the app logs companion stderr as diagnostics rather than errors, since failures arrive as protocol errors. Document Pi in the coding agents guide, the AI chat page, and the README: guided setup installs the companion, Pi uses the Pi CLI's sign-ins and default model, an AI Gateway key is optional, and the companion needs Node.js 22.15 or later. * test(harness): keep the pi-tui pin in step with pi-coding-agent The companion depends on pi-tui only because pi-mcp-adapter imports it while declaring it optional (nicobailon/pi-mcp-adapter#805). Upgrading @ai-sdk/harness-pi moves pi-coding-agent, and a pin left behind would make npm install a second, mismatched pi-tui. The test fails until the pin matches. * test(ai): follow the model catalog in guided setup tests The plan and apply tests repeated catalog default and fast model IDs, so master's model update broke them without any change in setup behavior. They now read those models from the catalog. * test(ai): keep the model catalog helper with the shared test helpers Unit test homes under tests/app accept only *.test.ts files, so the onboarding tests' catalog helper moves to tests/helpers/ai. * docs: tighten the guided setup and Pi changelog entries Name every provider and server preset guided setup offers, describe the role step as it now works, and shorten the Pi entry. * test(ai): type the guided setup test stubs for the test typecheck Master now typechecks the test suites: fetch fakes go through fetchStub, the chat ref is shallow like the real one, and mocks declare the arguments the tests inspect. * test: type the tabs module in the closed-documents spec The spec imported the tabs module by its served URL without a type, which fails the test type check on master. * test(ai): assert outcomes instead of copy in guided setup tests Drop the setup-prompt test, which checked prompt prose, the onboarding wrapper cases that restated discovery, and the coversGoals case. Story plays and the OpenRouter E2E flow now assert controls and saved models instead of sentences and catalog model names, and the fast-model helper checks the planned model's catalog entry instead of recomputing the choice. tests/AGENTS.md states the rule. * feat(ai): return desktop OpenRouter sign-in through a deep link The desktop app ran a hand-written HTTP server on a localhost port to receive OpenRouter's redirect, and OpenRouter labels apps with a localhost callback by host and port. OpenRouter now redirects to a page on the web app that opens openpencil://oauth/openrouter with the same query, and the desktop shell forwards that link to the webview as an oauth-callback event. The attempt that started sign-in checks the state and exchanges the code with its PKCE verifier, which never leaves the app. * feat(ai): ask OpenPencil's companions for their version The desktop app read a companion's version by following its executable's symlink up to a package.json. That only worked for the Unix npm and bun layouts: Windows .cmd and .exe shims and version-manager shims such as Volta and mise are not links into the package, so the version was always unknown and an outdated companion went unreported. The MCP server, stdio bridge, and Harness companion now print their version for --version, and the app runs each installed one with --version --help under a timeout. A release older than --version prints its help or exits without a version line, which reads as outdated. A bun global install on Windows now gets the bun update command too. * fix(ai): ask for a Pi sign-in when Pi has none readPiAccount returned an account whenever a home folder existed, so a chat with no Pi sign-in reached the Harness and failed with a provider error instead of the guided pi-sign-in fix. It now reports whether Pi's auth.json exists, without reading it, and the capability allows that one check. * fix(ai): keep the attachments of a message that was not sent A message that never reached the chat came back to the composer as text only: its image previews were revoked and its referenced layers dropped. The composer now takes back the whole submission, or releases the previews when newer text replaced it. A message counts as sent once the chat holds it, so a failure after that no longer hands it back to be sent twice. * refactor(app): read the app version from one constant Four modules each derived the app version from the build define with the same test fallback. * refactor(ai): report chat submission errors from their own module Reverting turns from master and keeping unsent drafts together took useChatSubmission past the composition-root limit. The test for reverted turns now passes the setup messages the submission reports. * refactor(app): keep the app version with the runtime config Tools typecheck src/constants.ts through app imports without the Vite defines, so the version constant moves to src/app/runtime/version.ts. * test(desktop): check npm installs of the companions at the app's release version The scope test named the companion packages and version literally, so it would keep passing if the app requested something else. It now builds them from the app's package names and the release version a build embeds. * refactor(ai): parse OpenRouter, Pi, and sign-in callback data with Valibot The OpenRouter key info and code exchange checked their JSON with typeof chains, Pi's settings parsed JSON in a try before validating it, and the desktop sign-in trusted the shell's callback payload as typed. Each now goes through one schema. * fix(ai): take back a message whose images could not be prepared A message with images appears in the chat before its images are prepared, so a preparation failure counted as sent: the draft did not come back and its previews were already revoked. A message now counts as sent once it is dispatched; a failure before that removes the shown message and hands the draft back, and the composer's previews are revoked only after dispatch. * fix(ai): restore an unsent message only in its own conversation Switching conversations while a message was being sent could restore it into the newly opened one. The draft now comes back only if the conversation is unchanged, and its previews are released otherwise. * refactor(app): keep one app version constant The update window added an APP_VERSION to src/constants.ts beside the one in src/app/runtime/version.ts. The tools typecheck reaches src/constants.ts without the Vite defines, so the update window now reads the runtime one. --------- Co-authored-by: GitttHomie <134371845+GitttHomie@users.noreply.github.com>
2026-10-07 08:46:57 +00:00
The first time you open OpenPencil, guided setup asks what AI should help with and what you already use: a [coding agent](./coding-agents) such as Claude Code, Codex, Gemini CLI, or Pi in the desktop app, an API account, or a local or company server. It connects that access and assigns it to the **Design agent** and **Vision** roles, keeping anything you configured by hand. With OpenRouter you can sign in instead of pasting an API key. Skip it to start designing, and run it again from **Settings → AI & agents → Run guided setup**. To configure models by hand:
1. Open the AI chat panel (<kbd>⌘</kbd><kbd>J</kbd>)
2. Click the settings icon
3. Add a model and configure its provider, model ID, credentials, and capabilities
4. Save the model and assign it to **Design agent**
You can configure multiple reusable models and separately assign models for design work, reviews, fast tasks, and image input. Models using the same provider connection reuse its stored credential.
The chat composer grows with multiline prompts and can pin the current canvas selection as explicit node context. Assistant messages show provider reasoning in collapsible sections and provide a per-response copy action. Image attachments remain available for visual references when a Vision model is configured. Streaming responses use a hardened Markdown renderer with Shiki-highlighted code blocks; unsafe link protocols and embedded data images are blocked.
feat(settings): configure tool access, MCP failures, and step limits * feat(settings): configure tool access and agent step limits Built-in AI exposed only a hardcoded subset of the tool registry, and the maximum agent steps was a constant, so users could neither enable extended tools such as create_component nor adjust long-running tasks. Built-in AI and the local MCP server now keep independent, locally saved tool permissions over one shared catalog, with searchable read-only and side-effect groups and per-target defaults. Chat settings gain a validated maximum-steps field whose captured value drives the stop condition, remaining-step warnings, and limit detection for each message. Tool access, the local server, browser access, and MCP connections are grouped under a single Automation settings page. Closes #573 Closes #584 * refactor(settings): split automation into MCP and Tool access pages The Automation page mixed a permission matrix with server endpoints behind a Tools/Connections switch, and the view switch was indistinguishable from the provider switch. The nested scroll region showed three of 110 tools. Rename the MCP-facing page to MCP and give tool permissions their own Tool access page. The page owns a fixed toolbar for the target, count, defaults, and search, so the list uses the full dialog body and no row is clipped. * fix(automation): explain MCP startup failures with localized guidance Every startup failure collapsed into "MCP server did not become healthy": the spawn layer recorded the real error but the runtime discarded it, and health probes could not distinguish a rejected token from a missing server. The message also surfaced raw English text as the alert heading. Classify failures by reason (not installed, denied command, early exit, startup timeout, rejected token, unexpected response, unreachable) and render translated heading and guidance from the catalog, keeping captured stderr or HTTP status as labeled diagnostic detail. * refactor(ui): share one collapsible disclosure primitive Six features each wired Reka's collapsible with their own motion classes and one settings-only theme token, so the same interaction drifted in spacing, icon size, and reduced-motion handling. Add AppCollapsible with a family theme and move the settings disclosure and the model editor's advanced settings onto it. Chat and frame-preset call sites keep their distinct visuals for a follow-up. * fix(automation): explain MCP failures with localized details The failure alert carried raw English error text as its heading, and the diagnostic payload sat in a sibling block outside the alert with no relationship to it. Classify failures by reason, render translated heading and guidance from the catalog, and keep the payload in a collapsible inside the alert, which unmounts while collapsed so the live region announces only the summary. Add a copy action for issue reports. Find the executable where a graphical launch can: extend PATH with the common global bin directories before the lookup and report the searched directories as diagnostic detail. * fix(automation): keep MCP failure details out of reasons already explained An unreachable address and a rejected token already name their cause in the translated guidance, so repeating it under Details added noise. Details now carry only output the summary cannot: stderr, HTTP status, or an unknown error message. * test(settings): browse every MCP failure reason in Storybook The failure copy lived inside the settings panel, so reviewing the eight reasons meant reproducing each failure and the mapping could only be checked through the panel's dependencies. Extract MCPFailureAlert, which owns the reason-to-copy mapping, detail visibility, copy action, and restart action, and add a story covering every reason plus the collapsed-details behavior. * fix(ui): order alert details above the recovery actions The alert rendered its action buttons before the details slot, so the collapsible explanation of a failure appeared under the controls it explains. Details now render directly after the description. * fix(automation): correct MCP failure classification and detail Review follow-ups on the failure diagnostics. Only 401 and 403 mean the server refused our token; any other status now reports an unexpected response instead of telling the user to replace a token that was never the problem. The install hint rendered the whole diagnostic detail as its package argument, so searched directories appeared inside the install command. The install target is now a domain constant and the searched directories stay as detail, which not-installed failures surface again since they are the actionable desktop diagnostic. Exited failures also record the process exit code and signal so copied diagnostics stay conclusive when stderr is empty. The bundled PATH test now covers the append branch instead of only the unchanged path. * feat(settings): accept custom values for presets and retention Retention was a closed set of three counts while the AI step limit was a free number, so two bounded numeric preferences looked and behaved differently for no product reason. Add a shared preset-or-custom field: presets stay one click, the escape hatch reveals a validated numeric field, and the model carries only the resolved number. Diagnostics retention becomes a bounded number (50 to 20,000) with the presets as shortcuts, and the hardcoded revalidation in the panel is replaced by one domain resolver. * fix(settings): label the preset and custom fields Replacing the labeled provider field with the shared control left the AI step limit as a bare select with a detached hint paragraph, outside the settings group, so nothing on screen said what the number meant. The accessibility name came from aria-label, which is why behavior tests passed while the panel was unreadable. Move both controls into labeled settings rows with their descriptions, and give the revealed field its own accessible name so the two controls in one row differ. The specs now assert the control lives inside the row that names it, which is the check that would have caught this. * fix(mcp): allow the desktop app origin by default A server started manually bound the port and answered curl but the app webview could not use it: no CORS origin was configured, so the browser blocked every fetch and the app reported the server as unhealthy. The workaround required an undocumented environment variable. Allow the desktop app origins by default, accept a comma-separated override, and document the default in the CLI help and the security notes. Authenticated requests still need the bearer token, and browsers set Origin themselves, so only the app webview can present these origins. * fix(settings): address review findings on the new controls Copy details awaited nothing and confirmed the copy before the write finished. VueUse never rejects and falls back to a legacy write, so the await is what makes the confirmation honest rather than an error branch. The preset field only left custom mode when a preset arrived; a non-preset value assigned from the owner left the select showing a value absent from its options with the field still hidden. The watcher now follows the model in both directions. The story play functions queried the revealed field by the row label, which Testing Library matches as a whole string, so those interactions could not find it. The Storybook smoke assertion also assumed a button or tab, which skipped every story built from other primitives.
2026-09-17 20:55:58 +00:00
## Step limit
In **Settings → AI & agents → Chat**, set **Maximum steps per message** to a whole number from 1 to 1,000. The default is 50. Press Enter or leave the field to save a valid value; invalid drafts do not replace the saved preference. Higher limits allow longer tool-driven tasks but can increase latency and provider cost.
The built-in AI captures this limit when each message starts. Stopping, remaining-step warnings, and the **Continue** action use that same budget. Changing it does not interrupt an ongoing request; the next message or continuation uses the new limit. A step is one model iteration and can include multiple tool calls. ACP and Pi agents manage their own limits.
## Tool access
Open **Settings → Tool access** to choose which tools direct AI model connections can use. Search by name or description, expand read-only or side-effect groups, and toggle individual tools or an entire group. Group switches affect all tools in that group, not only search results. **Restore defaults** restores the compact default tool set; extended tools such as `create_component` can be enabled individually.
Preferences are saved locally and apply to the next message, including in an existing conversation. They do not change an already-running request. Enabling many tools increases the schemas sent to the model.
The **Local MCP** segment has independent settings for clients connected to OpenPencil's MCP server, including ACP and Pi agents. Restart the server and reconnect stdio clients after changing those settings. Remote MCP connections, WebMCP access, and Pi's shell/filesystem permissions remain separate.
Tool toggles control which tools are offered, not which operations scripts may perform. An enabled `eval` or other script-capable tool can perform design operations whose dedicated tools are disabled; these switches are not a sandbox.
## Saved Conversations
Use **Conversation history** to return to a saved chat, start a **New chat**, or rename or delete a conversation. History and attachment previews are stored locally; **All chats** lets you browse transcripts from other documents.
A conversation belonging to another document is read-only until you open that document. A saved agent transcript is not a guarantee that its external agent session can resume: when resumption is unavailable, start a new chat. Local history is not cloud synchronization or a backup.
In the Chat settings beside the model overview, choose whether reasoning is **Collapsed by default**, **Expand while thinking**, or **Expanded by default**. Disclosure animations follow the app's reduced-motion preference. Expanding older reasoning does not force the conversation to scroll to the bottom.
## Supported Providers
2026-03-09 14:43:11 +00:00
| Provider | Models | Setup |
| ------------------------ | ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| **OpenRouter** | Claude, GPT, Gemini, DeepSeek, Qwen, and others | API key from [openrouter.ai](https://openrouter.ai) |
| **Anthropic** | Claude Sonnet 4.6, Claude Opus 4.6 | API key from [console.anthropic.com](https://console.anthropic.com) |
| **OpenAI** | GPT-5.3 Codex, GPT-4.1, o3, o4-mini | API key from [platform.openai.com](https://platform.openai.com) |
| **Google AI** | Gemini 3.1 Pro, Gemini 3 Flash | API key from [aistudio.google.dev](https://aistudio.google.dev) |
| **Z.ai** | GLM-5.1, GLM-5, GLM-4.7, GLM-4.5 family | API key from [docs.z.ai](https://docs.z.ai/devpack/quick-start) |
| **MiniMax** | MiniMax M3, M2.7, M2.7-highspeed, M2.5, M2.1 | API key from [platform.minimax.io](https://platform.minimax.io/user-center/basic-information/interface-key) |
| **OpenAI-compatible** | Any endpoint with OpenAI API format | Custom base URL + key. Supports Completions and Responses API toggle. |
| **Anthropic-compatible** | Any endpoint with Anthropic API format | Custom base URL + key |
2026-03-09 14:43:11 +00:00
No backend, no subscription — your key talks directly to the provider. Browser requests are subject to each provider's CORS policy, and model deployments vary in how reliably they stream tool calls. See [BYOK provider and model compatibility](./byok-provider-compatibility) for measured results and reproduction steps.
## External MCP connections
feat(settings): configure tool access, MCP failures, and step limits * feat(settings): configure tool access and agent step limits Built-in AI exposed only a hardcoded subset of the tool registry, and the maximum agent steps was a constant, so users could neither enable extended tools such as create_component nor adjust long-running tasks. Built-in AI and the local MCP server now keep independent, locally saved tool permissions over one shared catalog, with searchable read-only and side-effect groups and per-target defaults. Chat settings gain a validated maximum-steps field whose captured value drives the stop condition, remaining-step warnings, and limit detection for each message. Tool access, the local server, browser access, and MCP connections are grouped under a single Automation settings page. Closes #573 Closes #584 * refactor(settings): split automation into MCP and Tool access pages The Automation page mixed a permission matrix with server endpoints behind a Tools/Connections switch, and the view switch was indistinguishable from the provider switch. The nested scroll region showed three of 110 tools. Rename the MCP-facing page to MCP and give tool permissions their own Tool access page. The page owns a fixed toolbar for the target, count, defaults, and search, so the list uses the full dialog body and no row is clipped. * fix(automation): explain MCP startup failures with localized guidance Every startup failure collapsed into "MCP server did not become healthy": the spawn layer recorded the real error but the runtime discarded it, and health probes could not distinguish a rejected token from a missing server. The message also surfaced raw English text as the alert heading. Classify failures by reason (not installed, denied command, early exit, startup timeout, rejected token, unexpected response, unreachable) and render translated heading and guidance from the catalog, keeping captured stderr or HTTP status as labeled diagnostic detail. * refactor(ui): share one collapsible disclosure primitive Six features each wired Reka's collapsible with their own motion classes and one settings-only theme token, so the same interaction drifted in spacing, icon size, and reduced-motion handling. Add AppCollapsible with a family theme and move the settings disclosure and the model editor's advanced settings onto it. Chat and frame-preset call sites keep their distinct visuals for a follow-up. * fix(automation): explain MCP failures with localized details The failure alert carried raw English error text as its heading, and the diagnostic payload sat in a sibling block outside the alert with no relationship to it. Classify failures by reason, render translated heading and guidance from the catalog, and keep the payload in a collapsible inside the alert, which unmounts while collapsed so the live region announces only the summary. Add a copy action for issue reports. Find the executable where a graphical launch can: extend PATH with the common global bin directories before the lookup and report the searched directories as diagnostic detail. * fix(automation): keep MCP failure details out of reasons already explained An unreachable address and a rejected token already name their cause in the translated guidance, so repeating it under Details added noise. Details now carry only output the summary cannot: stderr, HTTP status, or an unknown error message. * test(settings): browse every MCP failure reason in Storybook The failure copy lived inside the settings panel, so reviewing the eight reasons meant reproducing each failure and the mapping could only be checked through the panel's dependencies. Extract MCPFailureAlert, which owns the reason-to-copy mapping, detail visibility, copy action, and restart action, and add a story covering every reason plus the collapsed-details behavior. * fix(ui): order alert details above the recovery actions The alert rendered its action buttons before the details slot, so the collapsible explanation of a failure appeared under the controls it explains. Details now render directly after the description. * fix(automation): correct MCP failure classification and detail Review follow-ups on the failure diagnostics. Only 401 and 403 mean the server refused our token; any other status now reports an unexpected response instead of telling the user to replace a token that was never the problem. The install hint rendered the whole diagnostic detail as its package argument, so searched directories appeared inside the install command. The install target is now a domain constant and the searched directories stay as detail, which not-installed failures surface again since they are the actionable desktop diagnostic. Exited failures also record the process exit code and signal so copied diagnostics stay conclusive when stderr is empty. The bundled PATH test now covers the append branch instead of only the unchanged path. * feat(settings): accept custom values for presets and retention Retention was a closed set of three counts while the AI step limit was a free number, so two bounded numeric preferences looked and behaved differently for no product reason. Add a shared preset-or-custom field: presets stay one click, the escape hatch reveals a validated numeric field, and the model carries only the resolved number. Diagnostics retention becomes a bounded number (50 to 20,000) with the presets as shortcuts, and the hardcoded revalidation in the panel is replaced by one domain resolver. * fix(settings): label the preset and custom fields Replacing the labeled provider field with the shared control left the AI step limit as a bare select with a detached hint paragraph, outside the settings group, so nothing on screen said what the number meant. The accessibility name came from aria-label, which is why behavior tests passed while the panel was unreadable. Move both controls into labeled settings rows with their descriptions, and give the revealed field its own accessible name so the two controls in one row differ. The specs now assert the control lives inside the row that names it, which is the check that would have caught this. * fix(mcp): allow the desktop app origin by default A server started manually bound the port and answered curl but the app webview could not use it: no CORS origin was configured, so the browser blocked every fetch and the app reported the server as unhealthy. The workaround required an undocumented environment variable. Allow the desktop app origins by default, accept a comma-separated override, and document the default in the CLI help and the security notes. Authenticated requests still need the bearer token, and browsers set Origin themselves, so only the app webview can present these origins. * fix(settings): address review findings on the new controls Copy details awaited nothing and confirmed the copy before the write finished. VueUse never rejects and falls back to a legacy write, so the await is what makes the confirmation honest rather than an error branch. The preset field only left custom mode when a preset arrived; a non-preset value assigned from the owner left the select showing a value absent from its options with the field still hidden. The watcher now follows the model in both directions. The story play functions queried the revealed field by the row label, which Testing Library matches as a whole string, so those interactions could not find it. The Storybook smoke assertion also assumed a button or tab, which skipped every story built from other primitives.
2026-09-17 20:55:58 +00:00
Desktop ACP agents can also use trusted remote [Model Context Protocol](https://modelcontextprotocol.io/) servers. In **Settings → MCP**, under MCP connections, add a named Streamable HTTP endpoint, optionally save a bearer token, and enable the connection. OpenPencil stores the token in the configured credential backend rather than ordinary settings and resolves it only when starting the ACP session.
Remote servers must use HTTPS. Loopback HTTP endpoints are accepted for local development. Review and trust a server before enabling it: its tools may read external data or perform actions with the credentials you provide. OpenPencil's built-in design MCP server remains attached automatically and does not need to be added here.
## What It Can Do
feat(settings): configure tool access, MCP failures, and step limits * feat(settings): configure tool access and agent step limits Built-in AI exposed only a hardcoded subset of the tool registry, and the maximum agent steps was a constant, so users could neither enable extended tools such as create_component nor adjust long-running tasks. Built-in AI and the local MCP server now keep independent, locally saved tool permissions over one shared catalog, with searchable read-only and side-effect groups and per-target defaults. Chat settings gain a validated maximum-steps field whose captured value drives the stop condition, remaining-step warnings, and limit detection for each message. Tool access, the local server, browser access, and MCP connections are grouped under a single Automation settings page. Closes #573 Closes #584 * refactor(settings): split automation into MCP and Tool access pages The Automation page mixed a permission matrix with server endpoints behind a Tools/Connections switch, and the view switch was indistinguishable from the provider switch. The nested scroll region showed three of 110 tools. Rename the MCP-facing page to MCP and give tool permissions their own Tool access page. The page owns a fixed toolbar for the target, count, defaults, and search, so the list uses the full dialog body and no row is clipped. * fix(automation): explain MCP startup failures with localized guidance Every startup failure collapsed into "MCP server did not become healthy": the spawn layer recorded the real error but the runtime discarded it, and health probes could not distinguish a rejected token from a missing server. The message also surfaced raw English text as the alert heading. Classify failures by reason (not installed, denied command, early exit, startup timeout, rejected token, unexpected response, unreachable) and render translated heading and guidance from the catalog, keeping captured stderr or HTTP status as labeled diagnostic detail. * refactor(ui): share one collapsible disclosure primitive Six features each wired Reka's collapsible with their own motion classes and one settings-only theme token, so the same interaction drifted in spacing, icon size, and reduced-motion handling. Add AppCollapsible with a family theme and move the settings disclosure and the model editor's advanced settings onto it. Chat and frame-preset call sites keep their distinct visuals for a follow-up. * fix(automation): explain MCP failures with localized details The failure alert carried raw English error text as its heading, and the diagnostic payload sat in a sibling block outside the alert with no relationship to it. Classify failures by reason, render translated heading and guidance from the catalog, and keep the payload in a collapsible inside the alert, which unmounts while collapsed so the live region announces only the summary. Add a copy action for issue reports. Find the executable where a graphical launch can: extend PATH with the common global bin directories before the lookup and report the searched directories as diagnostic detail. * fix(automation): keep MCP failure details out of reasons already explained An unreachable address and a rejected token already name their cause in the translated guidance, so repeating it under Details added noise. Details now carry only output the summary cannot: stderr, HTTP status, or an unknown error message. * test(settings): browse every MCP failure reason in Storybook The failure copy lived inside the settings panel, so reviewing the eight reasons meant reproducing each failure and the mapping could only be checked through the panel's dependencies. Extract MCPFailureAlert, which owns the reason-to-copy mapping, detail visibility, copy action, and restart action, and add a story covering every reason plus the collapsed-details behavior. * fix(ui): order alert details above the recovery actions The alert rendered its action buttons before the details slot, so the collapsible explanation of a failure appeared under the controls it explains. Details now render directly after the description. * fix(automation): correct MCP failure classification and detail Review follow-ups on the failure diagnostics. Only 401 and 403 mean the server refused our token; any other status now reports an unexpected response instead of telling the user to replace a token that was never the problem. The install hint rendered the whole diagnostic detail as its package argument, so searched directories appeared inside the install command. The install target is now a domain constant and the searched directories stay as detail, which not-installed failures surface again since they are the actionable desktop diagnostic. Exited failures also record the process exit code and signal so copied diagnostics stay conclusive when stderr is empty. The bundled PATH test now covers the append branch instead of only the unchanged path. * feat(settings): accept custom values for presets and retention Retention was a closed set of three counts while the AI step limit was a free number, so two bounded numeric preferences looked and behaved differently for no product reason. Add a shared preset-or-custom field: presets stay one click, the escape hatch reveals a validated numeric field, and the model carries only the resolved number. Diagnostics retention becomes a bounded number (50 to 20,000) with the presets as shortcuts, and the hardcoded revalidation in the panel is replaced by one domain resolver. * fix(settings): label the preset and custom fields Replacing the labeled provider field with the shared control left the AI step limit as a bare select with a detached hint paragraph, outside the settings group, so nothing on screen said what the number meant. The accessibility name came from aria-label, which is why behavior tests passed while the panel was unreadable. Move both controls into labeled settings rows with their descriptions, and give the revealed field its own accessible name so the two controls in one row differ. The specs now assert the control lives inside the row that names it, which is the check that would have caught this. * fix(mcp): allow the desktop app origin by default A server started manually bound the port and answered curl but the app webview could not use it: no CORS origin was configured, so the browser blocked every fetch and the app reported the server as unhealthy. The workaround required an undocumented environment variable. Allow the desktop app origins by default, accept a comma-separated override, and document the default in the CLI help and the security notes. Authenticated requests still need the bearer token, and browsers set Origin themselves, so only the app webview can present these origins. * fix(settings): address review findings on the new controls Copy details awaited nothing and confirmed the copy before the write finished. VueUse never rejects and falls back to a legacy write, so the await is what makes the confirmation honest rather than an error branch. The preset field only left custom mode when a preset arrived; a non-preset value assigned from the owner left the select showing a value absent from its options with the field still hidden. The watcher now follows the model in both directions. The story play functions queried the revealed field by the row label, which Testing Library matches as a whole string, so those interactions could not find it. The Storybook smoke assertion also assumed a button or tab, which skipped every story built from other primitives.
2026-09-17 20:55:58 +00:00
The configurable tool catalog covers these categories; the tools offered to a model depend on your Tool access settings:
- **Create** — frames, shapes, text, components, pages. Renders JSX for complex layouts.
- **Style** — fills, strokes, effects, opacity, corner radius, blend modes.
2026-03-09 14:43:11 +00:00
- **Layout** — auto-layout, grid, alignment, spacing, sizing.
feat: behaviours and preview mode (#893) * feat: author behaviours on main components A main component or component set can behave as a Switch, Checkbox, Slider, or Tabs, after Reka UI's primitives. The behaviour lives in OpenPencil plugin data: boolean values bind to variant or boolean properties with the values meaning on and off, a number keeps its own range since Figma has no number property, and the control's subcomponents bind to the component's slots. A Behaviour section in the properties panel adds, binds, and removes it, each as one undo step, and flags required bindings that are missing. The canvas-only layout's pill becomes a component that preview will reuse. * feat: preview instances with behaviours on the canvas View > Preview (Cmd+Alt+Enter) puts the canvas in preview: a lone canvas switches to the canvas-only layout with a Previewing pill, and a split canvas previews on its own side. Clicking a Switch or Checkbox flips it, dragging a Slider moves its thumb and range, and clicking a Tabs trigger shows its panel. Preview keeps its state on copies of the instances it touched, in a private graph with the document's ids, and the canvas draws those copies in place of the originals, so the document, undo, autosave, and collaborators never see it. Escape or the pill leaves preview, Reset restores every control, and editing shortcuts, labels, and outlines stay off while previewing. * feat: translate behaviour and preview strings; cover preview with an e2e flow * refactor(vue): reuse VariantDefinitionControl for behaviour property options * refactor: split variant actions and preview interactions by domain Variant authoring was one 706-line closure; it is now graph queries (model), undo snapshots (history), property definition edits (definitions), and the editor facade (index). Preview interactions move into play/kinds, one module per control, registered by behaviour kind so a new kind cannot ship without its contract and interaction. Behaviour contracts are keyed by kind. In the Vue SDK, slot and variant authoring controls get their own folders beside component-props and behaviour, and the app's variant section joins slot/ and behaviour/. * refactor: keep the behaviour model in scene-graph's plugin-data registry Master now defines every OpenPencil plugin-data key in one typed registry in scene-graph. The behaviour schema registers there as a field, and the model and contracts move beside slots, exported from the package root; the @open-pencil/core/behaviours subpath is gone. * feat: interaction states and keyboard focus in preview A behaviour can bind a variant property to the default, hover, pressed, focus, and disabled states; binding it maps values named like those states. Preview switches the instance's copy to the matching variant as the pointer hovers, presses, and releases, keeps other values when the set draws the combination and falls back to rest otherwise, and skips disabled instances. Tab moves visible keyboard focus between controls, Space, Enter, arrows, Home, and End use the focused one, and Escape takes visible focus off before leaving preview. A Button kind covers controls that only have states. * feat: toggle, radio, group, progress, collapsible, and accordion behaviours Radio group, toggle group, and accordion hold their items in a slot; each item is an instance with its own behaviour, so a press inside the slot goes to the group, which turns the pressed item on and the others off through the item's own interaction. Progress shares the slider's number handling through rangeControl, and a collapsible shows and hides its content slot from its trigger, remembering its open state even when no property draws it. Tabs and groups share arrow-key navigation. * feat: text field, textarea, and number field behaviours A behaviour value can now be text, bound to a text property, so preview types into a copy of the field through the same property path the editor uses. A bound Filled value switches to the placeholder variant when the field empties. A number field keeps its own range, shows its value through a text property, and steps from its increment and decrement slots and the arrow keys. Text fields show focus from a click, and the focused control receives every key; Option still types, and only Cmd or Ctrl combinations stay shortcuts. * fix: keep behaviour bindings when saving as .fig Saving as .fig gives component properties new GUIDs, but behaviours kept the old ids in their plugin data, so every binding read as missing after reopening. The export now renames the ids behaviours bind with the same GUIDs, on its own copy of the document. * fix: let previewed controls resize layout imported from .fig Layers from a .fig keep the sizes Figma computed, and auto layout prefers them, so an opened collapsible or accordion item kept its closed height in preview. When preview shows, hides, or retypes a layer in a copy, it drops those sizes from the layer's copied ancestors so auto layout sizes them again; untouched layers keep Figma's sizes. * fix: publish behaviours and other plugin content with library assets Every OpenPencil plugin-data field now declares its role: content that exists only as plugin data (behaviours, OkHCL picks), format copies of node fields written for files, or bookkeeping about where a document or node came from. Library snapshots keep a node's content plugin data, including other plugins' entries, and drop the rest; the asset hash counts the same entries, so a behaviour-only change is offered as an update while a .fig round trip still changes nothing. * feat: name behaviour rows by meaning and create what they need The Behaviour section named every main value "Value" under a "Values" heading, and a component without matching properties left an empty picker with no way forward. Rows are now named for the control (On, Checked, Pressed, Text), rows the control needs or already uses come first, and the optional rest folds under More options; a button keeps its states in view. An empty row creates what it needs in one undo step: a text layer and text property, Off and On variants on a set, or a slot frame for a part. The missing chip names the row it means and takes you there. * fix(dom-css): position free layers, hug content, and round ellipses HTML and Tailwind export stacked the layers of frames without auto layout in block flow, wrote fixed pixel sizes for auto layout frames set to Hug and for auto-sizing text, and drew ellipses as boxes. Layers a parent does not lay out are now absolutely positioned at their coordinates inside a relative frame, hugging axes are left to the content, and ellipses get a 50% radius. * feat: run preview as live Reka UI islands over the canvas Preview simulated controls on the canvas: copies of instances, a handler per kind, its own key routing, and append-only text. It now runs them as real components. Each top-level layer that holds an instance with a behaviour becomes an island: its layers are projected to DOM through dom-css into a shadow root laid over the pane at its pan and zoom, and each behaviour mounts its Reka UI primitives on its layers, so text fields are real inputs and focus, keys, and layout are the browser's. Core's resolvePlayState shows instances in a state on a private graph, so the component's variants draw it, and controls are keyed by layer path so a variant switch keeps their DOM. The canvas leaves island layers to the islands, and the canvas play runtime and its key routing are gone. * fix: derive variant properties from Property=Value component names figma.combineAsVariants and Combine as variants only derived variant properties from slash-separated names, so components named as Figma names variants, such as State=On, Size=Large, became a set with no properties. Both now derive each named property and its values, after the slash form. * feat: script and tool access to behaviours by name Behaviour contracts follow Reka UI's anatomy: tabs keep their triggers in the list slot and their content panels in a panels slot, and a slot of repeated parts names the Reka part of its children. A behaviour spec names component properties and slots instead of ids and resolves to the stored behaviour and back, with errors that list what the component has. Scripts get an `openpencil` global next to `figma`, in the Figma API's style: setBehaviour, getBehaviour with bindValue, bindPart, states, and missing, behaviourKinds, and createSlot. The eval tool, the CLI, and app automation compile scripts through one compileScript, so the CLI now returns the last expression as the others do. MCP and AI chat get set_behaviour, get_behaviour, and create_slot. * feat: write controls in design JSX with Reka UI's element names `<Switch.Root modelValue="State">` renders a main component, or a set when its children are variants, that behaves as a switch, and `<Switch.Thumb>` the slot that draws its thumb, one slot across the set's variants. Inputs become the text property of a field, tab triggers and panels go in their List and Panels slots, and a group's items are `<RadioGroup.Item of={…} />` instances in its Items slot. JSX export writes components with behaviours the same way, so they render back unchanged. The authoring reference documents controls, and the codegen and chat prompts now include it verbatim instead of dedenting its code examples. * chore: format the CLI export test * docs: document slots, behaviours, preview, and the openpencil API The components guide covers slots, behaviours, and preview with its shortcut; scripting covers the openpencil global and eval's last- expression result; the MCP and AI chat pages list the new tools; the features overview, README, and roadmap mention working controls. The chat prompt says how to build a control, and the codegen prompt builds components with behaviours on their Reka UI primitives. * chore: format the eval CLI test * docs: explain behaviours and preview islands, and guide the openpencil API A development page explains the behaviour model, the four authoring surfaces, how preview islands turn a control's state into live Reka UI components, and how to add a kind; the architecture page links it. The Core guide sets the rules for OpenPencilAPI: Figma-only `figma`, OpenPencil features on `openpencil` in the same style, one compileScript, names over ids, and docs with every member. Package READMEs mention the openpencil global, PlayIslands, Reka-named JSX, and the behaviour model. Design JSX's behaviour modules move into a behaviours folder instead of a suffixed sibling. * refactor: center pasted layers through translate centerNodesAt repeated translate's loop, which test:dupes reports on master too. * fix: validate behaviour ranges and guess on and off by name A number value now needs max above min and a positive step: the schema, specs, and the panel reject a range a slider cannot step through. Binding a variant property guesses on and off by value name, as specs do, and a boolean property gets no on/off pair. Part bindings are read through partBinding, a replaced document restarts preview from its designed state, and the e2e preview shortcut uses ControlOrMeta. * feat: make the Behaviour section say what to do next A slider's range fields now carry inline Min, Max, Step, and Start labels. States offers Add state variants, which adds a Default, Hover, Pressed, Focus, and Disabled variant and binds them; Add Off and On variants and Add state variants turn a lone main component into a component set first, and a part's slot can be added to a set, in every variant under one slot id. Rows that could do nothing are gone: no empty pickers and no hints to combine variants by hand, and an unbound Disabled is left to the states. A warning line names what is still needed and replaces the missing chip, and the Switch's main value is called Checked. * fix: keep each slot to one part and keep creating slots at hand A slot draws one part, so the Behaviour section no longer offers a slot another part uses, and specs (the openpencil API, tools, and JSX) reject binding one slot to two parts. A part's picker keeps an action to add a new slot in its footer, so adding the first slot no longer hides it for the other parts.
2026-10-06 13:23:05 +00:00
- **Components** — create components, instances, component sets, and slots. Manage overrides. Give components Reka UI behaviours so they work in preview.
- **Variables** — create/edit variables, collections, modes. Bind to fills.
2026-03-09 14:43:11 +00:00
- **Query** — find nodes, XPath selectors, read properties, list pages, fonts, selection.
feat: add visual diff and patch apply tools and openpencil diff (#810) * feat(core): add visual diff and patch apply tools diff_visual renders two nodes at one scale through the existing raster export, compares them with pixelmatch, and returns the diff PNG with the changed ratio and region in source-node coordinates. It takes export_image's scale and maxEdge inputs. FigmaAPI gains a CanvasKit-backed raster codec and a pageId export option, so the app and headless CLI decode pixels and render nodes off the current page. diff_apply applies diff_create and diff_show patches through the Figma API, validates every node before changing any, and supports dryRun and force. diff_show now simulates changes on a detached copy with the same property code. One serializer and parser back all three. diffDocuments compares two documents page by page by name path. Image tool results now reach models as media with their metadata as text, for any tool rather than export_image alone. diff_create, diff_jsx, and diff_visual join the default AI tool set, and the diff tools are no longer hidden from WebMCP. * feat(cli): add diff commands and agent diff guidance openpencil diff create, jsx, show, apply, and visual run the Core diff tools on a file or the running app; apply writes back with --write or --output like eval. diff files compares two documents page by page and exits 1 when they differ. The chat prompt asks the agent to edit in place and to verify risky edits against a reference copy with diff_jsx, diff_create, and diff_visual. The skill, CLI reference, MCP tool table, and a new Comparing Designs page document the commands and tools. * feat(core): diff and patch node trees as JSX attributes diff_create, diff_show, diff_apply, and diffDocuments used a hand-rolled `key: value` property format that covered about fifteen properties, matched children by name path, and could not see moves. Nodes are now projected to the attributes the JSX export prints, and jsondiffpatch matches children (by ID or by name path) and detects moves. Patches list `-`/`+` attribute lines per node plus moved, added, and removed children. diff_apply checks every hunk first, applies attribute changes through the renderer's prop handling, and changes only the fields an attribute moves, so IDs, instance links, and other state survive. diff_show takes JSX attributes instead of a JSON props object. design-jsx gains sceneNodeAttributes, parseJSXAttributes, and jsxNodeFields for this, and the export round-trip property table is shared so every case is also diffed and applied. `diff files` loads its documents in order so node IDs, and so its patches, are deterministic. * fix(core): keep diff_apply atomic and diff files honest about differences - Added nodes render before anything else changes; if one fails, for example on a missing component, the rendered ones are deleted and nothing else is committed. - A hunk with an attribute the renderer ignores fails instead of reporting "unchanged". - diffDocuments reports `changed` from page statuses, and a page only one document has gets its status but no patch, since patches do not add or remove pages. diff files uses it, so an added empty page no longer reads as a match. - diff files rejects a --page neither document has and a --depth that is not a non-negative integer, exiting 2; diff_create's depth is validated the same way.
2026-10-03 17:00:13 +00:00
- **Inspect** — `get_jsx` for JSX roundtrip view, `diff_create` and `diff_jsx` for structural diffs, `diff_visual` for pixel diffs, `describe` for semantic role and design issue detection.
- **Analyze** — color palette, typography audit, spacing consistency, cluster detection.
2026-03-09 14:43:11 +00:00
- **Export** — PNG, SVG, JSX with Tailwind classes. Vision-based verification via `export_image`.
- **Vector** — boolean operations, path manipulation.
2026-03-09 14:43:11 +00:00
## Visual Verification
feat: add visual diff and patch apply tools and openpencil diff (#810) * feat(core): add visual diff and patch apply tools diff_visual renders two nodes at one scale through the existing raster export, compares them with pixelmatch, and returns the diff PNG with the changed ratio and region in source-node coordinates. It takes export_image's scale and maxEdge inputs. FigmaAPI gains a CanvasKit-backed raster codec and a pageId export option, so the app and headless CLI decode pixels and render nodes off the current page. diff_apply applies diff_create and diff_show patches through the Figma API, validates every node before changing any, and supports dryRun and force. diff_show now simulates changes on a detached copy with the same property code. One serializer and parser back all three. diffDocuments compares two documents page by page by name path. Image tool results now reach models as media with their metadata as text, for any tool rather than export_image alone. diff_create, diff_jsx, and diff_visual join the default AI tool set, and the diff tools are no longer hidden from WebMCP. * feat(cli): add diff commands and agent diff guidance openpencil diff create, jsx, show, apply, and visual run the Core diff tools on a file or the running app; apply writes back with --write or --output like eval. diff files compares two documents page by page and exits 1 when they differ. The chat prompt asks the agent to edit in place and to verify risky edits against a reference copy with diff_jsx, diff_create, and diff_visual. The skill, CLI reference, MCP tool table, and a new Comparing Designs page document the commands and tools. * feat(core): diff and patch node trees as JSX attributes diff_create, diff_show, diff_apply, and diffDocuments used a hand-rolled `key: value` property format that covered about fifteen properties, matched children by name path, and could not see moves. Nodes are now projected to the attributes the JSX export prints, and jsondiffpatch matches children (by ID or by name path) and detects moves. Patches list `-`/`+` attribute lines per node plus moved, added, and removed children. diff_apply checks every hunk first, applies attribute changes through the renderer's prop handling, and changes only the fields an attribute moves, so IDs, instance links, and other state survive. diff_show takes JSX attributes instead of a JSON props object. design-jsx gains sceneNodeAttributes, parseJSXAttributes, and jsxNodeFields for this, and the export round-trip property table is shared so every case is also diffed and applied. `diff files` loads its documents in order so node IDs, and so its patches, are deterministic. * fix(core): keep diff_apply atomic and diff files honest about differences - Added nodes render before anything else changes; if one fails, for example on a missing component, the rendered ones are deleted and nothing else is committed. - A hunk with an attribute the renderer ignores fails instead of reporting "unchanged". - diffDocuments reports `changed` from page statuses, and a page only one document has gets its status but no patch, since patches do not add or remove pages. diff files uses it, so an added empty page no longer reads as a match. - diff files rejects a --page neither document has and a --depth that is not a non-negative integer, exiting 2; diff_create's depth is validated the same way.
2026-10-03 17:00:13 +00:00
The assistant can verify its work visually. When `export_image` is enabled, it can capture a screenshot after creating or modifying designs and checks the result against the original request. This catches layout issues, missing elements, and color mismatches that text-only responses would miss. `diff_visual`, enabled by default, compares an edited node with a reference copy and returns the changed pixels and region, so the assistant can confirm an edit stayed within its target.
2026-03-09 14:43:11 +00:00
## Example Prompts
- "Create a card with a title, description, and a blue button"
- "Make all buttons on this page use the same border radius"
- "What fonts are used in this file?"
- "Change the background of the selected frame to a gradient from blue to purple"
- "Export the selected frame as SVG"
- "Find all text nodes with font size less than 12"
2026-03-09 14:43:11 +00:00
- "Describe the selected component — what role does it look like?"
- "Show me the JSX for this frame"
feat: behaviours and preview mode (#893) * feat: author behaviours on main components A main component or component set can behave as a Switch, Checkbox, Slider, or Tabs, after Reka UI's primitives. The behaviour lives in OpenPencil plugin data: boolean values bind to variant or boolean properties with the values meaning on and off, a number keeps its own range since Figma has no number property, and the control's subcomponents bind to the component's slots. A Behaviour section in the properties panel adds, binds, and removes it, each as one undo step, and flags required bindings that are missing. The canvas-only layout's pill becomes a component that preview will reuse. * feat: preview instances with behaviours on the canvas View > Preview (Cmd+Alt+Enter) puts the canvas in preview: a lone canvas switches to the canvas-only layout with a Previewing pill, and a split canvas previews on its own side. Clicking a Switch or Checkbox flips it, dragging a Slider moves its thumb and range, and clicking a Tabs trigger shows its panel. Preview keeps its state on copies of the instances it touched, in a private graph with the document's ids, and the canvas draws those copies in place of the originals, so the document, undo, autosave, and collaborators never see it. Escape or the pill leaves preview, Reset restores every control, and editing shortcuts, labels, and outlines stay off while previewing. * feat: translate behaviour and preview strings; cover preview with an e2e flow * refactor(vue): reuse VariantDefinitionControl for behaviour property options * refactor: split variant actions and preview interactions by domain Variant authoring was one 706-line closure; it is now graph queries (model), undo snapshots (history), property definition edits (definitions), and the editor facade (index). Preview interactions move into play/kinds, one module per control, registered by behaviour kind so a new kind cannot ship without its contract and interaction. Behaviour contracts are keyed by kind. In the Vue SDK, slot and variant authoring controls get their own folders beside component-props and behaviour, and the app's variant section joins slot/ and behaviour/. * refactor: keep the behaviour model in scene-graph's plugin-data registry Master now defines every OpenPencil plugin-data key in one typed registry in scene-graph. The behaviour schema registers there as a field, and the model and contracts move beside slots, exported from the package root; the @open-pencil/core/behaviours subpath is gone. * feat: interaction states and keyboard focus in preview A behaviour can bind a variant property to the default, hover, pressed, focus, and disabled states; binding it maps values named like those states. Preview switches the instance's copy to the matching variant as the pointer hovers, presses, and releases, keeps other values when the set draws the combination and falls back to rest otherwise, and skips disabled instances. Tab moves visible keyboard focus between controls, Space, Enter, arrows, Home, and End use the focused one, and Escape takes visible focus off before leaving preview. A Button kind covers controls that only have states. * feat: toggle, radio, group, progress, collapsible, and accordion behaviours Radio group, toggle group, and accordion hold their items in a slot; each item is an instance with its own behaviour, so a press inside the slot goes to the group, which turns the pressed item on and the others off through the item's own interaction. Progress shares the slider's number handling through rangeControl, and a collapsible shows and hides its content slot from its trigger, remembering its open state even when no property draws it. Tabs and groups share arrow-key navigation. * feat: text field, textarea, and number field behaviours A behaviour value can now be text, bound to a text property, so preview types into a copy of the field through the same property path the editor uses. A bound Filled value switches to the placeholder variant when the field empties. A number field keeps its own range, shows its value through a text property, and steps from its increment and decrement slots and the arrow keys. Text fields show focus from a click, and the focused control receives every key; Option still types, and only Cmd or Ctrl combinations stay shortcuts. * fix: keep behaviour bindings when saving as .fig Saving as .fig gives component properties new GUIDs, but behaviours kept the old ids in their plugin data, so every binding read as missing after reopening. The export now renames the ids behaviours bind with the same GUIDs, on its own copy of the document. * fix: let previewed controls resize layout imported from .fig Layers from a .fig keep the sizes Figma computed, and auto layout prefers them, so an opened collapsible or accordion item kept its closed height in preview. When preview shows, hides, or retypes a layer in a copy, it drops those sizes from the layer's copied ancestors so auto layout sizes them again; untouched layers keep Figma's sizes. * fix: publish behaviours and other plugin content with library assets Every OpenPencil plugin-data field now declares its role: content that exists only as plugin data (behaviours, OkHCL picks), format copies of node fields written for files, or bookkeeping about where a document or node came from. Library snapshots keep a node's content plugin data, including other plugins' entries, and drop the rest; the asset hash counts the same entries, so a behaviour-only change is offered as an update while a .fig round trip still changes nothing. * feat: name behaviour rows by meaning and create what they need The Behaviour section named every main value "Value" under a "Values" heading, and a component without matching properties left an empty picker with no way forward. Rows are now named for the control (On, Checked, Pressed, Text), rows the control needs or already uses come first, and the optional rest folds under More options; a button keeps its states in view. An empty row creates what it needs in one undo step: a text layer and text property, Off and On variants on a set, or a slot frame for a part. The missing chip names the row it means and takes you there. * fix(dom-css): position free layers, hug content, and round ellipses HTML and Tailwind export stacked the layers of frames without auto layout in block flow, wrote fixed pixel sizes for auto layout frames set to Hug and for auto-sizing text, and drew ellipses as boxes. Layers a parent does not lay out are now absolutely positioned at their coordinates inside a relative frame, hugging axes are left to the content, and ellipses get a 50% radius. * feat: run preview as live Reka UI islands over the canvas Preview simulated controls on the canvas: copies of instances, a handler per kind, its own key routing, and append-only text. It now runs them as real components. Each top-level layer that holds an instance with a behaviour becomes an island: its layers are projected to DOM through dom-css into a shadow root laid over the pane at its pan and zoom, and each behaviour mounts its Reka UI primitives on its layers, so text fields are real inputs and focus, keys, and layout are the browser's. Core's resolvePlayState shows instances in a state on a private graph, so the component's variants draw it, and controls are keyed by layer path so a variant switch keeps their DOM. The canvas leaves island layers to the islands, and the canvas play runtime and its key routing are gone. * fix: derive variant properties from Property=Value component names figma.combineAsVariants and Combine as variants only derived variant properties from slash-separated names, so components named as Figma names variants, such as State=On, Size=Large, became a set with no properties. Both now derive each named property and its values, after the slash form. * feat: script and tool access to behaviours by name Behaviour contracts follow Reka UI's anatomy: tabs keep their triggers in the list slot and their content panels in a panels slot, and a slot of repeated parts names the Reka part of its children. A behaviour spec names component properties and slots instead of ids and resolves to the stored behaviour and back, with errors that list what the component has. Scripts get an `openpencil` global next to `figma`, in the Figma API's style: setBehaviour, getBehaviour with bindValue, bindPart, states, and missing, behaviourKinds, and createSlot. The eval tool, the CLI, and app automation compile scripts through one compileScript, so the CLI now returns the last expression as the others do. MCP and AI chat get set_behaviour, get_behaviour, and create_slot. * feat: write controls in design JSX with Reka UI's element names `<Switch.Root modelValue="State">` renders a main component, or a set when its children are variants, that behaves as a switch, and `<Switch.Thumb>` the slot that draws its thumb, one slot across the set's variants. Inputs become the text property of a field, tab triggers and panels go in their List and Panels slots, and a group's items are `<RadioGroup.Item of={…} />` instances in its Items slot. JSX export writes components with behaviours the same way, so they render back unchanged. The authoring reference documents controls, and the codegen and chat prompts now include it verbatim instead of dedenting its code examples. * chore: format the CLI export test * docs: document slots, behaviours, preview, and the openpencil API The components guide covers slots, behaviours, and preview with its shortcut; scripting covers the openpencil global and eval's last- expression result; the MCP and AI chat pages list the new tools; the features overview, README, and roadmap mention working controls. The chat prompt says how to build a control, and the codegen prompt builds components with behaviours on their Reka UI primitives. * chore: format the eval CLI test * docs: explain behaviours and preview islands, and guide the openpencil API A development page explains the behaviour model, the four authoring surfaces, how preview islands turn a control's state into live Reka UI components, and how to add a kind; the architecture page links it. The Core guide sets the rules for OpenPencilAPI: Figma-only `figma`, OpenPencil features on `openpencil` in the same style, one compileScript, names over ids, and docs with every member. Package READMEs mention the openpencil global, PlayIslands, Reka-named JSX, and the behaviour model. Design JSX's behaviour modules move into a behaviours folder instead of a suffixed sibling. * refactor: center pasted layers through translate centerNodesAt repeated translate's loop, which test:dupes reports on master too. * fix: validate behaviour ranges and guess on and off by name A number value now needs max above min and a positive step: the schema, specs, and the panel reject a range a slider cannot step through. Binding a variant property guesses on and off by value name, as specs do, and a boolean property gets no on/off pair. Part bindings are read through partBinding, a replaced document restarts preview from its designed state, and the e2e preview shortcut uses ControlOrMeta. * feat: make the Behaviour section say what to do next A slider's range fields now carry inline Min, Max, Step, and Start labels. States offers Add state variants, which adds a Default, Hover, Pressed, Focus, and Disabled variant and binds them; Add Off and On variants and Add state variants turn a lone main component into a component set first, and a part's slot can be added to a set, in every variant under one slot id. Rows that could do nothing are gone: no empty pickers and no hints to combine variants by hand, and an unbound Disabled is left to the states. A warning line names what is still needed and replaces the missing chip, and the Switch's main value is called Checked. * fix: keep each slot to one part and keep creating slots at hand A slot draws one part, so the Behaviour section no longer offers a slot another part uses, and specs (the openpencil API, tools, and JSX) reject binding one slot to two parts. A part's picker keeps an action to add a new slot in its footer, so adding the first slot no longer hides it for the other parts.
2026-10-06 13:23:05 +00:00
- "Make this component a switch that I can try in preview"
## Tips
- Select nodes before asking — the assistant knows what's selected.
- Be specific about colors, sizes, and positions for precise results.
- The assistant can modify multiple nodes in one message.
feat(app): show who works on each page (#815) * fix(vue): keep the command palette open when a command opens a step CommandPaletteRoot emitted select for every item, including one that only opens its children, so a host that closes on select closed the palette instead of showing the step. useCommandPalette.select now reports whether a command ran, and the root emits only then. Disabled items were marked only with Reka's data-disabled; expose aria-disabled so assistive technology announces them. * feat(app): jump between pages from the command palette The palette had no way to reach a page. It now lists the pages visited recently in the tab, offers a Go to page step with every page, and finds any page by name. Recent pages are tracked per editor session from page changes and reset when the document is replaced. Palette items can be search-only, so pages beyond the recent ones appear only when the query matches them. The divider-page rule moves out of PageListRoot so the palette skips dividers the same way, and useCommandPalette is exported from the package root. * feat(canvas): draw agents' cursors as outlined sparkles Editor state's remoteCursors becomes presenceCursors with a kind, since the list now includes local agents. People keep the filled arrow; an agent is a sparkle outlined in its owner's color, with an outlined name pill, so whose agent it is reads from the outline. Cursor drawing moves out of the pen overlay into canvas/overlays/presence.ts. * feat(app): publish AI agent presence to collaborators The built-in chat now appears as an agent with a callsign while it replies, at the nodes its tools touch on the run's page, and goes idle (off the canvas) when the reply ends. Agents live in a per-document presence registry and are published in their owner's awareness state, so collaborators see each other's agents in the owner's color; the payload is metadata only. Peer awareness was cast without checks. It is now validated with Valibot, invalid fields are dropped rather than the peer, and names, selections, and agent counts are bounded. * feat(app): follow agents and list them in the share panel Following lived in collab and only knew people. It moves into the presence registry with a person-or-agent target, so you can follow anyone's agent, including your own outside a room: the view goes to the agent's page and keeps its cursor centered, stays attached while it idles between replies, and lets go when it leaves. A new editor action, centerOn, replaces reading the canvas size from the DOM. Peer cursors keep their zoom so following a person still matches it. The share panel lists everyone in the room with their agents, each with its status, page, and a follow toggle, and your own agents can be renamed inline. CollabPanel moves to collab-panel, and the two-browser relay helpers move out of the collab spec into tests/helpers/collab. * feat(app): show who works on each page Agents now publish the page they work on, set when a reply starts on its pinned page and moved by switch_page, so a page is marked before the agent's first edit. presenceByPage groups people and working agents by page. The Pages panel marks those pages with people's dots and agents' outlined sparkles in their owner colors, the command palette names who is on each page, and the chat says which page a reply is working on, with Go to page, while you view another one. * docs(collaboration): list the agent model among shared presence * test(app): stories for page presence markers and the chat's run location The run location notice reads app state, so it moves into useChatRunLocation and the component takes the agent and page as props. * test(vue): a canvas story for presence cursors Storybook now serves CanvasKit, so a story can render the real canvas: people's arrows and agents' outlined sparkles, with controls for names, colors, and zoom. * feat(canvas): mark agents with a sparkle label instead of a sparkle cursor A sparkle on its own did not read as a pointer. Agents now point with the same filled arrow as people, in their owner's color, and their outlined label starts with a sparkle. * refactor(app): split the collaboration theme by component One 18-slot theme served five components that each used a few slots, with variants that applied to one slot. Avatars, the share button, the presence list, page markers, and the mobile presence popover now have their own themes, exported as tv() like the rest of src/theme. * fix(app): truncate an agent's status before its name in the presence list In a narrow share panel the status kept its width and the callsign shrank to its first letter. * feat(app): right-align page badges in a trailing area of the page row Presence markers followed the page name. The row now has a trailing area, right-aligned with its own spacing, where markers and later page badges go. * fix(app): key page markers by person or agent, not by name Two people with the same name on a page, such as two Anonymous peers, gave page markers duplicate keys. Entries now carry a stable id.
2026-10-03 21:18:16 +00:00
- You can browse other pages while a reply runs: the assistant keeps working on the page where the message started, and its previews show when you return. While you're away, the chat says which page it is working on, with **Go to page** to return. If the assistant switches pages itself, your view follows.
2026-03-09 14:43:11 +00:00
- Use "undo" in the editor if you don't like the result — AI mutations support full undo.
- All layout is recomputed automatically after each tool execution.