feat: show AI agents on the canvas and to collaborators (#803)
* feat(canvas): draw agents' cursors as outlined sparkles Editor state's remoteCursors becomes presenceCursors with a kind, since the list now includes local agents. People keep the filled arrow; an agent is a sparkle outlined in its owner's color, with an outlined name pill, so whose agent it is reads from the outline. Cursor drawing moves out of the pen overlay into canvas/overlays/presence.ts. * feat(app): publish AI agent presence to collaborators The built-in chat now appears as an agent with a callsign while it replies, at the nodes its tools touch on the run's page, and goes idle (off the canvas) when the reply ends. Agents live in a per-document presence registry and are published in their owner's awareness state, so collaborators see each other's agents in the owner's color; the payload is metadata only. Peer awareness was cast without checks. It is now validated with Valibot, invalid fields are dropped rather than the peer, and names, selections, and agent counts are bounded. * docs(collaboration): list the agent model among shared presence * test(vue): a canvas story for presence cursors Storybook now serves CanvasKit, so a story can render the real canvas: people's arrows and agents' outlined sparkles, with controls for names, colors, and zoom. * feat(canvas): mark agents with a sparkle label instead of a sparkle cursor A sparkle on its own did not read as a pointer. Agents now point with the same filled arrow as people, in their owner's color, and their outlined label starts with a sparkle.
This commit is contained in:
Danila Poyarkov
GitHub
parent
0ff6b414e3
commit
749d7baec9
|
|
@ -4,6 +4,7 @@
|
|||
|
||||
### Breaking changes
|
||||
|
||||
- The editor state's `remoteCursors` is now `presenceCursors`, typed `PresenceCursor[]` from `@open-pencil/core/canvas`, and each cursor has a `kind` of `'person'` or `'agent'`.
|
||||
- `VariableBinding` and the `colorVariableBinding` paint field are gone from `@open-pencil/core`, `@open-pencil/core/kiwi`, and the Kiwi `Paint` type. `fig.kiwi` never defined the field, so only `.fig` files OpenPencil itself wrote before `colorVar` contain one; reopening such a file leaves the paint's colour unbound, and binding it again records it the way Figma does.
|
||||
- `encodeNodeChangeWithVariables`, `encodePaintWithVariableBinding`, and `encodeVarint` are removed from `@open-pencil/core` and `@open-pencil/core/kiwi`. They spliced a colour-variable binding into encoded bytes because the field had no schema entry; exports now write `colorVar`, which `fig.kiwi` defines, so nothing needs them. `parseVariableId` is unchanged.
|
||||
- The desktop app now requires macOS 13 or later; the web app supports Chrome 111, Edge 111, Firefox 128, and Safari 16.4 or later.
|
||||
|
|
@ -14,6 +15,7 @@
|
|||
|
||||
### Added
|
||||
|
||||
- See where the built-in AI chat is working: while it replies, a cursor whose outlined label shows a sparkle and a callsign such as *Fern* marks the layers it edits. In a shared room, collaborators see each other's agents in the color of the person running them.
|
||||
- Jump between pages from the command palette: it lists the pages you visited recently in the tab, **Go to page…** lists every page, and typing a page name finds it.
|
||||
- Choose how much an AI model thinks for each message from the chat composer, from Off to Extra high or the provider default. Anthropic, Google, DeepSeek, OpenAI, OpenRouter, and compatible models apply it, where reasoning effort previously reached only OpenAI and OpenRouter models. A model profile's thinking level, which replaces its free-text reasoning effort, sets the starting choice, and finished reasoning shows how long the model thought.
|
||||
- Preview designs progressively on the canvas as direct AI providers stream JSX, without saving partial designs or adding intermediate undo steps. A preview stays with its page: it hides while you view another page and returns when you come back.
|
||||
|
|
@ -75,6 +77,7 @@
|
|||
|
||||
### Security
|
||||
|
||||
- Validate cursors, selections, and names that collaborators send before drawing them, and cap their size, so a broken or hostile peer cannot crash or flood the canvas.
|
||||
- Evaluate `calc` expressions through `jsep` and an arithmetic allowlist that never compiles input into JavaScript, replacing the `expr-eval` dependency and its unpatched critical code-execution advisory (GHSA-q9v2-7m5w-4693).
|
||||
- Escape layer names and other text properties in JSX and Tailwind JSX export, so text from a document can no longer add attributes or JavaScript expressions that the AI and MCP `render` and `replace` tools would execute, and names containing `&` no longer change when the JSX is rendered back.
|
||||
|
||||
|
|
|
|||
|
|
@ -14,4 +14,4 @@ export { computeGuideRedline } from './guides/redlines'
|
|||
export { hitTestGuides, type GuideHit } from './guides/hit-test'
|
||||
export type { GuideOverlayState, GuidePreview, GuideSelection } from './guides/types'
|
||||
export { canvasLabelForeground } from './labels/color'
|
||||
export { SkiaRenderer, type RenderOverlays, type RulerTheme } from './renderer'
|
||||
export { SkiaRenderer, type PresenceCursor, type RenderOverlays, type RulerTheme } from './renderer'
|
||||
|
|
|
|||
|
|
@ -14,5 +14,6 @@ export { drawFlashes, drawLayoutInsertIndicator, drawMarquee, drawSnapGuides } f
|
|||
export { drawMeasurements } from './measurement'
|
||||
export { drawTextEditOverlay } from './text-edit'
|
||||
export { drawSelectionLabels } from '#core/canvas/labels/selection'
|
||||
export { drawPenOverlay, drawRemoteCursors } from '#core/canvas/pen-overlay'
|
||||
export { drawPenOverlay } from '#core/canvas/pen-overlay'
|
||||
export { drawPresenceCursors } from './presence'
|
||||
export { computeHandleVisibleVertices, drawNodeEditOverlay } from '#core/canvas/node-edit-overlay'
|
||||
|
|
|
|||
166
packages/core/src/canvas/overlays/presence.ts
Normal file
166
packages/core/src/canvas/overlays/presence.ts
Normal file
|
|
@ -0,0 +1,166 @@
|
|||
import type { Canvas } from 'canvaskit-wasm'
|
||||
|
||||
import type { SceneGraph } from '@open-pencil/scene-graph'
|
||||
import { getWorldMatrix } from '@open-pencil/scene-graph/coordinate'
|
||||
import Matrix from '@open-pencil/scene-graph/matrix'
|
||||
|
||||
import type { PresenceCursor, SkiaRenderer } from '#core/canvas/renderer'
|
||||
|
||||
const CURSOR_SIZE = 9
|
||||
const LABEL_PADDING_X = 4
|
||||
const LABEL_PADDING_Y = 2
|
||||
const LABEL_FONT_SIZE = 10
|
||||
const LABEL_OFFSET_X = 12
|
||||
const LABEL_OFFSET_Y = 20
|
||||
/** Longer names end with an ellipsis. */
|
||||
const LABEL_MAX_WIDTH = 160
|
||||
/** The sparkle that marks an agent's name. */
|
||||
const LABEL_SPARKLE_RADIUS = 4
|
||||
const LABEL_SPARKLE_GAP = 3
|
||||
|
||||
function drawSelection(r: SkiaRenderer, canvas: Canvas, graph: SceneGraph, cursor: PresenceCursor) {
|
||||
if (!cursor.selection?.length) return
|
||||
const { r: red, g, b } = cursor.color
|
||||
r.auxStroke.setColor(r.ck.Color4f(red, g, b, 0.6))
|
||||
r.auxStroke.setStrokeWidth(1.5)
|
||||
r.auxStroke.setPathEffect(null)
|
||||
for (const nodeId of cursor.selection) {
|
||||
const node = graph.getNode(nodeId)
|
||||
if (!node) continue
|
||||
// Map the corners through the node's world matrix, as the shape is rendered, so the
|
||||
// box tracks rotation, flips, and parent transforms.
|
||||
const corners = Matrix.mapPoints(getWorldMatrix(node, graph), [
|
||||
0,
|
||||
0,
|
||||
node.width,
|
||||
0,
|
||||
node.width,
|
||||
node.height,
|
||||
0,
|
||||
node.height
|
||||
])
|
||||
const box = new r.ck.PathBuilder()
|
||||
box.moveTo(corners[0] * r.zoom + r.panX, corners[1] * r.zoom + r.panY)
|
||||
for (let i = 2; i < corners.length; i += 2) {
|
||||
box.lineTo(corners[i] * r.zoom + r.panX, corners[i + 1] * r.zoom + r.panY)
|
||||
}
|
||||
box.close()
|
||||
const path = box.detachAndDelete()
|
||||
canvas.drawPath(path, r.auxStroke)
|
||||
path.delete()
|
||||
}
|
||||
}
|
||||
|
||||
function arrowPath(r: SkiaRenderer, x: number, y: number) {
|
||||
const S = CURSOR_SIZE
|
||||
const builder = new r.ck.PathBuilder()
|
||||
builder.moveTo(x, y)
|
||||
builder.lineTo(x, y + S * 1.35)
|
||||
builder.lineTo(x + S * 0.38, y + S * 1.0)
|
||||
builder.lineTo(x + S * 0.72, y + S * 1.5)
|
||||
builder.lineTo(x + S * 0.92, y + S * 1.38)
|
||||
builder.lineTo(x + S * 0.58, y + S * 0.88)
|
||||
builder.lineTo(x + S * 1.0, y + S * 0.82)
|
||||
builder.close()
|
||||
return builder.detachAndDelete()
|
||||
}
|
||||
|
||||
/** A four-point sparkle of radius `R` centered on `cx`, `cy`. */
|
||||
function sparklePath(r: SkiaRenderer, cx: number, cy: number, R: number) {
|
||||
const inner = R * 0.32
|
||||
const builder = new r.ck.PathBuilder()
|
||||
builder.moveTo(cx, cy - R)
|
||||
builder.lineTo(cx + inner, cy - inner)
|
||||
builder.lineTo(cx + R, cy)
|
||||
builder.lineTo(cx + inner, cy + inner)
|
||||
builder.lineTo(cx, cy + R)
|
||||
builder.lineTo(cx - inner, cy + inner)
|
||||
builder.lineTo(cx - R, cy)
|
||||
builder.lineTo(cx - inner, cy - inner)
|
||||
builder.close()
|
||||
return builder.detachAndDelete()
|
||||
}
|
||||
|
||||
/** The pointer, filled with a person's color, or for an agent its owner's, and outlined in white. */
|
||||
function drawArrow(r: SkiaRenderer, canvas: Canvas, x: number, y: number, cursor: PresenceCursor) {
|
||||
const path = arrowPath(r, x, y)
|
||||
r.auxStroke.setColor(r.ck.WHITE)
|
||||
r.auxStroke.setStrokeWidth(2)
|
||||
r.auxStroke.setPathEffect(null)
|
||||
canvas.drawPath(path, r.auxStroke)
|
||||
const { r: red, g, b } = cursor.color
|
||||
r.auxFill.setColor(r.ck.Color4f(red, g, b, 1))
|
||||
canvas.drawPath(path, r.auxFill)
|
||||
path.delete()
|
||||
}
|
||||
|
||||
/**
|
||||
* A name pill: filled for people; for agents outlined in the owner's color, with a
|
||||
* sparkle before the name.
|
||||
*/
|
||||
function drawLabel(r: SkiaRenderer, canvas: Canvas, x: number, y: number, cursor: PresenceCursor) {
|
||||
const provider = r.fontProvider
|
||||
if (!cursor.name || !provider) return
|
||||
const { r: red, g, b } = cursor.color
|
||||
const color = r.ck.Color4f(red, g, b, 1)
|
||||
const agent = cursor.kind === 'agent'
|
||||
const badge = agent ? LABEL_SPARKLE_RADIUS * 2 + LABEL_SPARKLE_GAP : 0
|
||||
const pillX = x + LABEL_OFFSET_X - LABEL_PADDING_X
|
||||
const pillY = y + LABEL_OFFSET_Y - LABEL_FONT_SIZE - LABEL_PADDING_Y + 2
|
||||
const pillHeight = LABEL_FONT_SIZE + LABEL_PADDING_Y * 2
|
||||
// Shaped like other canvas labels: kerning, fallback fonts, and RTL names.
|
||||
r.labelParagraphCache.use(
|
||||
r.ck,
|
||||
provider,
|
||||
cursor.name,
|
||||
LABEL_FONT_SIZE,
|
||||
LABEL_MAX_WIDTH,
|
||||
agent ? color : r.ck.WHITE,
|
||||
r.fontGeneration,
|
||||
({ paragraph, metrics }) => {
|
||||
const pill = r.ck.RRectXY(
|
||||
r.ck.XYWHRect(pillX, pillY, badge + metrics.width + LABEL_PADDING_X * 2, pillHeight),
|
||||
4,
|
||||
4
|
||||
)
|
||||
r.auxFill.setColor(agent ? r.ck.WHITE : color)
|
||||
canvas.drawRRect(pill, r.auxFill)
|
||||
if (agent) {
|
||||
r.auxStroke.setColor(color)
|
||||
r.auxStroke.setStrokeWidth(1)
|
||||
r.auxStroke.setPathEffect(null)
|
||||
canvas.drawRRect(pill, r.auxStroke)
|
||||
const sparkle = sparklePath(
|
||||
r,
|
||||
pillX + LABEL_PADDING_X + LABEL_SPARKLE_RADIUS,
|
||||
pillY + pillHeight / 2,
|
||||
LABEL_SPARKLE_RADIUS
|
||||
)
|
||||
r.auxFill.setColor(color)
|
||||
canvas.drawPath(sparkle, r.auxFill)
|
||||
sparkle.delete()
|
||||
}
|
||||
canvas.drawParagraph(
|
||||
paragraph,
|
||||
pillX + LABEL_PADDING_X + badge,
|
||||
pillY + (pillHeight - metrics.height) / 2
|
||||
)
|
||||
}
|
||||
)
|
||||
}
|
||||
|
||||
/** Cursors of people and agents in screen space, with outlines of what they selected. */
|
||||
export function drawPresenceCursors(
|
||||
r: SkiaRenderer,
|
||||
canvas: Canvas,
|
||||
graph: SceneGraph,
|
||||
cursors?: PresenceCursor[]
|
||||
): void {
|
||||
for (const cursor of cursors ?? []) {
|
||||
const x = cursor.x * r.zoom + r.panX
|
||||
const y = cursor.y * r.zoom + r.panY
|
||||
drawSelection(r, canvas, graph, cursor)
|
||||
drawArrow(r, canvas, x, y, cursor)
|
||||
drawLabel(r, canvas, x, y, cursor)
|
||||
}
|
||||
}
|
||||
|
|
@ -1,8 +1,5 @@
|
|||
import type { Canvas, Paint } from 'canvaskit-wasm'
|
||||
|
||||
import type { SceneGraph } from '@open-pencil/scene-graph'
|
||||
import { getWorldMatrix } from '@open-pencil/scene-graph/coordinate'
|
||||
import Matrix from '@open-pencil/scene-graph/matrix'
|
||||
import type { Vector } from '@open-pencil/scene-graph/primitives'
|
||||
|
||||
import { PEN_HANDLE_RADIUS, PEN_VERTEX_RADIUS, PEN_CLOSE_RADIUS_BOOST } from '#core/constants'
|
||||
|
|
@ -192,112 +189,3 @@ export function drawPenOverlay(
|
|||
canvas.drawCircle(v.x, v.y, radius, vertexStroke)
|
||||
}
|
||||
}
|
||||
|
||||
export function drawRemoteCursors(
|
||||
r: SkiaRenderer,
|
||||
canvas: Canvas,
|
||||
graph: SceneGraph,
|
||||
cursors?: RenderOverlays['remoteCursors']
|
||||
): void {
|
||||
if (!cursors || cursors.length === 0) return
|
||||
|
||||
const CURSOR_SIZE = 9
|
||||
const LABEL_PADDING_X = 4
|
||||
const LABEL_PADDING_Y = 2
|
||||
const LABEL_FONT_SIZE = 10
|
||||
const LABEL_OFFSET_X = 12
|
||||
const LABEL_OFFSET_Y = 20
|
||||
/** Longer names end with an ellipsis. */
|
||||
const LABEL_MAX_WIDTH = 160
|
||||
|
||||
for (const cursor of cursors) {
|
||||
const screenX = cursor.x * r.zoom + r.panX
|
||||
const screenY = cursor.y * r.zoom + r.panY
|
||||
const { r: cr, g, b } = cursor.color
|
||||
|
||||
if (cursor.selection?.length) {
|
||||
r.auxStroke.setColor(r.ck.Color4f(cr, g, b, 0.6))
|
||||
r.auxStroke.setStrokeWidth(1.5)
|
||||
r.auxStroke.setPathEffect(null)
|
||||
for (const nodeId of cursor.selection) {
|
||||
const node = graph.getNode(nodeId)
|
||||
if (!node) continue
|
||||
// Map the node's four corners through its world matrix (the same transform
|
||||
// the shape is rendered with) so a remote peer's selection box tracks the
|
||||
// node exactly, including rotation, flips, and any parent transforms.
|
||||
const m = getWorldMatrix(node, graph)
|
||||
const c = Matrix.mapPoints(m, [
|
||||
0,
|
||||
0,
|
||||
node.width,
|
||||
0,
|
||||
node.width,
|
||||
node.height,
|
||||
0,
|
||||
node.height
|
||||
])
|
||||
const box = new r.ck.PathBuilder()
|
||||
box.moveTo(c[0] * r.zoom + r.panX, c[1] * r.zoom + r.panY)
|
||||
for (let i = 2; i < c.length; i += 2) {
|
||||
box.lineTo(c[i] * r.zoom + r.panX, c[i + 1] * r.zoom + r.panY)
|
||||
}
|
||||
box.close()
|
||||
const immutableBox = box.detachAndDelete()
|
||||
canvas.drawPath(immutableBox, r.auxStroke)
|
||||
immutableBox.delete()
|
||||
}
|
||||
}
|
||||
|
||||
const S = CURSOR_SIZE
|
||||
const path = new r.ck.PathBuilder()
|
||||
path.moveTo(screenX, screenY)
|
||||
path.lineTo(screenX, screenY + S * 1.35)
|
||||
path.lineTo(screenX + S * 0.38, screenY + S * 1.0)
|
||||
path.lineTo(screenX + S * 0.72, screenY + S * 1.5)
|
||||
path.lineTo(screenX + S * 0.92, screenY + S * 1.38)
|
||||
path.lineTo(screenX + S * 0.58, screenY + S * 0.88)
|
||||
path.lineTo(screenX + S * 1.0, screenY + S * 0.82)
|
||||
path.close()
|
||||
|
||||
const immutablePath = path.detachAndDelete()
|
||||
r.auxStroke.setColor(r.ck.Color4f(1, 1, 1, 1))
|
||||
r.auxStroke.setStrokeWidth(2)
|
||||
r.auxStroke.setPathEffect(null)
|
||||
canvas.drawPath(immutablePath, r.auxStroke)
|
||||
|
||||
r.auxFill.setColor(r.ck.Color4f(cr, g, b, 1))
|
||||
canvas.drawPath(immutablePath, r.auxFill)
|
||||
immutablePath.delete()
|
||||
|
||||
const provider = r.fontProvider
|
||||
if (cursor.name && provider) {
|
||||
const pillX = screenX + LABEL_OFFSET_X - LABEL_PADDING_X
|
||||
const pillY = screenY + LABEL_OFFSET_Y - LABEL_FONT_SIZE - LABEL_PADDING_Y + 2
|
||||
const pillHeight = LABEL_FONT_SIZE + LABEL_PADDING_Y * 2
|
||||
// Shaped like other canvas labels: kerning, fallback fonts, and RTL names.
|
||||
r.labelParagraphCache.use(
|
||||
r.ck,
|
||||
provider,
|
||||
cursor.name,
|
||||
LABEL_FONT_SIZE,
|
||||
LABEL_MAX_WIDTH,
|
||||
r.ck.WHITE,
|
||||
r.fontGeneration,
|
||||
({ paragraph, metrics }) => {
|
||||
r.auxFill.setColor(r.ck.Color4f(cr, g, b, 1))
|
||||
const pill = r.ck.RRectXY(
|
||||
r.ck.XYWHRect(pillX, pillY, metrics.width + LABEL_PADDING_X * 2, pillHeight),
|
||||
4,
|
||||
4
|
||||
)
|
||||
canvas.drawRRect(pill, r.auxFill)
|
||||
canvas.drawParagraph(
|
||||
paragraph,
|
||||
pillX + LABEL_PADDING_X,
|
||||
pillY + (pillHeight - metrics.height) / 2
|
||||
)
|
||||
}
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -36,7 +36,7 @@ import * as RendererState from './renderer/state'
|
|||
import * as RenderText from './text'
|
||||
import { createGlyphSilhouetteCache } from './text/derived'
|
||||
import { TextPreparationCache } from './text/preparation-cache'
|
||||
export type { MeasurementMode, RenderOverlays, RulerTheme } from './renderer/types'
|
||||
export type { MeasurementMode, PresenceCursor, RenderOverlays, RulerTheme } from './renderer/types'
|
||||
import type {
|
||||
Image as CKImage,
|
||||
Path,
|
||||
|
|
@ -70,7 +70,7 @@ export interface PendingFontNode {
|
|||
import { EffectRasterCache } from './renderer/effect-raster-cache'
|
||||
import { TiledSceneController } from './renderer/tiles'
|
||||
import type { TransientCanvasPreview } from './renderer/transient-previews'
|
||||
import type { RenderOverlays, RulerTheme } from './renderer/types'
|
||||
import type { PresenceCursor, RenderOverlays, RulerTheme } from './renderer/types'
|
||||
|
||||
export class SkiaRenderer {
|
||||
ck: CanvasKit
|
||||
|
|
@ -266,10 +266,10 @@ export class SkiaRenderer {
|
|||
editState?: RenderOverlays['nodeEditState']
|
||||
) => void
|
||||
declare drawPenOverlay: (canvas: Canvas, penState: RenderOverlays['penState']) => void
|
||||
declare drawRemoteCursors: (
|
||||
declare drawPresenceCursors: (
|
||||
canvas: Canvas,
|
||||
graph: SceneGraph,
|
||||
cursors?: RenderOverlays['remoteCursors']
|
||||
cursors?: PresenceCursor[]
|
||||
) => void
|
||||
declare drawRulers: (
|
||||
canvas: Canvas,
|
||||
|
|
|
|||
|
|
@ -13,7 +13,7 @@ import * as Overlays from '#core/canvas/overlays'
|
|||
import * as AIOverlays from '#core/canvas/overlays/ai'
|
||||
import * as PenOverlay from '#core/canvas/pen-overlay'
|
||||
import type { SkiaRenderer } from '#core/canvas/renderer'
|
||||
import type { RenderOverlays } from '#core/canvas/renderer/types'
|
||||
import type { PresenceCursor, RenderOverlays } from '#core/canvas/renderer/types'
|
||||
import * as Rulers from '#core/canvas/rulers'
|
||||
import * as SceneRender from '#core/canvas/scene'
|
||||
import { renderEffects as renderShadowEffects } from '#core/canvas/shadows'
|
||||
|
|
@ -162,12 +162,8 @@ const rendererMethods: ThisType<SkiaRenderer> = {
|
|||
PenOverlay.drawPenOverlay(this, canvas, penState)
|
||||
},
|
||||
|
||||
drawRemoteCursors(
|
||||
canvas: Canvas,
|
||||
graph: SceneGraph,
|
||||
cursors?: RenderOverlays['remoteCursors']
|
||||
): void {
|
||||
PenOverlay.drawRemoteCursors(this, canvas, graph, cursors)
|
||||
drawPresenceCursors(canvas: Canvas, graph: SceneGraph, cursors?: PresenceCursor[]): void {
|
||||
Overlays.drawPresenceCursors(this, canvas, graph, cursors)
|
||||
},
|
||||
|
||||
drawRulers(
|
||||
|
|
|
|||
|
|
@ -59,7 +59,7 @@ export function drawOverlayPass(
|
|||
if (!measuring) r.drawAutoLayoutHover(canvas, graph, overlays.autoLayoutHover)
|
||||
r.drawNodeEditOverlay(canvas, graph, overlays.nodeEditState)
|
||||
r.drawPenOverlay(canvas, overlays.penState)
|
||||
r.drawRemoteCursors(canvas, graph, overlays.remoteCursors)
|
||||
r.drawPresenceCursors(canvas, graph, overlays.presenceCursors)
|
||||
}
|
||||
|
||||
export function drawChromePass(
|
||||
|
|
|
|||
|
|
@ -78,7 +78,7 @@ export function renderFromEditorState(
|
|||
} as RenderOverlays['penState'])
|
||||
: null,
|
||||
nodeEditState: state.nodeEditState ?? null,
|
||||
remoteCursors: state.remoteCursors,
|
||||
presenceCursors: state.presenceCursors,
|
||||
autoLayoutHover: state.autoLayoutHover
|
||||
},
|
||||
state.sceneVersion,
|
||||
|
|
|
|||
|
|
@ -15,6 +15,17 @@ export interface RulerTheme {
|
|||
|
||||
export type MeasurementMode = 'off' | 'shallow' | 'deep'
|
||||
|
||||
/** Where a collaborator or an agent is working, in world coordinates. */
|
||||
export interface PresenceCursor {
|
||||
kind: 'person' | 'agent'
|
||||
name: string
|
||||
/** A person's color, or the color of the person who runs the agent. */
|
||||
color: Color
|
||||
x: number
|
||||
y: number
|
||||
selection?: string[]
|
||||
}
|
||||
|
||||
export interface RenderOverlays {
|
||||
hoveredNodeId?: string | null
|
||||
measurementMode?: MeasurementMode
|
||||
|
|
@ -67,11 +78,5 @@ export interface RenderOverlays {
|
|||
selectedHandles?: Set<number>
|
||||
hoveredHandleInfo?: { segmentIndex: number; tangentField: 'tangentStart' | 'tangentEnd' } | null
|
||||
} | null
|
||||
remoteCursors?: Array<{
|
||||
name: string
|
||||
color: Color
|
||||
x: number
|
||||
y: number
|
||||
selection?: string[]
|
||||
}>
|
||||
presenceCursors?: PresenceCursor[]
|
||||
}
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@ export function createDefaultEditorSharedState(): EditorSharedState {
|
|||
return {
|
||||
activeTool: 'SELECT',
|
||||
snappingPreferences: { ...DEFAULT_SNAPPING_PREFERENCES },
|
||||
remoteCursors: [],
|
||||
presenceCursors: [],
|
||||
documentName: 'Untitled',
|
||||
rulerTheme: undefined,
|
||||
sceneVersion: 0
|
||||
|
|
|
|||
|
|
@ -15,7 +15,7 @@ import type { UndoManager } from '@open-pencil/scene-graph/undo'
|
|||
|
||||
import type { GuideOverlayState } from '#core/canvas/guides/types'
|
||||
import type { RulerTheme, SkiaRenderer } from '#core/canvas/renderer'
|
||||
import type { MeasurementMode, RenderOverlays } from '#core/canvas/renderer/types'
|
||||
import type { MeasurementMode, PresenceCursor, RenderOverlays } from '#core/canvas/renderer/types'
|
||||
import type { SnappingPreferences } from '#core/editor/preferences'
|
||||
import type { RotationPreview } from '#core/geometry'
|
||||
import type { TextEditor } from '#core/text/editor'
|
||||
|
|
@ -37,13 +37,7 @@ export type Tool =
|
|||
export interface EditorSharedState {
|
||||
activeTool: Tool
|
||||
snappingPreferences: SnappingPreferences
|
||||
remoteCursors: Array<{
|
||||
name: string
|
||||
color: Color
|
||||
x: number
|
||||
y: number
|
||||
selection?: string[]
|
||||
}>
|
||||
presenceCursors: PresenceCursor[]
|
||||
documentName: string
|
||||
rulerTheme?: RulerTheme
|
||||
sceneVersion: number
|
||||
|
|
|
|||
|
|
@ -20,6 +20,7 @@ Anyone with the link can join. The room stays active as long as at least one par
|
|||
- **Document changes** — every edit (shapes, text, properties, layout) syncs instantly
|
||||
- **Cursors** — see where each collaborator is pointing, with their name and color
|
||||
- **Selections** — highlighted selections are visible to everyone
|
||||
- **Agents** — the built-in AI chat appears as a cursor at the layers it is editing, its outlined label showing a sparkle and a callsign such as *Fern*. The cursor and outline have the color of the person running it, so you can tell whose agent it is. Only its name, kind, model, status, page, position, and edited layers are shared, never prompts or replies.
|
||||
|
||||
## Follow Mode
|
||||
|
||||
|
|
|
|||
54
packages/vue/src/canvas/PresenceCursors.stories.ts
Normal file
54
packages/vue/src/canvas/PresenceCursors.stories.ts
Normal file
|
|
@ -0,0 +1,54 @@
|
|||
import type { Meta, StoryObj } from '@storybook/vue3-vite'
|
||||
|
||||
import PresenceCursors from './examples/PresenceCursors.vue'
|
||||
|
||||
type Args = {
|
||||
person: string
|
||||
personColor: string
|
||||
agent: string
|
||||
otherAgent: string
|
||||
otherColor: string
|
||||
zoom: number
|
||||
}
|
||||
|
||||
const meta = {
|
||||
title: 'Vue SDK/Canvas/Presence Cursors',
|
||||
component: PresenceCursors,
|
||||
parameters: {
|
||||
docs: {
|
||||
description: {
|
||||
component:
|
||||
'People draw as arrows filled with their color; agents in the color of the person running them, with an outlined label that starts with a sparkle, and what they edit outlined.'
|
||||
}
|
||||
}
|
||||
},
|
||||
args: {
|
||||
person: 'Ana',
|
||||
personColor: '#eb574a',
|
||||
agent: 'Fern',
|
||||
otherAgent: 'Orbit',
|
||||
otherColor: '#338cf2',
|
||||
zoom: 1
|
||||
},
|
||||
argTypes: {
|
||||
personColor: { control: 'color' },
|
||||
otherColor: { control: 'color' },
|
||||
zoom: { control: { type: 'range', min: 0.25, max: 4, step: 0.25 } }
|
||||
}
|
||||
} satisfies Meta<Args>
|
||||
|
||||
export default meta
|
||||
type Story = StoryObj<Meta<Args>>
|
||||
|
||||
export const Room: Story = {}
|
||||
|
||||
export const LongNames: Story = {
|
||||
args: {
|
||||
person: 'A collaborator with a very long display name',
|
||||
agent: 'An agent callsign far longer than any we hand out'
|
||||
}
|
||||
}
|
||||
|
||||
export const ZoomedIn: Story = {
|
||||
args: { zoom: 2 }
|
||||
}
|
||||
53
packages/vue/src/canvas/examples/PresenceCursors.vue
Normal file
53
packages/vue/src/canvas/examples/PresenceCursors.vue
Normal file
|
|
@ -0,0 +1,53 @@
|
|||
<script setup lang="ts">
|
||||
import { watchEffect } from 'vue'
|
||||
|
||||
import { createEditor } from '@open-pencil/core/editor'
|
||||
import { parseColor } from '@open-pencil/scene-graph/color'
|
||||
|
||||
import CanvasRoot from '#vue/canvas/CanvasRoot.vue'
|
||||
import CanvasSurface from '#vue/canvas/CanvasSurface.vue'
|
||||
import { provideEditor } from '#vue/editor/context'
|
||||
|
||||
const { person, personColor, agent, otherAgent, otherColor, zoom } = defineProps<{
|
||||
person: string
|
||||
personColor: string
|
||||
agent: string
|
||||
otherAgent: string
|
||||
otherColor: string
|
||||
zoom: number
|
||||
}>()
|
||||
|
||||
/** The scene's middle in canvas pixels, which zooming keeps in place. */
|
||||
const CENTER = { x: 300, y: 200 }
|
||||
|
||||
const editor = createEditor()
|
||||
const card = editor.graph.createNode('RECTANGLE', editor.state.currentPageId, {
|
||||
name: 'Card',
|
||||
x: 220,
|
||||
y: 200,
|
||||
width: 160,
|
||||
height: 100,
|
||||
fills: [{ type: 'SOLID', color: parseColor('#e8e8eb'), opacity: 1, visible: true }]
|
||||
})
|
||||
provideEditor(editor)
|
||||
|
||||
watchEffect(() => {
|
||||
const owner = parseColor(personColor)
|
||||
const other = parseColor(otherColor)
|
||||
editor.state.panX = CENTER.x * (1 - zoom)
|
||||
editor.state.panY = CENTER.y * (1 - zoom)
|
||||
editor.state.zoom = zoom
|
||||
editor.state.presenceCursors = [
|
||||
{ kind: 'person', name: person, color: owner, x: 120, y: 100 },
|
||||
{ kind: 'agent', name: agent, color: owner, x: 220, y: 200, selection: [card.id] },
|
||||
{ kind: 'agent', name: otherAgent, color: other, x: 430, y: 150 }
|
||||
]
|
||||
editor.requestRender()
|
||||
})
|
||||
</script>
|
||||
|
||||
<template>
|
||||
<CanvasRoot :show-rulers="false">
|
||||
<CanvasSurface class="block h-[400px] w-[600px]" />
|
||||
</CanvasRoot>
|
||||
</template>
|
||||
|
|
@ -1,4 +1,5 @@
|
|||
import { Chat } from '@ai-sdk/vue'
|
||||
import { useEventListener } from '@vueuse/core'
|
||||
import { createUIMessageStream, DirectChatTransport, stepCountIs, ToolLoopAgent } from 'ai'
|
||||
import type {
|
||||
ChatTransport,
|
||||
|
|
@ -21,7 +22,7 @@ import { chatThinkingLevel } from '@/app/ai/chat/thinking'
|
|||
import { createAIModelRuntime, resolveModelConnectionAPIKey } from '@/app/ai/models'
|
||||
import type { ThinkingLevel } from '@/app/ai/models/types'
|
||||
import { createCanvasJSXPreview } from '@/app/ai/preview/canvas'
|
||||
import { createAITools, recordStep, runPageId, startRun } from '@/app/ai/tools'
|
||||
import { createAITools, endRun, recordStep, runPageId, startRun } from '@/app/ai/tools'
|
||||
import { enabledAIToolDefinitions } from '@/app/ai/tools/catalog'
|
||||
import { aiToolOverrides } from '@/app/ai/tools/preferences'
|
||||
import {
|
||||
|
|
@ -129,7 +130,7 @@ export function createToolLoopTransport({
|
|||
enabledAIToolDefinitions(aiToolOverrides.value).map((tool) => tool.name)
|
||||
)
|
||||
preview.clear()
|
||||
startRun(store, stepLimit)
|
||||
startRun(store, stepLimit, effectiveModelID)
|
||||
return {
|
||||
...options,
|
||||
stopWhen: stepCountIs(stepLimit),
|
||||
|
|
@ -139,7 +140,10 @@ export function createToolLoopTransport({
|
|||
...callSettings(providerID, cacheProviderOptions, thinkingLevel())
|
||||
}
|
||||
},
|
||||
onFinish: () => preview.clear(),
|
||||
onFinish: () => {
|
||||
preview.clear()
|
||||
endRun(store)
|
||||
},
|
||||
onStepFinish: ({ usage }) => {
|
||||
preview.clear()
|
||||
recordStep(store)
|
||||
|
|
@ -159,6 +163,7 @@ export function createToolLoopTransport({
|
|||
|
||||
function handleError(error: unknown): string {
|
||||
preview.clear()
|
||||
endRun(store)
|
||||
onError?.(error)
|
||||
return 'The provider rejected the request.'
|
||||
}
|
||||
|
|
@ -169,6 +174,8 @@ export function createToolLoopTransport({
|
|||
return resumableTransport({
|
||||
reconnectToStream: (options) => transport.reconnectToStream(options),
|
||||
async sendMessages(options) {
|
||||
// Stopping a reply ends the run without onFinish.
|
||||
if (options.abortSignal) useEventListener(options.abortSignal, 'abort', () => endRun(store))
|
||||
// DirectChatTransport handles error chunks, but not a rejected underlying stream.
|
||||
return createUIMessageStream<UIMessage>({
|
||||
execute: async ({ writer }) => {
|
||||
|
|
|
|||
|
|
@ -10,9 +10,9 @@ import { ensureGraphFonts } from '@/app/editor/fonts'
|
|||
import { useLibraryService } from '@/app/libraries'
|
||||
|
||||
import { aiToolDefinitions } from './catalog'
|
||||
import { moveRunToPage, runPageId, stepBudget } from './run'
|
||||
import { markRunWork, moveRunToPage, runPageId, stepBudget } from './run'
|
||||
|
||||
export { didHitStepLimit, recordStep, runPageId, startRun } from './run'
|
||||
export { didHitStepLimit, endRun, recordStep, runPageId, startRun } from './run'
|
||||
|
||||
export function createAITools(store: EditorStore, diagnosticContext?: AIDiagnosticContext) {
|
||||
let before: { pageId: string; snapshot: ReturnType<EditorStore['snapshotPage']> } | null = null
|
||||
|
|
@ -62,6 +62,7 @@ export function createAITools(store: EditorStore, diagnosticContext?: AIDiagnost
|
|||
}
|
||||
},
|
||||
onFlashNodes: (nodeIds) => {
|
||||
markRunWork(store, nodeIds)
|
||||
store.renderer?.aiClearActive()
|
||||
if (nodeIds.length > 0) {
|
||||
store.aiFlashDone(nodeIds)
|
||||
|
|
|
|||
|
|
@ -1,8 +1,10 @@
|
|||
import { computeContentBounds } from '@open-pencil/core/io'
|
||||
import type { StepBudget } from '@open-pencil/core/tools'
|
||||
|
||||
import { DEFAULT_AGENT_STEPS, resolveAgentStepLimit } from '@/app/ai/chat/step-limit'
|
||||
import { getActiveEditorStore } from '@/app/editor/active-store'
|
||||
import type { EditorStore } from '@/app/editor/active-store'
|
||||
import { addAgent, type AgentHandle } from '@/app/presence/registry'
|
||||
|
||||
class RunState {
|
||||
currentSteps = 0
|
||||
|
|
@ -10,6 +12,8 @@ class RunState {
|
|||
maxSteps = DEFAULT_AGENT_STEPS
|
||||
/** The page the run works on. The user's navigation does not move it; the agent's does. */
|
||||
pageId: string | null = null
|
||||
/** The built-in chat as other people see it; one per document, kept between replies. */
|
||||
agent: AgentHandle | null = null
|
||||
|
||||
start(maxSteps: number, pageId: string): void {
|
||||
this.currentSteps = 0
|
||||
|
|
@ -34,8 +38,28 @@ function getRunState(store?: EditorStore): RunState {
|
|||
}
|
||||
|
||||
/** Begin a run on the page the user is viewing. */
|
||||
export function startRun(store: EditorStore, maxSteps: number): void {
|
||||
getRunState(store).start(maxSteps, store.state.currentPageId)
|
||||
export function startRun(store: EditorStore, maxSteps: number, model?: string): void {
|
||||
const run = getRunState(store)
|
||||
run.start(maxSteps, store.state.currentPageId)
|
||||
run.agent ??= addAgent(store, 'chat', model)
|
||||
run.agent.update({ status: 'thinking', model })
|
||||
}
|
||||
|
||||
/** The reply finished, failed, or was stopped: the agent stays listed but leaves the canvas. */
|
||||
export function endRun(store: EditorStore): void {
|
||||
getRunState(store).agent?.update({ status: 'idle', cursor: undefined, selection: undefined })
|
||||
}
|
||||
|
||||
/** Point the run's agent at nodes a tool just created or changed. */
|
||||
export function markRunWork(store: EditorStore, nodeIds: string[]): void {
|
||||
const run = getRunState(store)
|
||||
const bounds = computeContentBounds(store.graph, nodeIds)
|
||||
if (!run.agent || !bounds) return
|
||||
run.agent.update({
|
||||
status: 'editing',
|
||||
cursor: { x: bounds.minX, y: bounds.minY, pageId: runPageId(store) },
|
||||
selection: nodeIds
|
||||
})
|
||||
}
|
||||
|
||||
export function recordStep(store?: EditorStore): void {
|
||||
|
|
|
|||
|
|
@ -2,10 +2,10 @@ import { ref } from 'vue'
|
|||
import type * as awarenessProtocol from 'y-protocols/awareness'
|
||||
|
||||
import { randomIndex } from '@open-pencil/core/random'
|
||||
import type { Color } from '@open-pencil/scene-graph/primitives'
|
||||
|
||||
import type { EditorStore } from '@/app/editor/active-store'
|
||||
import { PEER_COLORS, ROOM_ID_CHARS, ROOM_ID_LENGTH } from '@/constants'
|
||||
import { parsePeer } from '@/app/presence/schema'
|
||||
import { ROOM_ID_CHARS, ROOM_ID_LENGTH } from '@/constants'
|
||||
|
||||
import type { RemotePeer } from './types'
|
||||
|
||||
|
|
@ -18,41 +18,14 @@ type CursorState = {
|
|||
zoom?: number
|
||||
}
|
||||
|
||||
/** People in the room other than us, validated: awareness comes from other browsers. */
|
||||
export function buildRemotePeers(
|
||||
states: Map<number, Record<string, unknown>>,
|
||||
localClientId: number
|
||||
): RemotePeer[] {
|
||||
const peers: RemotePeer[] = []
|
||||
|
||||
states.forEach((peerState, clientId) => {
|
||||
if (clientId === localClientId) return
|
||||
const user = peerState.user as { name?: string; color?: Color } | undefined
|
||||
if (!user) return
|
||||
peers.push({
|
||||
clientId,
|
||||
name: user.name || 'Anonymous',
|
||||
color: user.color || PEER_COLORS[clientId % PEER_COLORS.length],
|
||||
cursor: peerState.cursor as RemotePeer['cursor'],
|
||||
selection: peerState.selection as string[]
|
||||
})
|
||||
})
|
||||
|
||||
return peers
|
||||
}
|
||||
|
||||
export function remotePeersToCursors(peers: RemotePeer[], currentPageId: string) {
|
||||
return peers
|
||||
.filter((p) => p.cursor && p.cursor.pageId === currentPageId)
|
||||
.map((p) => {
|
||||
const cursor = p.cursor as NonNullable<RemotePeer['cursor']>
|
||||
return {
|
||||
name: p.name,
|
||||
color: p.color,
|
||||
x: cursor.x,
|
||||
y: cursor.y,
|
||||
selection: p.selection
|
||||
}
|
||||
})
|
||||
return [...states].flatMap(([clientId, state]) =>
|
||||
clientId === localClientId ? [] : (parsePeer(clientId, state) ?? [])
|
||||
)
|
||||
}
|
||||
|
||||
export function createFollowActions(
|
||||
|
|
|
|||
|
|
@ -1,9 +1,12 @@
|
|||
import type { Ref } from 'vue'
|
||||
import { watch, type Ref } from 'vue'
|
||||
import type { Awareness } from 'y-protocols/awareness'
|
||||
|
||||
import { buildRemotePeers, remotePeersToCursors } from '@/app/collab/awareness'
|
||||
import type { Color } from '@open-pencil/scene-graph/primitives'
|
||||
|
||||
import { buildRemotePeers } from '@/app/collab/awareness'
|
||||
import type { CollabState } from '@/app/collab/types'
|
||||
import type { EditorStore } from '@/app/editor/active-store'
|
||||
import { presenceOf, setOwnerColor, setPeers } from '@/app/presence/registry'
|
||||
|
||||
type LocalAwarenessOptions = {
|
||||
state: Ref<CollabState>
|
||||
|
|
@ -50,8 +53,7 @@ export function createLocalAwarenessActions({
|
|||
)
|
||||
|
||||
state.value.peers = peers
|
||||
store.state.remoteCursors = remotePeersToCursors(peers, store.state.currentPageId)
|
||||
store.requestRender()
|
||||
setPeers(store, peers)
|
||||
}
|
||||
|
||||
function setLocalName(name: string) {
|
||||
|
|
@ -62,3 +64,21 @@ export function createLocalAwarenessActions({
|
|||
|
||||
return { broadcastAwareness, updateCursor, updateSelection, updatePeersList, setLocalName }
|
||||
}
|
||||
|
||||
/** Publish our agents to the room while connected; they take our collaborator color. */
|
||||
export function publishLocalAgents(
|
||||
store: EditorStore,
|
||||
getAwareness: () => Awareness | null,
|
||||
color: Color
|
||||
): () => void {
|
||||
setOwnerColor(store, color)
|
||||
const stop = watch(
|
||||
presenceOf(store).agents,
|
||||
(agents) => getAwareness()?.setLocalStateField('agents', agents),
|
||||
{ immediate: true }
|
||||
)
|
||||
return () => {
|
||||
stop()
|
||||
setOwnerColor(store, null)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -6,11 +6,13 @@ import * as Y from 'yjs'
|
|||
|
||||
import { randomIndex } from '@open-pencil/core/random'
|
||||
|
||||
import { publishLocalAgents } from '@/app/collab/local-awareness'
|
||||
import { connectCollabRoom } from '@/app/collab/room'
|
||||
import type { CollabRoomTransport } from '@/app/collab/transport'
|
||||
import type { CollabState } from '@/app/collab/types'
|
||||
import { bindCollabGraphEvents, registerYjsObservers } from '@/app/collab/yjs-sync'
|
||||
import type { EditorStore } from '@/app/editor/active-store'
|
||||
import { setPeers } from '@/app/presence/registry'
|
||||
import { PEER_COLORS } from '@/constants'
|
||||
|
||||
export type CollabRuntime = {
|
||||
|
|
@ -25,6 +27,7 @@ export type CollabRuntime = {
|
|||
suppressYjsEvents: boolean
|
||||
unbindGraphEvents: (() => void) | null
|
||||
stopZoomWatch: (() => void) | null
|
||||
stopAgentSync: (() => void) | null
|
||||
}
|
||||
|
||||
type ConnectCollabSessionOptions = {
|
||||
|
|
@ -60,6 +63,7 @@ type CollabSessionResources = {
|
|||
ydoc: Y.Doc | null
|
||||
unbindGraphEvents: (() => void) | null
|
||||
stopZoomWatch: (() => void) | null
|
||||
stopAgentSync: (() => void) | null
|
||||
resetFollow: () => void
|
||||
}
|
||||
|
||||
|
|
@ -75,7 +79,8 @@ export function createCollabRuntime(): CollabRuntime {
|
|||
suppressGraphSync: false,
|
||||
suppressYjsEvents: false,
|
||||
unbindGraphEvents: null,
|
||||
stopZoomWatch: null
|
||||
stopZoomWatch: null,
|
||||
stopAgentSync: null
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -125,6 +130,7 @@ export function createCollabConnectionActions({
|
|||
ydoc: runtime.ydoc,
|
||||
unbindGraphEvents: runtime.unbindGraphEvents,
|
||||
stopZoomWatch: runtime.stopZoomWatch,
|
||||
stopAgentSync: runtime.stopAgentSync,
|
||||
resetFollow
|
||||
})
|
||||
resetCollabRuntime(runtime)
|
||||
|
|
@ -199,6 +205,7 @@ export function connectCollabSession({
|
|||
broadcastAwareness()
|
||||
|
||||
runtime.stopZoomWatch = watchAwarenessZoom(store, () => runtime.awareness)
|
||||
runtime.stopAgentSync = publishLocalAgents(store, () => runtime.awareness, state.value.localColor)
|
||||
|
||||
runtime.unbindGraphEvents = bindCollabGraphEvents({
|
||||
store,
|
||||
|
|
@ -215,6 +222,7 @@ export function connectCollabSession({
|
|||
export function resetCollabRuntime(runtime: CollabRuntime) {
|
||||
runtime.unbindGraphEvents = null
|
||||
runtime.stopZoomWatch = null
|
||||
runtime.stopAgentSync = null
|
||||
runtime.room = null
|
||||
runtime.awareness = null
|
||||
runtime.persistence = null
|
||||
|
|
@ -239,7 +247,7 @@ export function disposeCollabSessionResources(resources: CollabSessionResources)
|
|||
void resources.persistence.destroy()
|
||||
}
|
||||
resources.ydoc?.destroy()
|
||||
resources.stopAgentSync?.()
|
||||
resources.resetFollow()
|
||||
resources.store.state.remoteCursors = []
|
||||
resources.store.requestRender()
|
||||
setPeers(resources.store, [])
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,11 +1,15 @@
|
|||
import type { Color } from '@open-pencil/scene-graph/primitives'
|
||||
|
||||
import type { AgentPresence, PresencePoint } from '@/app/presence/types'
|
||||
|
||||
export interface RemotePeer {
|
||||
clientId: number
|
||||
name: string
|
||||
color: Color
|
||||
cursor?: { x: number; y: number; pageId: string }
|
||||
cursor?: PresencePoint
|
||||
selection?: string[]
|
||||
/** Agents this person runs, as they publish them. */
|
||||
agents: AgentPresence[]
|
||||
}
|
||||
|
||||
export interface CollabState {
|
||||
|
|
|
|||
48
src/app/presence/callsigns.ts
Normal file
48
src/app/presence/callsigns.ts
Normal file
|
|
@ -0,0 +1,48 @@
|
|||
import { randomIndex } from '@open-pencil/core/random'
|
||||
|
||||
/** Short, distinct names that are easy to say: "ask Fern to align the cards". */
|
||||
const CALLSIGNS = [
|
||||
'Fern',
|
||||
'Orbit',
|
||||
'Pixel',
|
||||
'Juniper',
|
||||
'Comet',
|
||||
'Maple',
|
||||
'Nova',
|
||||
'Pebble',
|
||||
'Quill',
|
||||
'Sable',
|
||||
'Tango',
|
||||
'Willow',
|
||||
'Echo',
|
||||
'Indigo',
|
||||
'Kite',
|
||||
'Lumen',
|
||||
'Mosaic',
|
||||
'Nimbus',
|
||||
'Opal',
|
||||
'Pico',
|
||||
'Rune',
|
||||
'Sorrel',
|
||||
'Tidal',
|
||||
'Vesper',
|
||||
'Wren',
|
||||
'Zephyr',
|
||||
'Atlas',
|
||||
'Basil',
|
||||
'Cinder',
|
||||
'Dune',
|
||||
'Ember',
|
||||
'Flint'
|
||||
] as const
|
||||
|
||||
/** A callsign not in `taken`, picked at random so peers rarely choose the same one. */
|
||||
export function pickCallsign(taken: ReadonlySet<string>): string {
|
||||
const free = CALLSIGNS.filter((name) => !taken.has(name))
|
||||
if (free.length > 0) return free[randomIndex(free.length)] ?? CALLSIGNS[0]
|
||||
// Every callsign is taken: number the first one that is still unique.
|
||||
for (let n = 2; ; n++) {
|
||||
const name = `${CALLSIGNS[0]} ${n}`
|
||||
if (!taken.has(name)) return name
|
||||
}
|
||||
}
|
||||
121
src/app/presence/registry.ts
Normal file
121
src/app/presence/registry.ts
Normal file
|
|
@ -0,0 +1,121 @@
|
|||
import { shallowRef, type ShallowRef } from 'vue'
|
||||
|
||||
import type { PresenceCursor } from '@open-pencil/core/canvas'
|
||||
import { AI_ACTIVE_COLOR } from '@open-pencil/core/constants'
|
||||
import { randomHex } from '@open-pencil/core/random'
|
||||
import type { Color } from '@open-pencil/scene-graph/primitives'
|
||||
|
||||
import type { RemotePeer } from '@/app/collab/types'
|
||||
import type { EditorStore } from '@/app/editor/active-store'
|
||||
|
||||
import { pickCallsign } from './callsigns'
|
||||
import type { AgentKind, AgentPresence } from './types'
|
||||
|
||||
/** Agents' color outside a room, where there is no collaborator color to inherit. */
|
||||
const SOLO_AGENT_COLOR: Color = { ...AI_ACTIVE_COLOR, a: 1 }
|
||||
|
||||
interface Presence {
|
||||
/** Agents running in this app for the document. */
|
||||
agents: ShallowRef<readonly AgentPresence[]>
|
||||
/** People in the room, with their agents. */
|
||||
peers: ShallowRef<readonly RemotePeer[]>
|
||||
/** The local collaborator's color while in a room. */
|
||||
ownerColor: ShallowRef<Color | null>
|
||||
}
|
||||
|
||||
const presences = new WeakMap<EditorStore, Presence>()
|
||||
|
||||
export function presenceOf(store: EditorStore): Presence {
|
||||
const existing = presences.get(store)
|
||||
if (existing) return existing
|
||||
const presence: Presence = {
|
||||
agents: shallowRef([]),
|
||||
peers: shallowRef([]),
|
||||
ownerColor: shallowRef(null)
|
||||
}
|
||||
presences.set(store, presence)
|
||||
store.onEditorEvent('page:changed', () => refreshCursors(store))
|
||||
return presence
|
||||
}
|
||||
|
||||
function agentCursor(agent: AgentPresence, color: Color, pageId: string): PresenceCursor[] {
|
||||
if (agent.status === 'idle' || agent.cursor?.pageId !== pageId) return []
|
||||
const { x, y } = agent.cursor
|
||||
return [{ kind: 'agent', name: agent.name, color, x, y, selection: agent.selection }]
|
||||
}
|
||||
|
||||
/** Draw everyone working on the page on screen: people, their agents, and ours. */
|
||||
export function refreshCursors(store: EditorStore): void {
|
||||
const { agents, peers, ownerColor } = presenceOf(store)
|
||||
const pageId = store.state.currentPageId
|
||||
store.state.presenceCursors = [
|
||||
...peers.value.flatMap((peer): PresenceCursor[] => [
|
||||
...(peer.cursor?.pageId === pageId
|
||||
? [
|
||||
{
|
||||
kind: 'person' as const,
|
||||
name: peer.name,
|
||||
color: peer.color,
|
||||
...peer.cursor,
|
||||
selection: peer.selection
|
||||
}
|
||||
]
|
||||
: []),
|
||||
...peer.agents.flatMap((agent) => agentCursor(agent, peer.color, pageId))
|
||||
]),
|
||||
...agents.value.flatMap((agent) =>
|
||||
agentCursor(agent, ownerColor.value ?? SOLO_AGENT_COLOR, pageId)
|
||||
)
|
||||
]
|
||||
store.requestRepaint()
|
||||
}
|
||||
|
||||
export function setPeers(store: EditorStore, peers: readonly RemotePeer[]): void {
|
||||
presenceOf(store).peers.value = peers
|
||||
refreshCursors(store)
|
||||
}
|
||||
|
||||
export function setOwnerColor(store: EditorStore, color: Color | null): void {
|
||||
presenceOf(store).ownerColor.value = color
|
||||
refreshCursors(store)
|
||||
}
|
||||
|
||||
export interface AgentHandle {
|
||||
readonly id: string
|
||||
readonly name: string
|
||||
update: (patch: Partial<Omit<AgentPresence, 'id' | 'name' | 'kind'>>) => void
|
||||
remove: () => void
|
||||
}
|
||||
|
||||
/** Announce an agent working in this document under a callsign unique among visible agents. */
|
||||
export function addAgent(store: EditorStore, kind: AgentKind, model?: string): AgentHandle {
|
||||
const presence = presenceOf(store)
|
||||
const taken = new Set([
|
||||
...presence.agents.value.map((agent) => agent.name),
|
||||
...presence.peers.value.flatMap((peer) => peer.agents.map((agent) => agent.name))
|
||||
])
|
||||
const agent: AgentPresence = {
|
||||
id: randomHex(8),
|
||||
name: pickCallsign(taken),
|
||||
kind,
|
||||
model,
|
||||
status: 'idle'
|
||||
}
|
||||
presence.agents.value = [...presence.agents.value, agent]
|
||||
const replace = (next: AgentPresence | null) => {
|
||||
const agents = presence.agents.value
|
||||
presence.agents.value = next
|
||||
? agents.map((entry) => (entry.id === agent.id ? next : entry))
|
||||
: agents.filter((entry) => entry.id !== agent.id)
|
||||
refreshCursors(store)
|
||||
}
|
||||
return {
|
||||
id: agent.id,
|
||||
name: agent.name,
|
||||
update: (patch) => {
|
||||
const current = presence.agents.value.find((entry) => entry.id === agent.id)
|
||||
if (current) replace({ ...current, ...patch })
|
||||
},
|
||||
remove: () => replace(null)
|
||||
}
|
||||
}
|
||||
60
src/app/presence/schema.ts
Normal file
60
src/app/presence/schema.ts
Normal file
|
|
@ -0,0 +1,60 @@
|
|||
import * as v from 'valibot'
|
||||
|
||||
import type { RemotePeer } from '@/app/collab/types'
|
||||
import { PEER_COLORS } from '@/constants'
|
||||
|
||||
import { AGENT_KINDS, AGENT_STATUSES } from './types'
|
||||
|
||||
/** Bounds on what a peer can publish, so a broken or hostile peer cannot flood rendering. */
|
||||
export const MAX_AGENTS_PER_PEER = 16
|
||||
const MAX_SELECTION = 256
|
||||
const MAX_NAME = 40
|
||||
const MAX_ID = 64
|
||||
|
||||
const finite = v.pipe(v.number(), v.finite())
|
||||
const unit = v.pipe(finite, v.minValue(0), v.maxValue(1))
|
||||
const id = v.pipe(v.string(), v.maxLength(MAX_ID))
|
||||
const name = v.pipe(v.string(), v.trim(), v.minLength(1), v.maxLength(MAX_NAME))
|
||||
/** Optional fields fall back to absent instead of discarding the whole peer. */
|
||||
const lenient = <T extends v.GenericSchema>(schema: T) => v.fallback(v.optional(schema), undefined)
|
||||
|
||||
const color = v.object({ r: unit, g: unit, b: unit, a: v.fallback(unit, 1) })
|
||||
const point = v.object({ x: finite, y: finite, pageId: id })
|
||||
const selection = v.pipe(v.array(id), v.maxLength(MAX_SELECTION))
|
||||
|
||||
const agent = v.object({
|
||||
id,
|
||||
name,
|
||||
kind: v.picklist(AGENT_KINDS),
|
||||
model: lenient(v.pipe(v.string(), v.maxLength(MAX_NAME))),
|
||||
status: v.picklist(AGENT_STATUSES),
|
||||
cursor: lenient(point),
|
||||
selection: lenient(selection)
|
||||
})
|
||||
|
||||
const peerState = v.object({
|
||||
user: v.object({ name: lenient(name), color: lenient(color) }),
|
||||
cursor: lenient(point),
|
||||
selection: lenient(selection),
|
||||
agents: v.fallback(
|
||||
v.optional(
|
||||
v.pipe(v.array(v.fallback(v.nullable(agent), null)), v.maxLength(MAX_AGENTS_PER_PEER))
|
||||
),
|
||||
[]
|
||||
)
|
||||
})
|
||||
|
||||
/** Validate a peer's awareness state; awareness comes from other people's browsers. */
|
||||
export function parsePeer(clientId: number, state: unknown): RemotePeer | null {
|
||||
const result = v.safeParse(peerState, state)
|
||||
if (!result.success) return null
|
||||
const { user, cursor, selection: selected, agents } = result.output
|
||||
return {
|
||||
clientId,
|
||||
name: user.name ?? 'Anonymous',
|
||||
color: user.color ?? PEER_COLORS[clientId % PEER_COLORS.length],
|
||||
cursor,
|
||||
selection: selected,
|
||||
agents: (agents ?? []).flatMap((entry) => (entry ? [entry] : []))
|
||||
}
|
||||
}
|
||||
24
src/app/presence/types.ts
Normal file
24
src/app/presence/types.ts
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
export const AGENT_KINDS = ['chat', 'acp', 'harness', 'mcp'] as const
|
||||
export type AgentKind = (typeof AGENT_KINDS)[number]
|
||||
|
||||
export const AGENT_STATUSES = ['thinking', 'editing', 'idle'] as const
|
||||
export type AgentStatus = (typeof AGENT_STATUSES)[number]
|
||||
|
||||
export interface PresencePoint {
|
||||
x: number
|
||||
y: number
|
||||
pageId: string
|
||||
}
|
||||
|
||||
/** An agent as its owner publishes it: metadata only, never prompts or tool arguments. */
|
||||
export interface AgentPresence {
|
||||
id: string
|
||||
/** A callsign, unique among the agents in a room, such as "Fern". */
|
||||
name: string
|
||||
kind: AgentKind
|
||||
model?: string
|
||||
status: AgentStatus
|
||||
/** Where the agent last worked, derived from the nodes it touched. */
|
||||
cursor?: PresencePoint
|
||||
selection?: string[]
|
||||
}
|
||||
|
|
@ -12,6 +12,7 @@ import { runPageId } from '@/app/ai/tools'
|
|||
import { aiToolOverrides } from '@/app/ai/tools/preferences'
|
||||
import * as figmaFactory from '@/app/automation/bridge/figma-factory'
|
||||
import { createEditorStore } from '@/app/editor/session/create'
|
||||
import { presenceOf } from '@/app/presence/registry'
|
||||
import { appPreferences } from '@/app/settings/preferences/store'
|
||||
|
||||
import { MOCK_USAGE } from '#tests/helpers/chat/usage'
|
||||
|
|
@ -144,3 +145,22 @@ test('a run falls back to the page on screen when its page is deleted', async ()
|
|||
expect(runPageId(store)).toBe(b)
|
||||
})
|
||||
})
|
||||
|
||||
test("shows the chat's agent where its tools work, then takes it off the canvas", async () => {
|
||||
await withStore(async (store, { a }) => {
|
||||
let during: ReturnType<typeof presenceOf>['agents']['value'] = []
|
||||
await runMessage(store, [
|
||||
rectangle,
|
||||
async (editor) => {
|
||||
during = presenceOf(editor).agents.value
|
||||
}
|
||||
])
|
||||
const [shape] = store.graph.getChildren(a)
|
||||
expect(during).toMatchObject([
|
||||
{ kind: 'chat', status: 'editing', cursor: { x: shape?.x, y: shape?.y, pageId: a } }
|
||||
])
|
||||
expect(presenceOf(store).agents.value).toMatchObject([
|
||||
{ name: during[0]?.name, status: 'idle', cursor: undefined }
|
||||
])
|
||||
})
|
||||
})
|
||||
|
|
|
|||
68
tests/app/presence/registry.test.ts
Normal file
68
tests/app/presence/registry.test.ts
Normal file
|
|
@ -0,0 +1,68 @@
|
|||
import 'fake-indexeddb/auto'
|
||||
import { afterEach, expect, test } from 'bun:test'
|
||||
|
||||
import { createEditorStore } from '@/app/editor/session/create'
|
||||
import { addAgent, setOwnerColor, setPeers } from '@/app/presence/registry'
|
||||
|
||||
const stores: ReturnType<typeof createEditorStore>[] = []
|
||||
afterEach(() => {
|
||||
for (const store of stores.splice(0)) store.dispose()
|
||||
})
|
||||
|
||||
function setup() {
|
||||
const store = createEditorStore()
|
||||
stores.push(store)
|
||||
return { store, pageId: store.state.currentPageId, other: store.graph.addPage('Other').id }
|
||||
}
|
||||
|
||||
const red = { r: 1, g: 0, b: 0, a: 1 }
|
||||
|
||||
test('draws active agents on their page, in their owner color', () => {
|
||||
const { store, pageId, other } = setup()
|
||||
const here = addAgent(store, 'chat')
|
||||
const there = addAgent(store, 'chat')
|
||||
here.update({ status: 'editing', cursor: { x: 5, y: 6, pageId } })
|
||||
there.update({ status: 'editing', cursor: { x: 1, y: 1, pageId: other } })
|
||||
setOwnerColor(store, red)
|
||||
expect(store.state.presenceCursors).toEqual([
|
||||
{ kind: 'agent', name: here.name, color: red, x: 5, y: 6, selection: undefined }
|
||||
])
|
||||
})
|
||||
|
||||
test('hides idle agents and gives each a different callsign', () => {
|
||||
const { store, pageId } = setup()
|
||||
const agents = Array.from({ length: 5 }, () => addAgent(store, 'chat'))
|
||||
expect(new Set(agents.map((agent) => agent.name)).size).toBe(5)
|
||||
agents[0]?.update({ status: 'idle', cursor: { x: 0, y: 0, pageId } })
|
||||
expect(store.state.presenceCursors).toEqual([])
|
||||
})
|
||||
|
||||
test('draws people and their agents with the person color', () => {
|
||||
const { store, pageId } = setup()
|
||||
setPeers(store, [
|
||||
{
|
||||
clientId: 3,
|
||||
name: 'Ana',
|
||||
color: red,
|
||||
cursor: { x: 1, y: 2, pageId },
|
||||
agents: [
|
||||
{ id: 'a', name: 'Orbit', kind: 'mcp', status: 'thinking', cursor: { x: 3, y: 4, pageId } }
|
||||
]
|
||||
}
|
||||
])
|
||||
expect(
|
||||
store.state.presenceCursors.map(({ kind, name, color }) => ({ kind, name, color }))
|
||||
).toEqual([
|
||||
{ kind: 'person', name: 'Ana', color: red },
|
||||
{ kind: 'agent', name: 'Orbit', color: red }
|
||||
])
|
||||
})
|
||||
|
||||
test('follows the page on screen', async () => {
|
||||
const { store, other } = setup()
|
||||
addAgent(store, 'chat').update({ status: 'editing', cursor: { x: 1, y: 1, pageId: other } })
|
||||
expect(store.state.presenceCursors).toHaveLength(0)
|
||||
store.preparationController.acknowledgePresentation(Number.MAX_SAFE_INTEGER)
|
||||
await store.switchPage(other)
|
||||
expect(store.state.presenceCursors).toHaveLength(1)
|
||||
})
|
||||
49
tests/app/presence/schema.test.ts
Normal file
49
tests/app/presence/schema.test.ts
Normal file
|
|
@ -0,0 +1,49 @@
|
|||
import { describe, expect, test } from 'bun:test'
|
||||
|
||||
import { MAX_AGENTS_PER_PEER, parsePeer } from '@/app/presence/schema'
|
||||
|
||||
const user = { name: 'Ana', color: { r: 1, g: 0, b: 0, a: 1 } }
|
||||
const agent = {
|
||||
id: 'a1',
|
||||
name: 'Fern',
|
||||
kind: 'chat',
|
||||
status: 'editing',
|
||||
cursor: { x: 10, y: 20, pageId: '0:1' }
|
||||
}
|
||||
|
||||
describe('parsePeer', () => {
|
||||
test('reads a person with their agents', () => {
|
||||
expect(
|
||||
parsePeer(7, { user, cursor: { x: 1, y: 2, pageId: '0:1' }, agents: [agent] })
|
||||
).toMatchObject({
|
||||
clientId: 7,
|
||||
name: 'Ana',
|
||||
cursor: { x: 1, y: 2, pageId: '0:1' },
|
||||
agents: [{ name: 'Fern', status: 'editing', cursor: { x: 10, y: 20 } }]
|
||||
})
|
||||
})
|
||||
|
||||
test('ignores a state without a user', () => {
|
||||
expect(parsePeer(7, { cursor: { x: 1, y: 2, pageId: '0:1' } })).toBeNull()
|
||||
})
|
||||
|
||||
test('drops invalid fields instead of the whole peer', () => {
|
||||
const peer = parsePeer(7, {
|
||||
user: { name: 'Ana', color: 'red' },
|
||||
cursor: { x: Number.NaN, y: 2, pageId: '0:1' },
|
||||
selection: 'everything',
|
||||
agents: [agent, { ...agent, id: 'a2', kind: 'unknown' }, 42]
|
||||
})
|
||||
expect(peer).toMatchObject({ name: 'Ana', cursor: undefined, selection: undefined })
|
||||
expect(peer?.agents.map((entry) => entry.id)).toEqual(['a1'])
|
||||
})
|
||||
|
||||
test('bounds names and agent counts', () => {
|
||||
const tooMany = Array.from({ length: MAX_AGENTS_PER_PEER + 1 }, (_, i) => ({
|
||||
...agent,
|
||||
id: `a${i}`
|
||||
}))
|
||||
expect(parsePeer(7, { user, agents: tooMany })?.agents).toEqual([])
|
||||
expect(parsePeer(7, { user: { name: 'x'.repeat(200) } })?.name).toBe('Anonymous')
|
||||
})
|
||||
})
|
||||
|
|
@ -13,10 +13,16 @@ test('collaborator cursor names are shaped and long names end with an ellipsis',
|
|||
store.state.panX = 0
|
||||
store.state.panY = 0
|
||||
store.state.zoom = 1
|
||||
store.state.remoteCursors = [
|
||||
{ name: 'Orbit', color: blue, x: 80, y: 80 },
|
||||
{ name: 'Ava Yoder', color: red, x: 80, y: 160 },
|
||||
{ name: 'A collaborator with a very long display name', color: blue, x: 80, y: 240 }
|
||||
store.state.presenceCursors = [
|
||||
{ kind: 'person', name: 'Orbit', color: blue, x: 80, y: 80 },
|
||||
{ kind: 'person', name: 'Ava Yoder', color: red, x: 80, y: 160 },
|
||||
{
|
||||
kind: 'person',
|
||||
name: 'A collaborator with a very long display name',
|
||||
color: blue,
|
||||
x: 80,
|
||||
y: 240
|
||||
}
|
||||
]
|
||||
store.requestRender()
|
||||
})
|
||||
|
|
|
|||
40
tests/e2e/canvas/presence-cursors.spec.ts
Normal file
40
tests/e2e/canvas/presence-cursors.spec.ts
Normal file
|
|
@ -0,0 +1,40 @@
|
|||
import { expect, test, useEditorSetupWithClear } from '#tests/e2e/fixtures'
|
||||
|
||||
test.use({ viewport: { width: 900, height: 700 } })
|
||||
|
||||
const editor = useEditorSetupWithClear('/?test&no-chrome&no-rulers')
|
||||
|
||||
test('people and agents show arrows in their color, agents with an outlined sparkle label', async () => {
|
||||
await editor.page.evaluate(() => {
|
||||
const store = window.openPencil?.getStore?.()
|
||||
if (!store) throw new Error('OpenPencil store not initialized')
|
||||
const pageId = store.state.currentPageId
|
||||
const card = store.graph.createNode('RECTANGLE', pageId, {
|
||||
name: 'Card',
|
||||
x: 220,
|
||||
y: 220,
|
||||
width: 160,
|
||||
height: 100,
|
||||
fills: [
|
||||
{ type: 'SOLID', color: { r: 0.9, g: 0.9, b: 0.92, a: 1 }, opacity: 1, visible: true }
|
||||
]
|
||||
})
|
||||
const ana = { r: 0.92, g: 0.34, b: 0.29, a: 1 }
|
||||
const ben = { r: 0.2, g: 0.55, b: 0.95, a: 1 }
|
||||
store.state.panX = 0
|
||||
store.state.panY = 0
|
||||
store.state.zoom = 1
|
||||
store.state.presenceCursors = [
|
||||
{ kind: 'person', name: 'Ana', color: ana, x: 120, y: 120 },
|
||||
{ kind: 'agent', name: 'Fern', color: ana, x: 220, y: 220, selection: [card.id] },
|
||||
{ kind: 'agent', name: 'Orbit', color: ben, x: 440, y: 180 }
|
||||
]
|
||||
store.clearSelection()
|
||||
store.requestRender()
|
||||
})
|
||||
await editor.canvas.waitForRender()
|
||||
editor.canvas.assertNoErrors()
|
||||
expect(await editor.canvas.screenshotCanvasRegion(600, 420)).toMatchSnapshot(
|
||||
'presence-cursors.png'
|
||||
)
|
||||
})
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 11 KiB |
|
|
@ -231,7 +231,7 @@ test('two browser peers synchronize editing, awareness, departure, and reconnect
|
|||
})
|
||||
await expect
|
||||
.poll(() =>
|
||||
host.page.evaluate(() => window.openPencil?.getStore?.().state.remoteCursors.length)
|
||||
host.page.evaluate(() => window.openPencil?.getStore?.().state.presenceCursors.length)
|
||||
)
|
||||
.toBe(1)
|
||||
|
||||
|
|
@ -243,7 +243,7 @@ test('two browser peers synchronize editing, awareness, departure, and reconnect
|
|||
.toBe(0)
|
||||
await expect
|
||||
.poll(() =>
|
||||
host.page.evaluate(() => window.openPencil?.getStore?.().state.remoteCursors.length)
|
||||
host.page.evaluate(() => window.openPencil?.getStore?.().state.presenceCursors.length)
|
||||
)
|
||||
.toBe(0)
|
||||
|
||||
|
|
|
|||
13
tests/e2e/storybook/presence-cursors.spec.ts
Normal file
13
tests/e2e/storybook/presence-cursors.spec.ts
Normal file
|
|
@ -0,0 +1,13 @@
|
|||
import { expect, test } from '@playwright/test'
|
||||
|
||||
test('the canvas story starts CanvasKit and sizes its surface', async ({ page }) => {
|
||||
const errors: string[] = []
|
||||
page.on('pageerror', (error) => errors.push(error.message))
|
||||
await page.goto('/iframe.html?id=vue-sdk-canvas-presence-cursors--room&viewMode=story')
|
||||
const canvas = page.locator('canvas')
|
||||
// A canvas keeps its 300×150 default until CanvasKit creates a surface at its CSS size.
|
||||
await expect
|
||||
.poll(() => canvas.evaluate((element: HTMLCanvasElement) => element.width))
|
||||
.toBeGreaterThan(300)
|
||||
expect(errors).toEqual([])
|
||||
})
|
||||
Loading…
Reference in a new issue