AI Copilot (HITL)

LumiBase Studio ships with an AI Copilot that can take natural-language commands from an admin and execute the skills declared on the CMS. This is the seed of the Agent Harness Layer: the agent receives goal/context/permissions through the harness, calls tools per the registry, is risk-evaluated, and leaves an audit trail. Every dangerous action (touching the schema or deleting data) must go through Human-in-the-Loop (HITL): the AI creates an ai_approvals row awaiting admin approval rather than executing directly.

The historical documentation of the implementation phase (split into 4 modules for the AI agent) is in ai-first-specification.md. The strategic blueprint for expanding the Copilot into a control plane for agents is in agent-harness-layer.md. This file focuses on end-user behavior and the API contract.

The 4-module architecture

Relationship to the Agent Harness Layer

The current AI Copilot handles a short loop: chat → choose a skill → risk/capability check → execute or create an approval. The Agent Harness Layer extends this same principle to a longer loop:

Goal → Run → Plan → Tool calls → Evaluation → Approval → Artifact commit → Memory update

So the existing tables are seen as seeds:

• ai_approvals → the precursor to a general approval gate for plan/tool/artifact.
• ai_conversations + ai_messages → short-term context memory.
• ai_embeddings → the RAG knowledge base by site/collection/item.
• CORE_SKILLS → the precursor to a tool registry with an input schema, capability, rate limit, and risk policy.

CORE_SKILLS registry

Skills are declared centrally in packages/ai-skills/src/skills.ts. Each skill contains:

• name, description (the description is used for LLM tool calling).
• parameters — OpenAI-compatible JSON Schema.
• requiredCapabilities — must be satisfied against the session's capabilities.

Current skills: listCollections, createCollection, deleteCollection, createField, deleteField, listItems, createItem, updateItem, deleteItem.

The getAISkillsAsTools() helper returns an OpenAI function-calling tool list — used when integrating a real LLM.

Risk evaluation rules

A skill is classified as dangerous (requires HITL) if:

• It requires the 'schema:write' capability, or
• The skill name starts with 'delete'.

Otherwise it is safe → execute directly.

A wildcard '*' in the user's capabilities satisfies every requirement.

End-to-end flow

┌──────────────┐  POST /api/v1/ai/chat
│ AI Assistant │ ─────────► [Hono router]
│ (Studio UI)  │              │
└──────┬───────┘              ▼
       │              [AISecureHarness.execute]
       │                       │
       │            ┌──────────┼──────────┐
       │            ▼          ▼          ▼
       │       validate    capability   risk
       │       skill       check         eval
       │            │          │          │
       │            └──────────┴──────────┘
       │                       │
       │              ┌────────┴────────┐
       │              ▼                 ▼
       │         safe → run       dangerous →
       │         skill direct     INSERT ai_approvals
       │                                │
       │                          status=pending
       │                                ▼
       │                       respond { status:
       │                       'pending_approval',
       │                       approvalId }
       │                                │
       │             ┌──────────────────┘
       │             │
       ▼             ▼
┌──────────────┐  GET /api/v1/ai/approvals
│ Approvals    │ ─────────► [list pending for siteId]
│ Dashboard    │
│              │  POST /api/v1/ai/approvals/:id/decide
│              │ ─────────► executeApproved | rejectApproval
└──────────────┘

API contract

POST /api/v1/ai/chat

Request: { "message": string } (1-2000 chars after trim).

Response 200: { "data": { "status": "executed" | "pending_approval" | "denied", "data"?, "approvalId"?, "message"? } }.

Response 400: validation error.

GET /api/v1/ai/approvals

Returns up to 100 approvals with status='pending' in the current site, sorted createdAt DESC.

POST /api/v1/ai/approvals/:id/decide

Body: { "decision": "approved" | "rejected" }.

• "approved" → the harness executes the skill with the saved arguments and updates the status to approved (if execution succeeds). If the skill fails, the status stays pending so the admin can retry.
• "rejected" → updates the status to rejected.

Every query is scoped by siteId — a request from site B never sees site A's records (HTTP 403, without revealing existence).

Studio UI

AI Assistant panel

• A 48×48px floating button at bottom: 24px, right: 24px.
• Clicking opens a 320×480px panel, glassmorphism (backdrop-blur).
• Distinguishes the user vs assistant roles.
• Shows a pending_approval badge when applicable.
• History up to 50 messages, reset on page reload.

Approvals Dashboard

• apps/studio/src/modules/settings/ai-approvals.tsx.
• Card view; each card: skillName, arguments (JSON pretty 2 spaces), context.
• Two buttons Approve / Reject with a loading state — both disabled while processing.
• An empty state when there is nothing pending.

Multi-tenancy

• Every insert/select/update on the ai_approvals table is WHERE siteId = currentSiteId.
• Cross-site access → 403, without revealing whether the record exists in another site.
• The tenant middleware (withTenant) requires resolving siteId before the AI router operates.

Real LLM integration & Context Memory (RAG)

The LumiBase Copilot is fully integrated with large language models (LLMs) and a vector/chat-history store:

1. LLM Providers

The system supports multiple LLM providers (configured via the LLM_PROVIDER env, model via LLM_MODEL):

• OpenAI: Uses Chat Completions tool calling (e.g. gpt-4.1-nano, default gpt-4o-mini).
• Anthropic / Claude: Uses the Claude model family via native tool_use.
• Gemini: Uses the Gemini REST generateContent with function declarations.
• Workers AI: Runs directly at the edge on Cloudflare Workers AI (defaults to @cf/meta/llama-3.1-8b-instruct).
• Echo: A keyword matcher (fallback mock) for test environments or when API credentials are missing.

2. Context Memory (Conversation history)

The ai_conversations and ai_messages tables store the state of conversation threads:

• Whenever a message is sent to /api/v1/ai/chat, the system loads up to the 20 most recent messages as the context window passed to the LLM.
• Full management APIs are supported: GET /conversations, GET /conversations/:id/messages, DELETE /conversations/:id.

3. RAG Skills (aiSuggestField and aiContentAssist)

Integrates a vector-embedding service (embedding-service.ts) supporting OpenAI text-embedding-3-small and Workers AI @cf/baai/bge-base-en-v1.5:

• aiSuggestField: Analyzes the current schema and the admin's description, retrieves similar vectors from the database via cosine similarity to suggest an optimal field configuration.
• aiContentAssist: Helps generate or revise field content based on RAG context from items already stored in the system.

Property-based testing

The system has 15 property tests (fast-check, ≥100 iterations) fully listed in .kiro/specs/ai-first-cms-engine/design.md under "Correctness Properties". The tests are in:

• apps/cms/src/services/__tests__/ai-harness-*.property.test.ts — Properties 1-8 (Module B).
• apps/cms/src/routes/__tests__/ai-*.property.test.ts — Properties 9-12 (Module C).
• apps/studio/src/modules/settings/__tests__/ — Properties 13-14 (Module D).
• packages/database/src/__tests__/ — Property 15 (round-trip).