Roadmap & Task Breakdown

Scope: This roadmap covers LumiBase Studio — the admin panel for managing data, collections, permissions, etc. — and the LumiBase CMS API, which runs on both Cloudflare Workers and Docker.

The Consumer app (the end-user frontend) in apps/consumer (Next.js) is a demo that uses the SDK to call the Delivery API — it is not the CMS Studio.

Task language: short, ready to drop straight into an issue tracker. Each task has a clear scope, a deliverable, and a link to related docs.

Conventions:

[BE] apps/cms (Backend API - Hono)
[FE] apps/studio (Admin panel - React + Vite)
[DB] packages/database
[RT] packages/runtime (the Cloudflare/Docker abstraction layer)
[AI] packages/ai-skills + AI Copilot
[SDK] packages/sdk (a type-safe client for both Studio and consumer apps)
[DOC] apps/docs or docs in docs/
[OPS] infra/deploy/CI
Each PR should be on a feature/<phase>-<short-name> branch per the Git hygiene rule.

Overall status: Phase 0 → Phase G (GA hardening) is done. POST-GA and Dual Deployment + AI Copilot are complete. The current focus is polish, developer experience, and expanding the marketplace.

Active Ops Hardening Tasks

Sources: apps/docs/content/deployment/docker.md, apps/docs/content/guides/backup-recovery.md.

Phase 0 — Foundation (DONE)

Goal: a working monorepo skeleton, the core schema, Logto auth, CI.

Phase A — Schema engine (DONE)

Goal: create/manage collections & fields via API + UI.

Phase B — Items & extended Field system (DONE)

Phase C — Permissions & Access (DONE)

Phase C2 — Presets, Bookmarks, basic Translations (DONE)

[DB] Tables presets, translations.
[BE] CRUD /presets, scope resolution (user > role > site).
[BE] CRUD /translations (namespaces ui, field, content).
[BE] Locale settings (settings.locales.*).
[FE] Preset switcher + save/edit dialog in the list view.
[FE] Translations module (UI strings tab + content tab JSONB).
[FE] translatable-text interface (JSONB locale map).
[FE] i18n for the Studio UI (react-i18next bound to the translations API).

Phase D — Users, Files, Settings (DONE)

Phase E — Realtime / WebSocket (DONE)

Phase F — Advanced Extensions & Display Templates (DONE)

Phase G — Hardening & GA (DONE)

[BE] Additional Postgres RLS policies via the withRls() middleware (defense-in-depth).
[BE] Complete tag-based invalidation (revalidateTag webhook → Next.js consumer).
[BE] Backups + restore via the /api/v1/admin/backup + /api/v1/admin/restore endpoints (NDJSON bundle).
[BE] Config export/import CLI (apps/cms/scripts/config-cli.ts).
[FE] Accessibility audit + fix.
[FE] Bundle size audit, lazy module splitting (TanStack Router lazy-load).
[OPS] Load test (k6) for the delivery API and realtime — apps/cms/k6/.
[OPS] SLO dashboards (Workers Analytics Engine + Grafana for Docker).
[DOC] Public docs site (apps/docs — Vite + React + Markdown).

Phase POST-GA1 — Translation Memory + MT (DONE)

[DB] Tables translation_memory + glossary.
[BE] translation-memory.ts service with providers: DeepL, OpenAI, Workers AI, echo fallback.
[BE] Routes /api/v1/tm (list/upsert/fuzzy lookup/translate pipeline TM → glossary → MT).
[FE] TM integration UI in the Translations module.
[DOC] features/translation-memory.md.

Phase POST-GA2 — Collaborative cursors (DONE)

[BE] cursor-protocol.ts (CRDT-lite: last-write-wins position + Y-style update vector).
[BE] Broadcast via the Durable Object SiteRoom (CF) or in-process (Docker).
[FE] Render cursors + selection in WYSIWYG / text fields.

Phase POST-GA3 — Flows / Operations engine (DONE)

[DB] Tables flows, flow_runs, operations.
[BE] flow-service.ts runner service with operation types: condition, transform, http, mail, log, sleep, run-extension, item.create|update|delete, notify.
[BE] Routes /api/v1/flows + manual /run + /runs history.
[BE] Trigger types: webhook, event (item.*), schedule (cron), manual.
[FE] Automation → Flows module (list page).
[DOC] features/flows-automation.md.

Phase POST-GA4 — SCIM 2.0 provisioning (DONE)

[BE] /scim/v2/Users + /Groups + ServiceProviderConfig + Schemas + ResourceTypes (RFC 7644 subset).
[BE] A separate Bearer token auth (SCIM_TOKEN), not using Logto JWT.
[BE] Mapping: SCIM Group → LumiBase Team.
[DOC] features/scim-provisioning.md.

Phase POST-GA5 — Marketplace extensions (DONE)

[DB] Add columns signature, signatureAlg, publisherKeyId, publisher, marketplaceSlug, publishedAt, bundleSha256 to extensions.
[BE] Routes /api/v1/marketplace/extensions (list, detail, install, publish).
[BE] Signature verification: SHA-256 bundle + ed25519/RSA-PSS via WebCrypto, public keys loaded from the env MARKETPLACE_PUBLIC_KEYS.
[FE] Public Marketplace site using the real catalog API, SEO/static export/deploy checklist.
[DOC] Revenue sharing settled on Free-first; commercial checkout/payout split into a later backlog.
[DOC] features/marketplace.md.

Phase POST-GA6 — Materialized collections (DONE)

[DB] Table materialized_collections.
[BE] Routes /api/v1/materialize (register, refresh, drop).
[BE] Logical refresh strategy (count + lastRefreshedAt). Full denormalized write still open.
[DOC] features/materialized-collections.md.

Phase POST-GA7 — Advanced Permission Builder & RBAC (TODO)

Goal: upgrade the existing Access Control into a Role / Policy / Permission system equivalent to Directus but more fail-closed, with conflict detection, policy flags, role-based API keys, JSON import/export, and seeded system permissions. Reference: docs/vi/features/permission-builder-directus-investigation.md.

Mandatory preparation

[BE] Audit the current PermissionService: clearly document the existing compose behavior (OR rules, union fields, merge presets/validation) and cases that could silently widen permissions.
[DB] Design a backward-compatible migration for roles.admin_access/app_access → policy-level admin_access/app_access/enforce_tfa/ip_allow/ip_deny/valid_from/valid_until.
[DB] Add a stable key/system_key for roles/policies to support idempotent import/export.
[DOC] Finalize the list of system collections included in the Permission Builder and the sensitive/admin-only groups before seeding.
[BE] Define the JSON schema version lumibase.access@v1 for export/import of roles, policies, permission rows, bindings, and API key metadata.

Schema & evaluator hardening

Conflict detection

[BE] Create an AccessConflictService classifying compatible, warning, blocking for overlaps on the same collection + action.
[BE] Block conflicts of unconditional-vs-restricted rule, ["*"] vs whitelisted fields, validation/preset on the same field with different values, admin bypass + granular policy.
[BE] Endpoint POST /api/v1/access/conflicts/check takes a target role/user/api_key + add/remove policies and returns a diff with the source policy.
[BE] Integrate the conflict check into attaching role-policy, user-policy, api-key-policy; allow overriding a warning with an audit.
[BE] Integrate the conflict check into attaching role-policy and user-policy; a warning override is audited.
[FE] Role Detail calls the conflict check before attaching a policy; a blocking conflict prevents saving.
[FE] The Permission Matrix adds an Effective View showing the final permissions and source policies.
[TEST] Property tests for the conflict classifier with combinations of field/rule/preset/validation.

API Keys by Roles/Policies

[DB] Add api_keys, api_key_roles, api_key_policies with token hash, prefix, expire/revoke/last_used metadata.
[BE] Bearer auth looks up the API key by hash; the api_key principal type compiles permissions like a user.
[BE] Rotate/revoke an API key; the plaintext is only returned once at create/rotate.
[BE] Audit create/rotate/revoke/use-denied for API keys, without logging the plaintext token.
[SDK] Add client methods for API key CRUD, attach roles/policies, conflict preview.
[FE] Studio API Keys page: create, rotate, revoke, attach roles/policies, preview effective permissions.
[TEST] An API key cannot access Studio; a revoked/expired key gets 401; a key only sees the fields/rows allowed by policy.

Permission Builder Import / Export

[BE] GET /api/v1/access/export exports roles, policies, permissions, bindings, API key metadata with stable keys, containing no secrets.
[BE] POST /api/v1/access/import?dryRun=true parses/validates/diffs/conflict-checks but does not write the DB.
[BE] Import modes: merge, replace-managed, replace-all; applied in a transaction with an audited diff summary.
[BE] Idempotency tests: importing the same manifest multiple times creates no duplicates.
[SDK] Add access export/import client types.
[FE] Import dialog shows the diff, warnings, blocking conflicts, and dry-run results.
[OPS] CLI lumibase access export/import for CI/CD between dev/staging/prod.

System permissions & seeding

[DB] Update seed-dev.ts to seed policy_admin, role_administrator, policy_studio_self, policy_public.
[DB] Seed explicit permissions for the schema/access manager group: collections, fields, relations, roles, policies, permissions.
[DB] Ensure sensitive collections (system_state, audit_log, login_attempts, admin_backup_codes, scim_tokens, api_keys) are admin/security-only.
[FE] The Permission Builder groups system collections and hides sensitive collections from non-admins.
[TEST] The default public policy cannot read content/system collections without an explicit grant.

Extension access control

[DOC] Document the Directus extension permission layers: install/enable, sandbox scopes, accountability services, app module self-check.
[DB] Add a stable extensions.key and system access targets extensions, extension_modules, extension_endpoints, extension_operations.
[BE] Enforce extensions:read/configure/install/enable/delete/grant_capability on extension management routes.
[BE] Enforce extensions:execute before dispatching /api/v1/extensions/:name/*.
[BE] Extension data access defaults to actor permissions; service-account mode needs its own policy/capability and audit.
[FE] The Studio extension loader/module bar only shows extensions the principal is allowed to read.
[FE] The Permission Builder adds an Extension Access group to grant users/roles access to extensions.
[TEST] A user without extensions:execute cannot call an extension endpoint even if the extension is enabled.

[DB] Add a shares table with a dedicated share role, password hash, validity window, max uses, revoke.
[BE] Implement the share action: only a user with the share permission can create a share link; the read payload still goes through the share role's permissions.
[FE] The Share dialog only allows selecting a role with appAccess=false, adminAccess=false, and minimal read permissions.
[TEST] A share link only reads the fields/rows the share role is allowed; expired/max-uses/revoked are all denied.

Phase Docker Dual-Deployment (DONE)

Goal: run the entire stack on Docker without a Cloudflare account.

Phase AI-First Copilot (DONE)

Goal: an AI Agent that interacts safely with the CMS via HITL.

[DB] Table ai_approvals (id nanoid 21 + siteId + agentName + skillName + arguments jsonb + status + context + decidedAt + decidedBy).
[AI] Package @lumibase/ai-skills with CORE_SKILLS (listCollections, createCollection, deleteCollection, createField, deleteField, listItems, createItem, updateItem, deleteItem) + OpenAI tool definitions.
[BE] Service ai-harness.ts (validateSkill, checkCapabilities with wildcard *, evaluateRisk, execute, executeApproved, rejectApproval, runSkill 30s timeout).
[BE] Routes /api/v1/ai/chat, /api/v1/ai/approvals, /api/v1/ai/approvals/:id/decide.
[BE] fast-check property tests (15 properties, 100+ iterations) + integration tests.
[FE] components/ai-assistant.tsx floating panel 320×480 glassmorphism, max 50 messages.
[FE] modules/settings/ai-approvals.tsx card list of pending approvals with Approve/Reject.
[DOC] features/ai-copilot.md + keep the historical features/ai-first-specification.md.

Phase POST-GA — Advanced (TODO / In progress)

[AI] Integrate a real LLM provider (OpenAI / Anthropic / Workers AI) instead of the mock intent parser in /ai/chat.
[AI] Add context memory (conversation history) to the AI Copilot.
[AI] Skills aiSuggestField + aiContentAssist (RAG via embeddings).
[BE] Real materialized collection writes (not just logical refresh) — a physical table + a refresh trigger.
[BE] Multi-region Durable Objects sharding.
[FE] Marketplace browser UI in Studio (browse, 1-click install).
[FE] Flows visual editor (drag-drop graph) — currently only a list page.
[BE] SCIM Token rotation + audit.
[OPS] Automated multi-tenant isolation testing (k6 cross-site leak detection).

Phase Agent Harness Layer (DONE)

Goal: turn LumiBase into a control plane where humans, agents, data, workflows, and applications co-evolve under control. The detailed checklist is in agent-harness-implementation.md.

A. Foundational lifecycle

[DB] Add agent_goals, agent_runs, agent_plans, agent_tool_calls with siteId, lifecycle status, policy snapshot, budget, audit metadata, and indexes by siteId/runId/goalId.
[BE] Create AgentRunService to open/append/close/fail/retry a run; refactor AISecureHarness so every runtime execute is tied to a goalId/runId.
[TEST] Extended property tests for multi-tenant isolation, a failed run still keeps the audit trail, and retry does not duplicate tool calls/artifacts.

B. Tool Registry + capability policy

[DB] Add agent_tools and agent_permissions to declare input/output schema, required capabilities, risk policy, rate limit, and validity window.
[BE] Implement ToolRegistryService to load core skills + DB overrides; enforce disabled tool, capability, risk policy, and rate limit.
[FE] Studio "Agent Harness" page showing tools, risk, approvals, runs, artifacts, and memory.
[SDK] Add types/client methods for tools, capabilities, and risk policies.

C. General approval

[DB] Add agent_approvals for plan / tool_call / artifact / schema_diff; bridge backward-compatibly with ai_approvals.
[BE] Foundational approval policy engine: none, before_execute, before_commit, two_person_rule, owner_only, security_admin_only as a contract/policy field.
[FE] Elevate the Studio surface into an Agent Harness queue with subject type, status, and a decision surface.
[TEST] A dangerous plan does not execute before approval; a rejected/expired approval cannot commit.

D. Artifact Store + Evaluation Gate

[DB] Add agent_artifacts and agent_evaluations with content hash, version, status, eval kind/status/score/details.
[BE] First artifact writers: schema_diff, page_spec, component_spec, seed_data, api_spec, prompt, migration.
[BE] First eval runners: JSON schema validation, schema/migration guard, generated API spec validation, prompt safety check.
[FE] Minimal artifact review UI in the Studio Agent Harness: list artifacts, status, hash, generated app artifacts.
[TEST] An artifact that fails eval cannot be published; the artifact hash is stable; publish/rollback is idempotent.

E. Memory + App Generation MVP

[DB] Add agent_memory with scope, provenance, confidence, expiry, and an optional embedding.
[BE] RAG context builder that respects expiry, provenance, and secret redaction.
[AI] Skills generateAppSpec, generateApiDocs, generateSeedData produce an artifact payload instead of writing directly to content/schema.
[FE] A "Generate" action in the Agent Harness creates app artifacts from products/orders/customers with a budget and approval policy.
[TEST] E2E demo: generate a storefront from products/orders/customers → plan → artifacts → eval → approval → publish.

F. Operations

[BE] Metrics for run success/fail, approval latency, tool latency, eval fail rate, token/cost estimate, and budget stop reason.
[OPS] A Grafana "Agent Harness" dashboard and a dead-letter queue for repeatedly failing runs/tool calls.
[DOC] Update data-model.md, architecture/overview.md, OpenAPI, SDK docs, and runtime limitations per phase.

Phase POST-GA8 — Directus Data Model Parity (DONE)

Goal: upgrade the Data Model / Collections Builder so a collection in LumiBase has a contract as clear as Directus's: full metadata, primary key strategy, system fields, advanced field config, relation metadata, schema permissions, atomic diff/apply, SDK/typegen/OpenAPI, and parity tests. Detailed reference: docs/en/features/directus-data-model-parity-tasks.md.

Milestone 1 — Correctness fixes before expansion (DONE)

[FE] The collection wizard sends the correct top-level payload (note, accountability, versioning, singleton, primaryKeyType, storageMode) instead of cramming it into meta.
[BE] ItemService.patch/replace/softDelete enforce update/delete permission, row-level scope, and the field-level update allowlist.
[BE] Relation delete/dependency checks cover both manyCollection and oneCollection directions; block deleting a field/collection while a relation still points to it.
[TEST] Add regression tests for the wizard payload, update/delete permission, and relation dependency checks.

Milestone 2 — Collection metadata + primary key contract (DONE)

[DB] Add first-class collection columns: label, pluralLabel, hidden, system, primaryKeyField, primaryKeyType, storageMode, unarchiveValue, itemDuplicationFields, translations.
[BE] Backward-compatible backfill/migration; route validation and SchemaService use the new fields, keeping meta for extension/custom UI hints.
[SDK] Update the collection input/output types and schema client methods for the new metadata.
[FE] The wizard has Identity, Storage, System fields, Permissions defaults, and Review JSON steps.
[BE] Implement the primary key strategy for jsonb: nanoid, uuid, string; explicitly defer or block integer/bigInteger if there's no sequence yet.
[TEST] Create item respects the primary key strategy; a duplicate user-provided ID returns 409.

Milestone 3 — System fields and field configuration parity

[BE] Extend the compiled schema with systemFields (id, status, sort, user_created, user_updated, created_at, updated_at, deleted_at).
[FE] The Fields tab shows system fields in a locked group; allows configuring display/hidden/readonly/translations/width but not deletion.
[DB] Add field metadata: label, note, defaultValue, nullable, unique, indexed, searchable, length, precision, scale, special.
[FE] FieldInspector advanced tabs: Basics, Options, Display, Validation, Conditions, Layout, Storage, Translations.
[BE] Separate the create/update/rename/delete/migration field paths; reject changing type/name when data exists without a migration plan.
[TEST] FieldInspector does not lose unknown options/displayOptions/validation/conditions; risky changes return 409 or require confirmation.

Milestone 4 — Relations parity and deep read

[BE] Validate relation references: the collection/field exists, the relation name is not duplicated, onDelete is valid for the storage mode.
[DB] Extend relation metadata: type, aliasField, relatedDisplayTemplate, junctionManyField, junctionOneField.
[BE] Support relation types m2o, o2m, m2m; reserve m2a and return "not implemented" if chosen.
[BE] Implement relation expansion for item queries (fields=author.name,categories.*, deep[...]) with permission masking for related collections.
[TEST] M2O returns an object on expand; O2M/M2M return an array; batching avoids N+1 in common cases.

Milestone 5 — Schema permissions, diff/apply, and storage positioning

[BE] Add schema permission actions: schema:read/create/update/delete/migrate.
[BE] Apply requireSchemaPermission to collections/fields/relations/compiled schema routes and AI schema skills.
[BE] Expand the schema diff: collection metadata, field metadata, relation changes, risk classification, and runtime impact.
[BE] PUT /collections/:name/schema validates everything, computes the diff, applies transactionally when the runtime supports it, invalidates schema/permission/typegen cache, and emits schema.changed.
[FE] The Raw JSON schema tab shows the diff/risk before applying.
[DOC] Document the storage modes jsonb/materialized/physical/external, with a limitations badge in Studio.
[DOC] Create the design doc docs/en/architecture/physical-collections.md to decide on physical/external mode.

Milestone 6 — SDK, typegen, OpenAPI, docs, and parity tests

[SDK] Open the full schema resources: collections/fields/relations CRUD, field rename/delete options, schema diff/apply.
[SDK] Keep legacy methods or a deprecation wrapper; preserve the error code/path/risk metadata.
[SDK] Typegen includes the primary key type, system fields, nullable/required, readonly/generated, and relation-expanded response types.
[DOC] Update apps/cms/openapi.yaml, docs/en/features/collections-builder.md, docs/en/features/field-types-and-config.md, docs/en/data-model.md.
[DOC] Sync the Vietnamese version after the English contract stabilizes.
[TEST] Backend/frontend/SDK parity suite covering the acceptance criteria in directus-data-model-parity-tasks.md.

Cross-cutting checklist (every phase)

Update architecture.md if the structure changes.
Write unit + integration tests before merging; for complex logic use property-based testing (fast-check).
Update the OpenAPI spec (apps/cms/openapi.yaml) for every new endpoint.
Update the corresponding packages/sdk types.
Update docs in docs/features/ or apps/docs/content/.
Work directly on main per the current repo guidance; commit with conventional commits and push directly to reduce conflicts with parallel workstreams.
Ensure new routes work on BOTH runtimes (Cloudflare + Docker) — if it depends on a specific API, gate it with a feature flag and document it in features/runtime-abstraction.md.