Documents

Document channels (folder tree), per-channel pipelines and metadata extraction, and the PaddleOCR-VL parsing pipeline driven by openkms-cli.

Document channels and CRUD

Feature	Status	Description
Document overview	✅	Dashboard at /documents with channel count, document count (from API stats), quick actions
Channel management	✅	Create channels at /documents/channels (tree structure); rename, description, move, merge, delete; settings per channel
Document channel view	✅	Browse documents by channel at /documents/channels/:channelId; list from GET /api/documents?channel_id= with pagination (offset/limit, page size 10–100), status filter, applicable filter (lifecycle-derived; same rules as is_current_for_rag), and toolbar refresh; multi-select on the current page with bulk Process, Reset status (to uploaded for re-processing), Move, and Delete
Channel settings	✅	Per-channel pipeline, auto-process, metadata extraction (model + schema, supports object_type/list[object_type]), manual labels config at /documents/channels/:channelId/settings; tabbed UI (General, Processing, Metadata extraction, Manual Labels)
Document upload	✅	Upload to channel via modal (choose files, drag-and-drop); POST /api/documents/upload with channel_id; stores file to S3 under documents/{file_hash}/ (reads fall back to legacy {file_hash}/ until migrated via Console object storage). Accepted types: PDF, PNG/JPG/JPEG/WEBP, DOCX, PPTX, XLSX, EPUB, XMIND. XLSX: preview rows + markdown built at upload (status=completed or failed if unreadable); no VLM pipeline. XMIND: outline markdown from content.json / content.xml at upload (notes, labels, markers, hyperlinks; attachment paths listed); no VLM pipeline. DOCX/PPTX/EPUB (and other pipeline types): status=uploaded until processed
Document processing	✅	Process button on list/detail → POST /api/jobs (optional force_reparse; Job runs UI checkbox). Disabled with tooltip when the channel has no default pipeline (PDF/images/Office); XLSX / XMIND do not require a pipeline. XLSX runs use run_spreadsheet_preview; XMIND runs use run_mindmap_preview. Other types: force_reparse false reuses {file_hash}/result.json on storage when present; if the channel has metadata extraction and document metadata schema fields are all empty (checked via DB / GET /internal-api/documents/{id}/metadata-needs-extraction), the worker still runs openkms-cli to extract metadata only (parse skipped in CLI for paddleocr-doc-parse and baidu-doc-parse). force_reparse true or Retry on a failed run always runs the full VLM CLI again. Channel auto-process uses force_reparse=false
Document status	✅	Status badge (uploaded/pending/running/completed/failed) on document list and detail
Document detail	✅	View parsed Markdown at /documents/view/:id; layout skeleton while loading; Document Information: 3-column stats (Type, Size, Uploaded | Status, Markdown, File hash | Version panel with Versions + conditional Commit version when working copy changed since last snapshot); METADATA section includes Lineage & lifecycle below Extract (collapsed by default; expands for series, relationships, lifecycle, dates, and read-only Applicable); right panel: Markdown | Page Index (refresh parses markdown to tree; Page Index hidden for XLSX); Markdown read mode includes a Print action that opens Chrome/system print with a print-only single-column markdown layout; large books / long documents can defer the page-image pane until requested so the page stays responsive; XLSX: left panel Workbook with sheet tabs and scrollable grid from parsing_result; explicit versions (document_versions) not created on routine save; scrollable layout (min-height 720px)
Document markdown edit	✅	Edit/View toggle, textarea for markdown, Save (PUT /markdown; rebuilds page index), Restore from S3 (POST /restore-markdown; rebuilds page index); POST /rebuild-page-index for manual rebuild from current markdown; Save in the panel header uses the same outlined control style as Edit/Cancel with accent emphasis (icon + label); print mode is read-only and hides app chrome / image pane for cleaner browser PDF output
Document versions	✅	User-triggered checkpoints: POST /documents/{id}/versions snapshots current markdown and metadata (optional tag in API); list, preview, restore (POST .../versions/{vid}/restore); optional commit-current before restore; Commit version modal (optional tag)
Document metadata extraction	✅	Single METADATA section on detail page; Extract button uses channel's LLM; configurable schema per channel (key, label, type: text/date/enum/object_type/list[object_type], description); object_type_extraction_max_instances limits instance count for extraction
Document info & metadata edit	✅	Edit document name and channel (PUT /api/documents/{id}); Edit metadata fields inline (PUT /metadata); Move document to channel via modal
Document metadata (unified)	✅	All metadata (extracted + manual) in single metadata JSONB; manual labels configure in channel settings Manual Labels tab (type: object_type or list[object_type]); object-instance pickers in METADATA section
Channel description	✅	Channel description shown on channel page; stored in document_channels.description
Parsing export/import	✅	Icon buttons on document detail NAME row: download all parsing files as zip, upload a previously exported zip (auto chunked > 10 MB). Restores S3 files, markdown, parsing_result, status→completed, rebuilds page index. For backup and transferring parse results between deployments without re-processing

Document parsing (PaddleOCR-VL and Baidu Cloud)

• PaddleOCR-VL with mlx-vlm-server as VLM backend (paddleocr-doc-parse pipeline / default parse run)
• Baidu Cloud PaddleOCR-VL API (baidu-doc-parse pipeline / parse run --method baidu-doc-parse) — uploads to private Baidu BOS, submits presigned file_url to paddle-vl-parser; credentials OPENKMS_BAIDU_CLOUD_*, bucket OPENKMS_BAIDU_BOS_BUCKET, optional BAIDU_*_URL in openkms-cli/.env; disable BOS hotlink protection; page preview PNGs and bbox coords use 2× page_meta (same grid as local VLM)
• Supports: PDF, PNG, JPG, JPEG, WEBP; DOCX and PPTX are converted to PDF with LibreOffice (soffice / libreoffice) in the worker/CLI; EPUB is converted to PDF with MuPDF (mutool; package mupdf-tools in the worker image) using stream compression so large technical EPUBs stay under Baidu's 100MB PDF limit, then parsed like PDF
• Output: Markdown, layout detection, parsing result JSON
• Configurable: server URL, model, max concurrency
• Channel metadata extraction during pipeline: if the extraction LLM errors (e.g. HTTP 502), the CLI logs a warning and still completes the parse so the document can reach completed; use Extract on the document page when the model is healthy. Extracted fields are written to {file_hash}/extracted_metadata.json on storage; the worker merges them into the document row when the job finishes. Before LLM extraction, openkms-cli calls GET /internal-api/documents/{id}/metadata-needs-extraction; skips extraction when channel schema fields already have values. On re-process with cached parse output, metadata extraction runs only when that check reports empty fields (both paddleocr-doc-parse and baidu-doc-parse).

openkms-cli

• CLI at openkms-cli/ built with Typer (≥0.9.0)
• Tests: openkms-cli/tests/ — pip install -e ".[dev]" && pytest tests/ (VLM defaults merge / fetch wiring with mocks; parser restructure and bbox/layout helpers; no Paddle install required)
• Configuration: openkms_cli/settings.py (CliSettings, pydantic-settings) lists every supported env var via validation_alias; parse/pipeline/auth read through get_cli_settings(); Typer no longer duplicates env via envvar=
• Parse: openkms-cli parse run <input> [--output dir] [--method paddleocr-doc-parse|baidu-doc-parse] [--vlm-url ...]; inputs: PDF, images, DOCX, PPTX (LibreOffice), EPUB (mutool); VLM URL/model/key can follow GET /internal-api/models/document-parse-defaults when OPENKMS_API_URL is set, needed OPENKMS_VLM_* values are missing, and CLI auth succeeds; when OPENKMS_VLM_MODEL is set in the environment, the CLI sends ?model_name=... so the backend returns that vl/ocr row's URL and key, or the default row if there is no match
• Pipeline: openkms-cli pipeline list (list supported pipelines); openkms-cli pipeline run --pipeline-name paddleocr-doc-parse|baidu-doc-parse --input s3://.../original.<ext> – S3 or local input (stored key preserves extension); optional --s3-prefix (defaults to file hash), --skip-upload
• Metadata extraction: when channel has extraction_model_id and extraction_schema, worker passes --extract-metadata --extraction-model-name <model_name>; CLI fetches model config from GET /internal-api/models/config-by-name, extracts via pydantic-ai PromptedOutput(StructuredDict) with response_format: json_object (DeepSeek-compatible), PUTs to PUT /internal-api/documents/{id}/metadata (system path; no channel write ACL); LLM failure does not fail the pipeline after a successful parse
• Uses PaddleOCR-VL for parsing (optional: pip install openkms-cli[parse]); pipeline needs pip install openkms-cli[pipeline]; extraction needs pip install openkms-cli[metadata]; PageIndex tree built-in (md_to_tree uses # headings)
• Output structure on storage: documents/{file_hash}/original.{ext}, result.json, markdown.md, page_index.json (when pageindex installed), layout_det_*, block_*, markdown_out/* (logical paths in result.json may still use {file_hash}/…; API resolves to the new prefix)
• Backend integration: subprocess-invokable for async jobs
• Extensible: developers can add new Typer subapps in app.py