Content Sync Pipeline

Pipeline Overview

Frontmatter Strategy

Every .md file in the content folders starts with YAML frontmatter that maps directly to WordPress fields:

Schema

---
id: git-cheat-sheet              # required: stable doc identifier
title: Git Quick Reference       # required: post title
description: Essential Git...    # required: excerpt / meta description
category: code                   # required: must == top-level folder
tags: [cheatsheet, git]          # required: YAML list
content_type: cheatsheet         # required: registered content type
status: publish                  # optional: publish | draft, default publish
date: 2026-07-20                  # optional: publish date, date only (YYYY-MM-DD)
slug: git-cheat-sheet            # optional: URL slug, defaults to filename stem
template: reference              # optional: post template name
featured_image:                  # optional: URL to featured image
---

Validation Rules

Field	Rule	Verified
id	Required. Becomes _id86_doc_id post meta.	✅
title	Required. Becomes post_title.	✅
description	Required. Becomes post_excerpt.	✅
category	Required. Must match the first directory component. Auto-created in WordPress if new.	✅
tags	Required. Non-empty YAML list. Auto-created.	✅
content_type	Required. Must be a registered content type (cheatsheet, reference, checklist, glossary, prompt, snippet, pattern, template, runbook, faq, other). Becomes _id86_content_type post meta and drives the content-type badge.	✅
status	publish or draft. Defaults to publish.	✅
date	Date only, YYYY-MM-DD. Falls back to WordPress current time.	✅
slug	URL slug. Defaults to filename stem.	✅
template	Sets _wp_page_template post meta.	✅
featured_image	Sets post thumbnail from existing attachment URL.	✅

Metadata Contract

Meta Key	Source	Purpose	Verified
_id86_doc_id	Frontmatter id	Stable document identifier	✅
_id86_source_path	File path	Repository-relative path for post matching	✅
_id86_source_hash	SHA-256 of file	Change detection; skip when unchanged	✅
_id86_content_type	Frontmatter content_type (written only when unset)	Marks post as managed content; drives the content-type badge from the id86_content_types registry	✅

Structured Data Files

The control mirror's content-inventory.csv has an optional Schema Path column for PID-identified manually authored JSON-LD files under schema-seo/. A valid path is schema-seo/<PID>.json. The inventory and schema validators check that populated paths exist and contain valid JSON. This is currently a registration and validation contract only; sync.py and wp-sync-bridge.php do not yet load or publish those files.

Content PIDs are assigned only in content-inventory.csv. The separate keyword-research.csv register uses Keyword ID values and links selected opportunities through Target Post PID; keyword rows never become WordPress documents by themselves.

Post-Publication Ownership

Markdown is an initial-publication and archival source only. Once the bridge finds an existing WordPress post with status publish, it returns a protected result and does not call wp_update_post, even when --force is supplied. All live body, title, excerpt, tags, and other editorial changes must be made in the WordPress Gutenberg editor. The PID and WordPress Post ID remain the stable identity link in the inventory.

Schema that can be derived from Markdown should remain a sync-time generation task. In particular, glossary DefinedTermSet data should be generated from the visible H2 terms and definitions rather than duplicated in a separate file. See Glossary SEO.

Content Layout

The first folder level determines the WordPress category. Deeper nesting organizes content but does not change taxonomy.

code/git/git-cheat-sheet.md          → category: code
devops/docker/docker-compose.md      → category: devops
ai/prompt-engineering/prompt-patterns.md   → category: ai
security/ssh/ssh-hardening.md        → category: security
demo/styling-sample.md               → category: demo (styling verification fixture)

Markdown Rendering Contract

sync.py converts the Markdown body with markdown2 using these extras:

fenced-code-blocks, tables, strike, header-ids, footnotes, highlightjs-lang

The result is then serialized into Gutenberg block markup and stored as post_content, so posts open in the block editor as real blocks (no manual "Convert to blocks"). The two transforms run in order:

1. tools/code_block_pro/blocks.py transform_code_blocks — every fenced code block becomes a Code Block Pro Gutenberg block pre-rendered by Shiki (github-dark theme), highlighting baked in with no client-side tokenizer.
2. tools/blockify.py html_to_blocks — every remaining element (headings, paragraphs, tables, lists/task lists, blockquotes, admonitions, separators, footnotes) becomes a serialized block matching the editor's own output.

Styling and interactivity are applied at render time:

• css/article-content-v2.css + js/article-content.js (child theme, loaded on is_single())
• Code Block Pro plugin (code-block-pro + cbp-theme-pack) renders the saved Shiki HTML and adds the copy button via front.js
• opshell-docusaurus-admonitions plugin renders opshell/admonition blocks server-side

See Gutenberg Block Reference for the full block inventory, serialization formats, and the markdown→block mapping. See 3. Content → Article Body Elements for the authoring contract.

Sync Decision Flow

Sync Tool

Location: sync.py at the repository root.

Commands

Command	Effect
tools/sync-docs.sh	Full sync via wrapper script (auto-sources venv + CF creds)
tools/sync-docs.sh --dry-run	Validate documents without changing WordPress
tools/sync-docs.sh --force	Force pre-publication updates; published posts remain protected
tools/sync-docs.sh --skip-cache-purge	Skip LiteSpeed and Cloudflare cache purging
tools/sync-docs.sh --cron	Cron mode: git pull, HEAD check, quiet log to reports/cron-sync.log
tools/sync-docs.sh --legacy-html	Store classic HTML instead of serialized blocks (rollback escape hatch)
tools/sync-docs.sh --only PATH	Sync only documents whose source_path matches (pilot / targeted sync)
python sync.py [flags]	Direct Python invocation (requires venv + env setup)

Wrapper script

Location: tools/sync-docs.sh

A Bash wrapper that handles setup automatically and supports two modes:

Manual mode (interactive terminal):

tools/sync-docs.sh              # full sync with output
tools/sync-docs.sh --dry-run    # validate only
tools/sync-docs.sh --force      # update all

The script:

1. Activates the Python venv at .venv/bin/python
2. Sources ~/.ssh/cloudflare/.env if it exists (maps CF_FULL_CONTROL_TOKEN → CLOUDFLARE_API_TOKEN)
3. Passes all flags through to sync.py
4. Prints full output with report path

Cron mode (triggered by timer or scheduler):

tools/sync-docs.sh --cron

This mode:

1. Runs git fetch origin main and compares HEAD to origin/main
2. Exits immediately if no new commits (avoids unnecessary runs)
3. Pulls new commits with --ff-only when changes are detected
4. Runs sync.py quietly, appending output to reports/cron-sync.log
5. Exits cleanly even when there is nothing to sync

Alias shortcut

alias id86sync='/home/rezriz/github/ID86-instant-documentation/tools/sync-docs.sh'

Usage: id86sync --dry-run, id86sync --force, id86sync --cron

Requirements

markdown2>=2.5,<3
PyYAML>=5.4,<7

Configuration

config.yml is git-ignored. Copy config.example.yml:

wp:
  ssh_host: GC-US-M10
  ssh_user: rezriz
  path: /home/Dmg59ZFtKg6bIws1/id86net/public_html
  sudo_user: Dmg59ZFtKg6bIws1

Cloudflare credentials (optional, for cache purge):

Source	Variables	Purpose
GitHub Actions secrets	CLOUDFLARE_API_TOKEN, CLOUDFLARE_ZONE_ID	CI/CD pipeline
Local .env	~/.ssh/cloudflare/.env sets CF_FULL_CONTROL_TOKEN, CF_ZONE_ID	Local testing
sync-docs.sh	Auto-sources the .env file when present	Both manual and cron modes

Trigger Options

Two independent trigger paths — use either or both:

Path	When to use	Cost	Latency
GitHub Actions	CI/CD, push triggered	Free tier minutes	~1-3 min
Manual id86sync	On-demand after editing	Free	Instant
Cron id86sync --cron	Polling fallback when Actions are exhausted	Free	Up to 15 min

Setting up the cron trigger

Register a cron job on this VPS to poll for changes every 15 minutes:

crontab -e

Add the line:

*/15 * * * * /home/rezriz/github/ID86-instant-documentation/tools/sync-docs.sh --cron

The cron mode runs git fetch origin main, compares HEAD to origin/main, and only runs sync.py when new commits are detected. Output is appended to reports/cron-sync.log for debugging.

Logs

Log	Path	Format
Manual sync	Terminal stdout	Live output
Cron sync	reports/cron-sync.log	Append-only, timestamped per run
Sync reports	reports/sync-{YYYYMMDD-HHMMSS}.json	Per-sync structured JSON

SSH Fallback (GitHub Actions unavailable)

When GitHub Actions cannot run (e.g. account billing/spending-limit block stops jobs from starting), the same pipeline runs locally on GSM16 over SSH. sync.py is already fully SSH-based — it SCPs the payload and bridge to GC-US-M10 and runs wp eval-file through WP-CLI — so the local path is byte-for-byte equivalent to the Actions job.

Wrapper: tools/id86-sync.sh — runs sync.py from GSM16 and writes detector-compatible markers to /home/rezriz/logs/id86-sync.log:

tools/id86-sync.sh             # real sync (SSH → WordPress)
tools/id86-sync.sh --dry-run   # validate only

systemd units (user units on GSM16):

Unit	Role
id86-sync.timer / .service	Hourly sync (replaces the Actions push trigger)
id86-sync-watcher.path / .service	Watches the sync log; on failure dispatches opencode to fix the root cause

Auto-recovery reuses the generic detector (Linux-Server-Devops/Backup/auto-recovery/backup-failure-detector.sh) with job config backup-jobs-id86.conf:

/home/rezriz/logs/id86-sync.log|/home/rezriz/github/ID86-instant-documentation/tools/id86-sync.sh|/home/rezriz/github/ID86-instant-documentation

The detector scans the last run of the sync log, and when it finds ERROR: / failed markers it invokes opencode run (deepseek-v4-flash) with the canonical repo as working directory to fix the failing script, guarded by a 6-hour cooldown. Verify a scan with:

BACKUP_JOBS_CONFIG=.../backup-jobs-id86.conf .../backup-failure-detector.sh --dry-run

_id86_source_path convention: the canonical repo path is repo-relative (demo/tmux-cheatsheet.md). The CSV register keeps a content/ prefix (content/demo/...); the direct-publish script (id86-publish-scheduled.sh) strips it before writing the meta so posts are matched by the same key as sync.py and never duplicated.

PHP Bridge

Location: tools/wp-sync-bridge.php

Receives the JSON payload from sync.py and executes WordPress operations through WP-CLI. The bridge:

1. Looks up an existing post by _id86_source_path meta key
2. Compares _id86_source_hash — skips if unchanged (unless --force)
3. Resolves or creates the category by slug
4. Inserts or updates the post with frontmatter fields
5. Sets tags, date, template, and featured image
6. Stores all metadata keys — _id86_content_type is written from the frontmatter content_type field only when the meta is empty, so a manual override from the editor's Content Type metabox (ten registered types) is preserved across syncs
7. Returns a JSON report of created, updated, skipped, and failed documents

Post-Sync Steps

After the bridge completes, sync.py runs three cleanup steps. Each step tolerates failure:

1. Fuse.js search index: sudo -u Dmg59ZFtKg6bIws1 wp fusejs generate-index — Verified ✅
2. LiteSpeed cache: origin-direct PURGESINGLE per changed post URL + homepage (curl --resolve id86.net:443:127.0.0.1 "<url>/?LSCWP_CTRL=PURGESINGLE") — Verified ✅ (targeted by default; --purge-all for structural changes; the wp litespeed-purge all form is blocked by Cloudflare Access; see LiteSpeed Cache CLI)
3. Cloudflare cache: POST /client/v4/zones/{zone}/purge_cache with files:[urls] (targeted) or purge_everything (--purge-all) — Verified ✅

sync.py Internals

Location: sync.py at the repository root.

Function map

Function	Role
load_config()	Reads config.yml, merges from CLOUDFLARE_API_TOKEN/CLOUDFLARE_ZONE_ID env vars
parse_document(path, legacy_html=False)	Splits frontmatter from body, validates required fields (including content_type against the registered types), enforces category↔folder match, returns dict with all fields plus SHA-256 hash and serialized block content. The body is run through markdown2, then transform_code_blocks() (every fenced code block → Code Block Pro block pre-rendered by Shiki), then blockify.html_to_blocks() (every remaining element → serialized Gutenberg block). legacy_html=True skips the blockify step (pre-blockify behavior)
collect_documents(legacy_html=False)	Recursive glob for *.md files, skips .git/, .github/, README.md, tools/, calls parse_document() on each
sync(documents, config, force)	Writes JSON payload to temp file → SCP to GC-US-M10 → SCP bridge PHP → SSH + WP-CLI eval → cleanup temp files
rebuild_search_index(config)	Runs wp fusejs generate-index with sudo on GC-US-M10
purge_litespeed(config, urls)	Targeted origin-direct LiteSpeed purge (PURGESINGLE) of changed post URLs + homepage via curl --resolve ... on GC-US-M10; full=True uses purge_all (see LiteSpeed Cache CLI)
purge_cloudflare(config)	Calls Cloudflare API POST purge_cache with purge_everything: true
write_report(results, documents, force, dry_run)	Writes timestamped JSON report to reports/sync-{timestamp}.json

Payload structure

The JSON payload sent to GC-US-M10 via SCP:

{
  "documents": [
    {
      "doc_id": "git-cheat-sheet",
      "source_path": "code/git/git-cheat-sheet.md",
      "source_hash": "3a0f0217287159ef2849cc23aef6e0fa27a1603d7ed24e4e0c0e6f3e8d7c5b2a",
      "title": "Git Quick Reference",
      "description": "Essential Git commands for inspecting, branching...",
      "category": "code",
      "tags": ["cheatsheet", "git", "version-control"],
      "content_type": "cheatsheet",
      "status": "publish",
      "slug": "git-cheat-sheet",
      "date": "2026-07-20",
      "template": "",
      "featured_image": "",
      "content": "<!-- wp:heading {\"anchor\":\"start-a-repository\"} -->\n<h2 id=\"start-a-repository\" class=\"wp-block-heading\">Start a repository</h2>\n<!-- /wp:heading -->\n\n<!-- wp:paragraph -->\n<p>…</p>\n<!-- /wp:paragraph -->\n… (serialized Gutenberg block markup)"
    }
  ],
  "force": false
}

Error handling strategy

Step tolerance

Each post-sync step tolerates individual failure:

• Fuse.js reindex: prints warning, continues → sync still reports success
• LiteSpeed purge (origin-direct): prints warning if HTTP ≠ 200, continues → sync still reports success
• Cloudflare purge: prints warning (or skips if no token), continues → sync still reports success

This means the sync pipeline completes even if cache purging is temporarily unavailable.

Sync Report

Every sync writes a timestamped JSON report to reports/sync-{YYYYMMDD-HHMMSS}.json:

{
  "timestamp": "20260720-143000",
  "dry_run": false,
  "force": false,
  "total_documents": 5,
  "results": {
    "created": [{"source_path": "code/git/git-cheat-sheet.md", "message": "post 4095"}],
    "updated": [],
    "skipped": [],
    "errors": []
  },
  "summary": {"created": 1, "updated": 0, "skipped": 0, "errors": 0}
}

Reports accumulate in reports/ (git-ignored). In GitHub Actions, they are uploaded as build artifacts.

GitHub Actions Workflow

Location: .github/workflows/sync-wordpress.yml

Execution Flow

Triggers on:

• Push to main affecting code/**/*.md, devops/**/*.md, ai/**/*.md, security/**/*.md, demo/**/*.md, sync.py, tools/wp-sync-bridge.php, tools/code_block_pro/**, tools/code-block-pro/**, requirements.txt
• Manual workflow_dispatch

Required GitHub secrets:

Secret	Purpose
GC_US_M10_SSH_KEY	SSH private key for the rezriz user on GC-US-M10
CLOUDFLARE_API_TOKEN	Cloudflare API token with cache purge permission
CLOUDFLARE_ZONE_ID	Cloudflare zone ID for id86.net

Content Consistency Rules

Heading hierarchy

Because the single post template (GP Element 3663) already renders the post title as an H1, all markdown source files must start content at H2. Using an H1 in the markdown body creates a duplicate title on the rendered page.

---
title: Git Quick Reference
---

## Start a repository          ✅ correct — H2 is the first heading

### Daily workflow              ✅ H3 for subsections

---
title: Git Quick Reference
---

# Git Quick Reference          ❌ WRONG — H1 duplicates the template title

Last-updated date

The single post template shows the last modified date via the [id86_modified_date] shortcode, not the published date. This shortcode renders an opshell-style meta line — updated M j, Y · N min read — using get_the_modified_time('M j, Y'), and falls back to the publish date if no modified date exists. The N min read value is computed by id86_reading_time() from the post's word count at 220 words per minute (minimum 1 minute).

The date and read time update automatically whenever the sync pipeline updates the post, because both are derived from the stored post_content and post_modified at render time.

Frontmatter slug vs file path

The slug field in frontmatter overrides the WordPress URL slug. If omitted, the filename stem is used. The WordPress permalink structure follows the slug, not the folder path.

Example:

• File: devops/linux/plocate-cheat-sheet.md
• Slug: plocate-cheat-sheet (from frontmatter)
• URL: /devops/plocate-cheat-sheet/

Guardrails

Credential safety

• config.yml is git-ignored. Only config.example.yml is committed to the repository.
• Cloudflare API token is read from CLOUDFLARE_API_TOKEN environment variable or ~/.ssh/cloudflare/.env.
• GitHub Actions uses encrypted GitHub Secrets (GC_US_M10_SSH_KEY, CLOUDFLARE_API_TOKEN, CLOUDFLARE_ZONE_ID).
• The sync-docs.sh wrapper auto-sources ~/.ssh/cloudflare/.env when present.

Content safety

Rule	Description	Verified
Non-destructive deletes	Removing a .md file does NOT delete its WordPress post	✅
Fail closed on validation	Missing required frontmatter fields abort the entire sync	✅
Per-file error isolation	A failure in one document does not block other documents	✅
Source is authoritative	Title, content, excerpt, category, tags are always overwritten from markdown	✅
Category auto-creation	A new top-level folder auto-creates the WordPress category	✅
Draft isolation	Draft posts are created but not publicly visible	✅
Dry-run safety	Validates all documents without sending payload or changing WordPress	✅

Unchanged fields

The bridge never overwrites these post properties:

• post_author, post_password
• comment_status, ping_status
• post_thumbnail (only set when featured_image is present in frontmatter — never cleared)
• Any custom meta keys outside the _id86_* namespace

One-time Setup

Required steps to bootstrap the sync pipeline on a new machine:

1. Clone the repository

git clone git@github.com:donnyaw/ID86-instant-documentation.git
cd ID86-instant-documentation

2. Create Python virtual environment

python3 -m venv .venv
.venv/bin/pip install -r requirements.txt

3. Configure WordPress target

cp config.example.yml config.yml
# Edit config.yml with your SSH host, path, and sudo_user

4. Source Cloudflare credentials (optional)

source ~/.ssh/cloudflare/.env

5. Test with dry-run

tools/sync-docs.sh --dry-run

6. Register bash alias (optional)

echo "alias id86sync='$PWD/tools/sync-docs.sh'" >> ~/.bashrc
source ~/.bashrc

7. Install cron job (optional, for polling)

(crontab -l 2>/dev/null; echo '*/15 * * * * /home/rezriz/github/ID86-instant-documentation/tools/sync-docs.sh --cron') | crontab -

Code Repository

Automation code

The sync pipeline code lives in two locations:

Location	Purpose
github.com/donnyaw/ID86-instant-documentation	Markdown content + sync tools (committed and pushed)
/home/rezriz/github/Wordpress/wp-dev/themes/develop-internally/id86/	Local git tracking of all sync pipeline code

File inventory

File	Role
sync.py	Python sync engine — collect documents, build payload, transfer, post-sync steps
tools/blockify.py	Python: serializes article HTML into Gutenberg block markup (headings, tables, lists, admonitions, etc.)
tools/wp-sync-bridge.php	PHP bridge running under WP-CLI — create/update posts, store meta
tools/sync-docs.sh	Bash wrapper — venv + CF env setup, dual manual/cron modes
tools/id86-sync.sh	SSH-fallback wrapper — runs sync.py from GSM16, logs detector-compatible markers
tools/deploy-homepage.php	One-time deployer for homepage GP Elements and theme code
tools/code_block_pro/blocks.py	Python: converts markdown2 <pre><code> into Code Block Pro Gutenberg blocks via Shiki
tools/code-block-pro/render.js	Node: Shiki renderer (VS Code engine, github-dark) for code-block HTML
config.yml	Git-ignored WordPress target configuration
config.example.yml	Committed template with documented fields
requirements.txt	Python dependencies: markdown2, PyYAML
.github/workflows/sync-wordpress.yml	GitHub Actions CI/CD workflow
reports/	Git-ignored sync report output directory

Sync Verification

All items verified in production end-to-end test on 2026-07-20:

#	Test	Result
1	New markdown guide creates one WordPress post	✅ Created post 4138
2	Draft status prevents public visibility	✅ Post 4137 in draft status
3	Custom slug from frontmatter	✅ frontmatter-test-all-fields
4	Date-only frontmatter field applied as post_date	✅ 2026-07-20 00:00:00
5	Unchanged guide skipped via hash	✅ Skipped with "is unchanged"
6	Edited guide updated via hash mismatch	✅ Updated post in-place
7	New category auto-created from folder	✅ Category testing created
8	_id86_doc_id meta stored	✅ frontmatter-test
9	_id86_source_path meta stored	✅ code/testing/frontmatter-test.md
10	_id86_source_hash meta stored	✅ SHA-256 written
11	_id86_content_type meta stored from frontmatter (default only when unset)	✅ cheatsheet
12	Fuse.js search index regenerated	✅ "Success: Search index regenerated"
13	LiteSpeed cache purged (targeted)	✅ HTTP 200 per PURGESINGLE URL
14	Cloudflare cache purged	✅ API returned success: true
15	Sync report written to reports/	✅ Timestamped JSON written
16	Dry-run mode does not change WordPress	✅ 0 changes on dry-run
17	Non-destructive on file removal	✅ Removing test files left posts intact

Block serialization migration (2026-08-13)

The blockify serializer (tools/blockify.py) was added and all 19 published posts migrated from Classic HTML to serialized Gutenberg blocks with a one-time id86sync --force. Verified end to end:

#	Test	Result
1	Serialized output matches the editor's Convert-to-blocks format	✅ Byte-structural match with stored editor output (revision 4355); fixes its corrupt admonition title + stray empty paragraphs
2	Pilot set (4 topics: code / devops / ai / demo) synced in isolation first	✅ --only <path> --force updated exactly one post per run
3	Front-end renders blocks on public posts	✅ wp-block-heading, wp-block-table + has-fixed-layout, CBP blocks, opshell-admonition classes present
4	WP_Block_Parser finds no freeform fragments	✅ 0 non-whitespace fragments across all 19 posts
5	Edge-case elements render	✅ nested lists, footnotes (core/html), blockquote+cite, separator, task checklists
6	Task-checklist markers preserved	✅ literal [ ]/[x] intact in every core/list-item
7	Non-pilot posts untouched during staging	✅ only the 4 pilot posts were blocks until the full --force run
8	Reading time sane	✅ long review = 3219 words → "15 min read"
9	Rollback path documented	✅ id86sync --legacy-html --force; backup at ~/backups/id86-blockify-20260813-134036/