Skip to main content

Content Sync Pipeline

Pipeline Overview​

Frontmatter Strategy​

Every .md file in the content folders starts with YAML frontmatter that maps directly to WordPress fields:

Schema​

---
id: git-cheat-sheet # required: stable doc identifier
title: Git Quick Reference # required: post title
description: Essential Git... # required: SEO meta description (SEOPress)
excerpt: Essential Git... # required: native excerpt for content cards/archives
category: code # required: must == top-level folder
tags: [cheatsheet, git] # required: YAML list
content_type: cheatsheet # required: registered content type
status: publish # optional: publish | draft, default publish
date: 2026-07-20 # optional: publish date, date only (YYYY-MM-DD)
slug: git-cheat-sheet # optional: URL slug, defaults to filename stem
template: reference # optional: post template name
featured_image: # optional: URL to featured image
---

Validation Rules​

FieldRuleVerified
idRequired. Becomes _id86_doc_id post meta.✅
titleRequired. Becomes post_title.✅
descriptionRequired. Becomes _seopress_titles_desc (SEO meta description).✅
excerptRequired. Becomes native post_excerpt (content cards/archives). Mandatory and separate from description.✅
categoryRequired. Must match the first directory component. Auto-created in WordPress if new.✅
tagsRequired. Non-empty YAML list. Auto-created.✅
content_typeRequired. Must be a registered content type (cheatsheet, quick-reference, checklist, glossary, prompt, snippet, pattern, template, runbook, faq, other). Becomes _id86_content_type post meta and drives the content-type badge.✅
statuspublish or draft. Defaults to publish.✅
dateDate only, YYYY-MM-DD. Falls back to WordPress current time.✅
slugURL slug. Defaults to filename stem.✅
templateSets _wp_page_template post meta.✅
featured_imageSets post thumbnail from existing attachment URL.✅

Metadata Contract​

Meta KeySourcePurposeVerified
_id86_doc_idFrontmatter idStable document identifier✅
_id86_source_pathFile pathRepository-relative path for post matching✅
_id86_source_hashSHA-256 of fileChange detection; skip when unchanged✅
_id86_content_typeFrontmatter content_type (written only when unset)Marks post as managed content; drives the content-type badge from the id86_content_types registry✅

Structured Data Files​

The control mirror's content-inventory.csv has an optional Schema Path column for PID-identified manually authored JSON-LD files under schema-seo/. A valid path is schema-seo/<PID>.json. The inventory and schema validators check that populated paths exist and contain valid JSON. This is currently a registration and validation contract only; sync.py and wp-sync-bridge.php do not yet load or publish those files.

Content PIDs are assigned only in content-inventory.csv. The separate keyword-research.csv register uses Keyword ID values and links selected opportunities through Target Post PID; keyword rows never become WordPress documents by themselves.

Post-Publication Ownership​

Markdown is an initial-publication and archival source only. Once the bridge finds an existing WordPress post with status publish, it returns a protected result and does not call wp_update_post, even when --force is supplied. All live body, title, excerpt, tags, and other editorial changes must be made in the WordPress Gutenberg editor. The PID and WordPress Post ID remain the stable identity link in the inventory.

Schema that can be derived from Markdown should remain a sync-time generation task. In particular, glossary DefinedTermSet data should be generated from the visible H2 terms and definitions rather than duplicated in a separate file. See Glossary SEO.

SCS-authored content (brief / article halves)​

When a document is produced through the SCS (SEO Content System) pipeline, the Markdown file carries two halves separated by HTML comments:

<!-- scs:brief -->
## 1. Decision
... (research, SERP, outline, sources, QA handoff — never published)

<!-- scs:article -->
## Overview
... (the published article body)
  • sync.py and wp-sync-bridge.php consume only the article half (after <!-- scs:article -->). The brief half is the audit trail and stays local.
  • The markers must not appear in the published body. The bridge converts the article half to Gutenberg blocks exactly as it does for hand-authored ID86 documents.
  • SCS front matter is a superset of the ID86 contract: the standard fields (title, description, category, tags, content_type, status, date, slug) are the publication contract; the nested scs: block (identity, writing context, links, research, editorial) is consumed by SCS and ignored by the ID86 sync pipeline.
  • ACF content_type (hub / pillar / cluster / supporting) is not read from front matter; it comes from clusters.kc_type in the SCS CSV control plane and is written by the SCS bridge. The ID86 sync pipeline does not manage ACF fields.
  • SCS-managed posts are keyed by scs_pid post meta (in addition to _id86_source_path / _id86_source_hash) so the two pipelines never overwrite each other's posts.

See the SCS WordPress Contract and Front Matter & Markers for the full mapping.

Content Layout​

The first folder level determines the WordPress category. Deeper nesting organizes content but does not change taxonomy.

code/git/git-cheat-sheet.md → category: code
devops/docker/docker-compose.md → category: devops
ai/prompt-engineering/prompt-patterns.md → category: ai
security/ssh/ssh-hardening.md → category: security
demo/styling-sample.md → category: demo (styling verification fixture)

Markdown Rendering Contract​

sync.py converts the Markdown body with markdown2 using these extras:

fenced-code-blocks, tables, strike, header-ids, footnotes, highlightjs-lang

The result is then serialized into Gutenberg block markup and stored as post_content, so posts open in the block editor as real blocks (no manual "Convert to blocks"). The two transforms run in order:

  1. tools/code_block_pro/blocks.py transform_code_blocks — every fenced code block becomes a Code Block Pro Gutenberg block pre-rendered by Shiki (github-dark theme), highlighting baked in with no client-side tokenizer.
  2. tools/blockify.py html_to_blocks — every remaining element (headings, paragraphs, tables, lists/task lists, blockquotes, admonitions, separators, footnotes) becomes a serialized block matching the editor's own output.

Styling and interactivity are applied at render time:

  • css/article-content-v2.css + js/article-content.js (child theme, loaded on is_single())
  • Code Block Pro plugin (code-block-pro + cbp-theme-pack) renders the saved Shiki HTML and adds the copy button via front.js
  • opshell-docusaurus-admonitions plugin renders opshell/admonition blocks server-side

See Gutenberg Block Reference for the full block inventory, serialization formats, and the markdown→block mapping. See 3. Content → Article Body Elements for the authoring contract.

Sync Decision Flow​

Sync Tool​

Location: sync.py at the repository root.

Commands​

CommandEffect
tools/sync-docs.shFull sync via wrapper script (auto-sources venv + CF creds)
tools/sync-docs.sh --dry-runValidate documents without changing WordPress
tools/sync-docs.sh --forceForce pre-publication updates; published posts remain protected
tools/sync-docs.sh --skip-cache-purgeSkip LiteSpeed and Cloudflare cache purging
tools/sync-docs.sh --cronCron mode: git pull, HEAD check, quiet log to reports/cron-sync.log
tools/sync-docs.sh --legacy-htmlStore classic HTML instead of serialized blocks (rollback escape hatch)
tools/sync-docs.sh --only PATHSync only documents whose source_path matches (pilot / targeted sync)
python sync.py [flags]Direct Python invocation (requires venv + env setup)

Wrapper script​

Location: tools/sync-docs.sh

A Bash wrapper that handles setup automatically and supports two modes:

Manual mode (interactive terminal):

tools/sync-docs.sh # full sync with output
tools/sync-docs.sh --dry-run # validate only
tools/sync-docs.sh --force # update all

The script:

  1. Activates the Python venv at .venv/bin/python
  2. Sources ~/.ssh/cloudflare/.env if it exists (maps CF_FULL_CONTROL_TOKEN → CLOUDFLARE_API_TOKEN)
  3. Passes all flags through to sync.py
  4. Prints full output with report path

Cron mode (triggered by timer or scheduler):

tools/sync-docs.sh --cron

This mode:

  1. Runs git fetch origin main and compares HEAD to origin/main
  2. Exits immediately if no new commits (avoids unnecessary runs)
  3. Pulls new commits with --ff-only when changes are detected
  4. Runs sync.py quietly, appending output to reports/cron-sync.log
  5. Exits cleanly even when there is nothing to sync

Alias shortcut​

alias id86sync='/home/rezriz/github/ID86-instant-documentation/tools/sync-docs.sh'

Usage: id86sync --dry-run, id86sync --force, id86sync --cron

Requirements​

markdown2>=2.5,<3
PyYAML>=5.4,<7

Configuration​

config.yml is git-ignored. Copy config.example.yml:

wp:
ssh_host: GC-US-M10
ssh_user: rezriz
path: /home/Dmg59ZFtKg6bIws1/id86net/public_html
sudo_user: Dmg59ZFtKg6bIws1

Cloudflare credentials (optional, for cache purge):

SourceVariablesPurpose
GitHub Actions secretsCLOUDFLARE_API_TOKEN, CLOUDFLARE_ZONE_IDCI/CD pipeline
Local .env~/.ssh/cloudflare/.env sets CF_FULL_CONTROL_TOKEN, CF_ZONE_IDLocal testing
sync-docs.shAuto-sources the .env file when presentBoth manual and cron modes

Trigger Options​

Two independent trigger paths — use either or both:

PathWhen to useCostLatency
GitHub ActionsCI/CD, push triggeredFree tier minutes~1-3 min
Manual id86syncOn-demand after editingFreeInstant
Cron id86sync --cronPolling fallback when Actions are exhaustedFreeUp to 15 min

Setting up the cron trigger​

Register a cron job on this VPS to poll for changes every 15 minutes:

crontab -e

Add the line:

*/15 * * * * /home/rezriz/github/ID86-instant-documentation/tools/sync-docs.sh --cron

The cron mode runs git fetch origin main, compares HEAD to origin/main, and only runs sync.py when new commits are detected. Output is appended to reports/cron-sync.log for debugging.

Logs​

LogPathFormat
Manual syncTerminal stdoutLive output
Cron syncreports/cron-sync.logAppend-only, timestamped per run
Sync reportsreports/sync-{YYYYMMDD-HHMMSS}.jsonPer-sync structured JSON

SSH Fallback (GitHub Actions unavailable)​

When GitHub Actions cannot run (e.g. account billing/spending-limit block stops jobs from starting), the same pipeline runs locally on GSM16 over SSH. sync.py is already fully SSH-based — it SCPs the payload and bridge to GC-US-M10 and runs wp eval-file through WP-CLI — so the local path is byte-for-byte equivalent to the Actions job.

Wrapper: tools/id86-sync.sh — runs sync.py from GSM16 and writes detector-compatible markers to /home/rezriz/logs/id86-sync.log:

tools/id86-sync.sh # real sync (SSH → WordPress)
tools/id86-sync.sh --dry-run # validate only

systemd units (user units on GSM16):

UnitRole
id86-sync.timer / .serviceHourly sync (replaces the Actions push trigger)
id86-sync-watcher.path / .serviceWatches the sync log; on failure dispatches opencode to fix the root cause

Auto-recovery reuses the generic detector (Linux-Server-Devops/Backup/auto-recovery/backup-failure-detector.sh) with job config backup-jobs-id86.conf:

/home/rezriz/logs/id86-sync.log|/home/rezriz/github/ID86-instant-documentation/tools/id86-sync.sh|/home/rezriz/github/ID86-instant-documentation

The detector scans the last run of the sync log, and when it finds ERROR: / failed markers it invokes opencode run (deepseek-v4-flash) with the canonical repo as working directory to fix the failing script, guarded by a 6-hour cooldown. Verify a scan with:

BACKUP_JOBS_CONFIG=.../backup-jobs-id86.conf .../backup-failure-detector.sh --dry-run

_id86_source_path convention: the canonical repo path is repo-relative (demo/tmux-cheatsheet.md). The CSV register keeps a content/ prefix (content/demo/...); the direct-publish script (id86-publish-scheduled.sh) strips it before writing the meta so posts are matched by the same key as sync.py and never duplicated.

PHP Bridge​

Location: tools/wp-sync-bridge.php

Receives the JSON payload from sync.py and executes WordPress operations through WP-CLI. The bridge:

  1. Looks up an existing post by _id86_source_path meta key
  2. Compares _id86_source_hash — skips if unchanged (unless --force)
  3. Resolves or creates the category by slug
  4. Inserts or updates the post with frontmatter fields
  5. Sets tags, date, template, and featured image
  6. Stores all metadata keys — _id86_content_type is written from the frontmatter content_type field only when the meta is empty, so a manual override from the editor's Content Type metabox (ten registered types) is preserved across syncs
  7. Returns a JSON report of created, updated, skipped, and failed documents

Post-Sync Steps​

After the bridge completes, sync.py runs three cleanup steps. Each step tolerates failure:

  1. Meilisearch search index: posts are re-indexed via save_post hooks (bulk: wp eval-file reindex.php) — Verified ✅
  2. LiteSpeed cache: origin-direct PURGESINGLE per changed post URL + homepage (curl --resolve id86.net:443:127.0.0.1 "<url>/?LSCWP_CTRL=PURGESINGLE") — Verified ✅ (targeted by default; --purge-all for structural changes; the wp litespeed-purge all form is blocked by Cloudflare Access; see LiteSpeed Cache CLI)
  3. Cloudflare cache: POST /client/v4/zones/{zone}/purge_cache with files:[urls] (targeted) or purge_everything (--purge-all) — Verified ✅

sync.py Internals​

Location: sync.py at the repository root.

Function map​

FunctionRole
load_config()Reads config.yml, merges from CLOUDFLARE_API_TOKEN/CLOUDFLARE_ZONE_ID env vars
parse_document(path, legacy_html=False)Splits frontmatter from body, validates required fields (including content_type against the registered types), enforces category↔folder match, returns dict with all fields plus SHA-256 hash and serialized block content. The body is run through markdown2, then transform_code_blocks() (every fenced code block → Code Block Pro block pre-rendered by Shiki), then blockify.html_to_blocks() (every remaining element → serialized Gutenberg block). legacy_html=True skips the blockify step (pre-blockify behavior)
collect_documents(legacy_html=False)Recursive glob for *.md files, skips .git/, .github/, README.md, tools/, calls parse_document() on each
sync(documents, config, force)Writes JSON payload to temp file → SCP to GC-US-M10 → SCP bridge PHP → SSH + WP-CLI eval → cleanup temp files
rebuild_search_index(config)Triggers Meilisearch reindex (wp eval-file reindex.php) on GC-US-M10
purge_litespeed(config, urls)Targeted origin-direct LiteSpeed purge (PURGESINGLE) of changed post URLs + homepage via curl --resolve ... on GC-US-M10; full=True uses purge_all (see LiteSpeed Cache CLI)
purge_cloudflare(config)Calls Cloudflare API POST purge_cache with purge_everything: true
write_report(results, documents, force, dry_run)Writes timestamped JSON report to reports/sync-{timestamp}.json

Payload structure​

The JSON payload sent to GC-US-M10 via SCP:

{
"documents": [
{
"doc_id": "git-cheat-sheet",
"source_path": "code/git/git-cheat-sheet.md",
"source_hash": "3a0f0217287159ef2849cc23aef6e0fa27a1603d7ed24e4e0c0e6f3e8d7c5b2a",
"title": "Git Quick Reference",
"description": "Essential Git commands for inspecting, branching...",
"category": "code",
"tags": ["cheatsheet", "git", "version-control"],
"content_type": "cheatsheet",
"status": "publish",
"slug": "git-cheat-sheet",
"date": "2026-07-20",
"template": "",
"featured_image": "",
"content": "<!-- wp:heading {\"anchor\":\"start-a-repository\"} -->\n<h2 id=\"start-a-repository\" class=\"wp-block-heading\">Start a repository</h2>\n<!-- /wp:heading -->\n\n<!-- wp:paragraph -->\n<p>…</p>\n<!-- /wp:paragraph -->\n… (serialized Gutenberg block markup)"
}
],
"force": false
}

Error handling strategy​

Step tolerance​

Each post-sync step tolerates individual failure:

  • Meilisearch reindex: prints warning, continues → sync still reports success
  • LiteSpeed purge (origin-direct): prints warning if HTTP ≠ 200, continues → sync still reports success
  • Cloudflare purge: prints warning (or skips if no token), continues → sync still reports success

This means the sync pipeline completes even if cache purging is temporarily unavailable.

Sync Report​

Every sync writes a timestamped JSON report to reports/sync-{YYYYMMDD-HHMMSS}.json:

{
"timestamp": "20260720-143000",
"dry_run": false,
"force": false,
"total_documents": 5,
"results": {
"created": [{"source_path": "code/git/git-cheat-sheet.md", "message": "post 4095"}],
"updated": [],
"skipped": [],
"errors": []
},
"summary": {"created": 1, "updated": 0, "skipped": 0, "errors": 0}
}

Reports accumulate in reports/ (git-ignored). In GitHub Actions, they are uploaded as build artifacts.

GitHub Actions Workflow​

Location: .github/workflows/sync-wordpress.yml

Execution Flow​

Triggers on:

  • Push to main affecting code/**/*.md, devops/**/*.md, ai/**/*.md, security/**/*.md, demo/**/*.md, sync.py, tools/wp-sync-bridge.php, tools/code_block_pro/**, tools/code-block-pro/**, requirements.txt
  • Manual workflow_dispatch

Required GitHub secrets:

SecretPurpose
GC_US_M10_SSH_KEYSSH private key for the rezriz user on GC-US-M10
CLOUDFLARE_API_TOKENCloudflare API token with cache purge permission
CLOUDFLARE_ZONE_IDCloudflare zone ID for id86.net

Content Consistency Rules​

Heading hierarchy​

Because the single post template (GP Element 3663) already renders the post title as an H1, all markdown source files must start content at H2. Using an H1 in the markdown body creates a duplicate title on the rendered page.

---
title: Git Quick Reference
---

## Start a repository ✅ correct — H2 is the first heading

### Daily workflow ✅ H3 for subsections
---
title: Git Quick Reference
---

# Git Quick Reference ❌ WRONG — H1 duplicates the template title

Last-updated date​

The single post template shows the last modified date via the [id86_modified_date] shortcode, not the published date. This shortcode renders an opshell-style meta line — updated M j, Y · N min read — using get_the_modified_time('M j, Y'), and falls back to the publish date if no modified date exists. The N min read value is computed by id86_reading_time() from the post's word count at 220 words per minute (minimum 1 minute).

The date and read time update automatically whenever the sync pipeline updates the post, because both are derived from the stored post_content and post_modified at render time.

Frontmatter slug vs file path​

The slug field in frontmatter overrides the WordPress URL slug. If omitted, the filename stem is used. The WordPress permalink structure follows the slug, not the folder path.

Example:

  • File: devops/linux/plocate-cheat-sheet.md
  • Slug: plocate-cheat-sheet (from frontmatter)
  • URL: /devops/plocate-cheat-sheet/

Guardrails​

Credential safety​

  • config.yml is git-ignored. Only config.example.yml is committed to the repository.
  • Cloudflare API token is read from CLOUDFLARE_API_TOKEN environment variable or ~/.ssh/cloudflare/.env.
  • GitHub Actions uses encrypted GitHub Secrets (GC_US_M10_SSH_KEY, CLOUDFLARE_API_TOKEN, CLOUDFLARE_ZONE_ID).
  • The sync-docs.sh wrapper auto-sources ~/.ssh/cloudflare/.env when present.

Content safety​

RuleDescriptionVerified
Non-destructive deletesRemoving a .md file does NOT delete its WordPress post✅
Fail closed on validationMissing required frontmatter fields abort the entire sync✅
Per-file error isolationA failure in one document does not block other documents✅
Source is authoritativeTitle, content, excerpt, category, tags are always overwritten from markdown✅
Category auto-creationA new top-level folder auto-creates the WordPress category✅
Draft isolationDraft posts are created but not publicly visible✅
Dry-run safetyValidates all documents without sending payload or changing WordPress✅

Unchanged fields​

The bridge never overwrites these post properties:

  • post_author, post_password
  • comment_status, ping_status
  • post_thumbnail (only set when featured_image is present in frontmatter — never cleared)
  • Any custom meta keys outside the _id86_* namespace

One-time Setup​

Required steps to bootstrap the sync pipeline on a new machine:

1. Clone the repository​

git clone git@github.com:donnyaw/ID86-instant-documentation.git
cd ID86-instant-documentation

2. Create Python virtual environment​

python3 -m venv .venv
.venv/bin/pip install -r requirements.txt

3. Configure WordPress target​

cp config.example.yml config.yml
# Edit config.yml with your SSH host, path, and sudo_user

4. Source Cloudflare credentials (optional)​

source ~/.ssh/cloudflare/.env

5. Test with dry-run​

tools/sync-docs.sh --dry-run

6. Register bash alias (optional)​

echo "alias id86sync='$PWD/tools/sync-docs.sh'" >> ~/.bashrc
source ~/.bashrc

7. Install cron job (optional, for polling)​

(crontab -l 2>/dev/null; echo '*/15 * * * * /home/rezriz/github/ID86-instant-documentation/tools/sync-docs.sh --cron') | crontab -

Code Repository​

Automation code​

The sync pipeline code lives in two locations:

LocationPurpose
github.com/donnyaw/ID86-instant-documentationMarkdown content + sync tools (committed and pushed)
/home/rezriz/github/Wordpress/wp-dev/themes/develop-internally/id86/Local git tracking of all sync pipeline code

File inventory​

FileRole
sync.pyPython sync engine — collect documents, build payload, transfer, post-sync steps
tools/blockify.pyPython: serializes article HTML into Gutenberg block markup (headings, tables, lists, admonitions, etc.)
tools/wp-sync-bridge.phpPHP bridge running under WP-CLI — create/update posts, store meta
tools/sync-docs.shBash wrapper — venv + CF env setup, dual manual/cron modes
tools/id86-sync.shSSH-fallback wrapper — runs sync.py from GSM16, logs detector-compatible markers
tools/deploy-homepage.phpOne-time deployer for homepage GP Elements and theme code
tools/code_block_pro/blocks.pyPython: converts markdown2 <pre><code> into Code Block Pro Gutenberg blocks via Shiki
tools/code-block-pro/render.jsNode: Shiki renderer (VS Code engine, github-dark) for code-block HTML
config.ymlGit-ignored WordPress target configuration
config.example.ymlCommitted template with documented fields
requirements.txtPython dependencies: markdown2, PyYAML
.github/workflows/sync-wordpress.ymlGitHub Actions CI/CD workflow
reports/Git-ignored sync report output directory

Sync Verification​

All items verified in production end-to-end test on 2026-07-20:

#TestResult
1New markdown guide creates one WordPress post✅ Created post 4138
2Draft status prevents public visibility✅ Post 4137 in draft status
3Custom slug from frontmatter✅ frontmatter-test-all-fields
4Date-only frontmatter field applied as post_date✅ 2026-07-20 00:00:00
5Unchanged guide skipped via hash✅ Skipped with "is unchanged"
6Edited guide updated via hash mismatch✅ Updated post in-place
8_id86_doc_id meta stored✅ frontmatter-test
9_id86_source_path meta stored✅ code/git/git-cheat-sheet.md
10_id86_source_hash meta stored✅ SHA-256 written
11_id86_content_type meta stored from frontmatter (default only when unset)✅ cheatsheet
12Meilisearch index updated✅ "Indexed N posts"
13LiteSpeed cache purged (targeted)✅ HTTP 200 per PURGESINGLE URL
14Cloudflare cache purged✅ API returned success: true
15Sync report written to reports/✅ Timestamped JSON written
16Dry-run mode does not change WordPress✅ 0 changes on dry-run
17Non-destructive on file removal✅ Removing test files left posts intact

Block serialization migration (2026-08-13)​

The blockify serializer (tools/blockify.py) was added and all 19 published posts migrated from Classic HTML to serialized Gutenberg blocks with a one-time id86sync --force. Verified end to end:

#TestResult
1Serialized output matches the editor's Convert-to-blocks format✅ Byte-structural match with stored editor output (revision 4355); fixes its corrupt admonition title + stray empty paragraphs
2Pilot set (4 topics: code / devops / ai / security) synced in isolation first✅ --only <path> --force updated exactly one post per run
3Front-end renders blocks on public posts✅ wp-block-heading, wp-block-table + has-fixed-layout, CBP blocks, opshell-admonition classes present
4WP_Block_Parser finds no freeform fragments✅ 0 non-whitespace fragments across all 19 posts
5Edge-case elements render✅ nested lists, footnotes (core/html), blockquote+cite, separator, task checklists
6Task-checklist markers preserved✅ literal [ ]/[x] intact in every core/list-item
7Non-pilot posts untouched during staging✅ only the 4 pilot posts were blocks until the full --force run
8Reading time sane✅ long review = 3219 words → "15 min read"
9Rollback path documented✅ id86sync --legacy-html --force; backup at ~/backups/id86-blockify-20260813-134036/