Search Engine Indexing

Purpose

When the site is public, ensure every new documentation post published from the GitHub Markdown repository is discoverable through the sitemap and publish-time notifications. During development, Cloudflare Zero Trust intentionally blocks crawlers and indexing is not expected.

Current Automation Stack

Layer	Mechanism	Provider
Sitemap generation	XML sitemap updated on every post save	SEOPress + Pro
Instant indexing	IndexNow submission on publish	SEOPress + Pro and participating engines
Google Indexing API	Not used for ordinary documentation	Google restricts it to `JobPosting` and `BroadcastEvent`
Publish-time ping	`wp_insert_post` triggers WP ping / search-engine notify	WordPress core
Content trigger	GitHub `push` → `sync.py` → `wp_insert_post`	GitHub Actions

Why Not a Custom API Solution

The GSC Indexing API only accepts pages with JobPosting or BroadcastEvent structured data — not general documentation. The Search Console API reads performance data (queries, clicks, positions) but cannot submit URLs. There is no public API to force-index regular pages; Google decides crawl timing.

Therefore indexing automation relies on:

XML sitemap submitted once in Search Console — new posts are discovered on each crawl pass.
IndexNow / Instant Indexing — fires automatically on publish for participating engines, the closest thing to "auto-register new content."
WordPress publish-time ping — already triggered by sync.py's wp_insert_post.

How the Sync Triggers Indexing

SEOPress WP-CLI Commands

SEOPress exposes official WP-CLI commands for SEO automation. These manage metadata, settings, and license — not indexing itself (indexing is automatic on publish).

Settings Backup / Restore

wp seopress settings export --destination='/home/Dmg59ZFtKg6bIws1/id86net/public_html/wp-content/uploads/seopress'
wp seopress settings import --from='/home/Dmg59ZFtKg6bIws1/id86net/public_html/wp-content/uploads/seopress'

Per-Post SEO Metadata

wp post meta get 123 _seopress_titles_title
wp post meta update 123 _seopress_titles_title "my super SEO title"
wp post meta delete 123 _seopress_titles_title

AI Metadata Generation

wp seopress ai metadata --id=1,2,3,4,5 --meta=desc --language=en_US
wp seopress ai alt_text --id=1,2,3 --language=en_US

Global Settings

wp option get seopress_titles_option_name
wp option patch update seopress_titles_option_name seopress_titles_home_site_title "my super homepage title"
wp option delete seopress_titles_option_name

License Management

wp seopress-pro license activate --key=YOUR_LICENSE_KEY
wp seopress-pro license status
wp seopress-pro license update
wp seopress-pro license reset

SEOPress Metadata Sync (implemented)

The sync pipeline writes dedicated per-post SEO metadata at publish time via wp-sync-bridge.php:

sync.py → wp_insert_post → update_post_meta(_seopress_titles_title,  <title>)
                        → update_post_meta(_seopress_titles_desc,   <description>)
                        → post_excerpt = <description>

This applies SEO titles/descriptions automatically during initial publication and unpublished updates. Published posts are protected from later Markdown syncs; live SEO edits belong in WordPress/SEOPress. See SEO Metadata Automation for the full flow, authoring rules, and verification commands.

GSC Verification Status

Item	Status
Property	`sc-domain:id86.net` — verified
Verification method	DNS TXT (`google-site-verification=...` — value in `~/.ssh/google/.env`, `GOOGLE_SITE_VERIFICATION_TOKEN`) added via Cloudflare API
TXT record location	`id86.net` zone, DNS record ID `260f23c3195298481155aaaac7fa4d5b`
Service account access	`siteFullUser` — API pulls work end-to-end
Sitemap	`https://id86.net/sitemaps.xml` — endpoint responds, but current external check returns HTML instead of valid XML; fix before launch

Rule: do not remove the DNS TXT record — removing it revokes GSC ownership. To harden, add a second verification method (e.g. HTML file) from GSC Settings → Ownership verification.

Site Visibility State (under development)

id86.net is still under development and protected by Cloudflare Zero Trust Access (email OTP). This is intentional and must remain enabled during development. Consequences:

Googlebot is blocked — all public requests get a 302 to the Access login (rezriz.cloudflareaccess.com/cdn-cgi/access/login/...), including crawlers. Google's current inspection is stale (last crawl 2026-07-14) and reads "excluded by noindex" even though the live origin serves index, follow.
GSC data stays empty until launch — Google cannot crawl, so Search Analytics shows nothing.
GA4 only counts authenticated visitors during development (expected).

These are expected during development, not misconfigurations. See the Launch Checklist below.

Development-Safe Checks

Run checks without changing Cloudflare Access:

cd /home/rezriz/github/seo-content/web-project/id86.net
python3 scripts/check-indexing.py --sitemap --robots --development \
  --report reports/indexing-development.json

The report records the expected Access redirect as development_protected. Do not add a crawler bypass while the site is private.

The checker also verifies response content types and parses the sitemap XML. A HTTP 200 response is not sufficient: an HTML error or login page must fail the check.

Launch Checklist (when site goes public)

Decide whether to remove Zero Trust protection or add a narrowly scoped Cloudflare Access bypass policy for verified bot user-agents (Googlebot, Google-Read-Aloud, Bingbot, etc.) on the id86.net Access application. Do not perform this during development.
Re-verify Googlebot gets a real 200 on https://id86.net/ and https://id86.net/sitemaps.xml from outside Cloudflare.
Trigger a fresh crawl: GSC → URL Inspection → https://id86.net/ → Request indexing.
Confirm the stale "noindex" inspection clears after the new crawl.
Confirm Search Analytics starts accumulating clicks/impressions.

Purpose​

Current Automation Stack​

Why Not a Custom API Solution​

How the Sync Triggers Indexing​

SEOPress WP-CLI Commands​

Settings Backup / Restore​

Per-Post SEO Metadata​

AI Metadata Generation​

Global Settings​

License Management​

SEOPress Metadata Sync (implemented)​

GSC Verification Status​

Site Visibility State (under development)​

Development-Safe Checks​

Launch Checklist (when site goes public)​

Verification Checklist​