Skip to main content

Search Engine Indexing

Purpose​

When the site is public, ensure every new documentation post published from the GitHub Markdown repository is discoverable through the sitemap and publish-time notifications. During development, Cloudflare Zero Trust intentionally blocks crawlers and indexing is not expected.

Current Automation Stack​

LayerMechanismProvider
Sitemap generationXML sitemap updated on every post saveSEOPress + Pro
Instant indexingIndexNow submission on publishSEOPress + Pro and participating engines
Google Indexing APINot used for ordinary documentationGoogle restricts it to JobPosting and BroadcastEvent
Publish-time pingwp_insert_post triggers WP ping / search-engine notifyWordPress core
Content triggerGitHub push → sync.py → wp_insert_postGitHub Actions

Why Not a Custom API Solution​

The GSC Indexing API only accepts pages with JobPosting or BroadcastEvent structured data — not general documentation. The Search Console API reads performance data (queries, clicks, positions) but cannot submit URLs. There is no public API to force-index regular pages; Google decides crawl timing.

Therefore indexing automation relies on:

  1. XML sitemap submitted once in Search Console — new posts are discovered on each crawl pass.
  2. IndexNow / Instant Indexing — fires automatically on publish for participating engines, the closest thing to "auto-register new content."
  3. WordPress publish-time ping — already triggered by sync.py's wp_insert_post.

How the Sync Triggers Indexing​

SEOPress WP-CLI Commands​

SEOPress exposes official WP-CLI commands for SEO automation. These manage metadata, settings, and license — not indexing itself (indexing is automatic on publish).

Settings Backup / Restore​

wp seopress settings export --destination='/home/Dmg59ZFtKg6bIws1/id86net/public_html/wp-content/uploads/seopress'
wp seopress settings import --from='/home/Dmg59ZFtKg6bIws1/id86net/public_html/wp-content/uploads/seopress'

Per-Post SEO Metadata​

wp post meta get 123 _seopress_titles_title
wp post meta update 123 _seopress_titles_title "my super SEO title"
wp post meta delete 123 _seopress_titles_title

AI Metadata Generation​

wp seopress ai metadata --id=1,2,3,4,5 --meta=desc --language=en_US
wp seopress ai alt_text --id=1,2,3 --language=en_US

Global Settings​

wp option get seopress_titles_option_name
wp option patch update seopress_titles_option_name seopress_titles_home_site_title "my super homepage title"
wp option delete seopress_titles_option_name

License Management​

wp seopress-pro license activate --key=YOUR_LICENSE_KEY
wp seopress-pro license status
wp seopress-pro license update
wp seopress-pro license reset

SEOPress Metadata Sync (implemented)​

The sync pipeline writes dedicated per-post SEO metadata at publish time via wp-sync-bridge.php:

sync.py → wp_insert_post → update_post_meta(_seopress_titles_title, <title>)
→ update_post_meta(_seopress_titles_desc, <description>)
→ post_excerpt = <description>

This applies SEO titles/descriptions automatically during initial publication and unpublished updates. Published posts are protected from later Markdown syncs; live SEO edits belong in WordPress/SEOPress. See SEO Metadata Automation for the full flow, authoring rules, and verification commands.

GSC Verification Status​

ItemStatus
Propertysc-domain:id86.net — verified
Verification methodDNS TXT (google-site-verification=... — value in ~/.ssh/google/.env, GOOGLE_SITE_VERIFICATION_TOKEN) added via Cloudflare API
TXT record locationid86.net zone, DNS record ID 260f23c3195298481155aaaac7fa4d5b
Service account accesssiteFullUser — API pulls work end-to-end
Sitemaphttps://id86.net/sitemaps.xml — endpoint responds, but current external check returns HTML instead of valid XML; fix before launch

Rule: do not remove the DNS TXT record — removing it revokes GSC ownership. To harden, add a second verification method (e.g. HTML file) from GSC Settings → Ownership verification.

Site Visibility State (under development)​

id86.net is still under development and protected by Cloudflare Zero Trust Access (email OTP). This is intentional and must remain enabled during development. Consequences:

  • Googlebot is blocked — all public requests get a 302 to the Access login (rezriz.cloudflareaccess.com/cdn-cgi/access/login/...), including crawlers. Google's current inspection is stale (last crawl 2026-07-14) and reads "excluded by noindex" even though the live origin serves index, follow.
  • GSC data stays empty until launch — Google cannot crawl, so Search Analytics shows nothing.
  • GA4 only counts authenticated visitors during development (expected).

These are expected during development, not misconfigurations. See the Launch Checklist below.

Development-Safe Checks​

Run checks without changing Cloudflare Access:

cd /home/rezriz/github/seo-content/web-project/id86.net
python3 scripts/check-indexing.py --sitemap --robots --development \
--report reports/indexing-development.json

The report records the expected Access redirect as development_protected. Do not add a crawler bypass while the site is private.

The checker also verifies response content types and parses the sitemap XML. A HTTP 200 response is not sufficient: an HTML error or login page must fail the check.

Launch Checklist (when site goes public)​

  • Decide whether to remove Zero Trust protection or add a narrowly scoped Cloudflare Access bypass policy for verified bot user-agents (Googlebot, Google-Read-Aloud, Bingbot, etc.) on the id86.net Access application. Do not perform this during development.
  • Re-verify Googlebot gets a real 200 on https://id86.net/ and https://id86.net/sitemaps.xml from outside Cloudflare.
  • Trigger a fresh crawl: GSC → URL Inspection → https://id86.net/ → Request indexing.
  • Confirm the stale "noindex" inspection clears after the new crawl.
  • Confirm Search Analytics starts accumulating clicks/impressions.

Verification Checklist​

  • GSC property verified (sc-domain:id86.net)
  • Service account can query Search Analytics API
  • SEOPress XML sitemap returns valid XML with an XML content type (https://id86.net/sitemaps.xml)
  • Sitemap submitted once in Google Search Console
  • New posts appear in sitemap after sync (no manual steps)
  • IndexNow / Instant Indexing enabled in SEOPress Pro
  • Development indexing check script supports sitemap, robots.txt, URL checks, and JSON reports
  • robots.txt returns plain text and includes the sitemap URL
  • After a sync, confirm the new URL is eventually indexed via GSC URL Inspection
  • Launch: crawler access policy decided and tested (see Launch Checklist)