Our five product workflows were completely invisible to the first-pass crawl.
Not ranked badly, absent. Every product name, benefit title and description was
fetched from an internal endpoint after page load and written into the DOM with
innerHTML, so the HTML a crawler received contained one relevant string:
Loading Platform Architecture....
We rebuilt the site so that content is server-rendered. This is what we found.
The failure was measurable, and it was zero
The check takes one command:
curl -s https://archmir.com/platform/expertarch | grep -c "Expert Level Reasoning Layer"
Before the rebuild that returned 0. The page had a title, a description and
Open Graph tags, all injected server-side by string replacement in a template,
but no body content. The metadata described a page whose content did not exist
until JavaScript ran.
This matters more than it used to. Googlebot indexes in two passes: the first reads raw HTML, and anything requiring JavaScript is queued for the Web Rendering Service, where it may wait hours or weeks. Crawlers that never execute JavaScript, including social unfurlers and a growing set of AI retrieval agents, never see the content at all.
What we changed
The content moved from a client-side fetch into Server Components. The interactive behaviour stayed, but it now attaches to markup that is already in the HTML rather than creating it.
| Before | After |
|---|---|
Catalogue fetched from /api/content/platform, injected with innerHTML | Rendered by the server; present in the initial HTML |
Product pages built at runtime by a _renderIsolationMode() function | Statically generated at build time, one file per product per language |
Language switched with a ?lang= query parameter | Separate indexable paths, with hreflang between them |
| A single Open Graph image for every URL | A generated card per route |
Heading outline started at h3 with no h2 above it | One h1, then a real h2/h3 hierarchy |
The interactive parts, tab switching, scroll reveals, video playback and the contact form, are client components mounted over server-rendered markup. Users lose nothing.
Three things that were not obvious
Hiding content behind opacity: 0 is a real risk. Our scroll animations
start every section invisible and reveal it with JavaScript. Googlebot renders
JavaScript and would eventually see it, but a non-JS client would see a blank
page. A <noscript> block that neutralises the animation costs nothing and
removes the failure mode entirely.
Blocking a URL in robots.txt does not remove it from the index. We had
planned to disallow the old ?lang= URLs. That would have been backwards: a URL
a crawler cannot fetch is a URL whose redirect it can never discover, so the old
address would have stayed indexed indefinitely. Redirects need to be crawlable
to work.
AI crawlers need an explicit invitation. Google-Extended and
Applebot-Extended are opt-out tokens for AI grounding. Saying nothing leaves
the decision to a default. If you want your documentation to be reachable by
generative engines, list them under Allow.
The part that is not automatable
Rendering strategy, structured data and Core Web Vitals are engineering problems, and engineering problems have deterministic fixes. What none of them substitute for is having something specific and verifiable to say, which is why the numbers in this post are the ones we measured on our own site rather than benchmarks quoted from someone else's.