A private preview, a public launch, and a retired page need different treatment. Write the intended state before changing robots.txt or page metadata. The technical configuration should implement that decision, not accidentally make the decision for the owner.
Use three separate questions: who may access the content, whether a crawler may fetch it, and whether an accessible resource should appear in search. These controls are related but not interchangeable. A private document should not depend on a polite crawler instruction for protection.
Google's robots introduction, checked September 9, 2026, explains that robots.txt manages crawling and is not a security mechanism or a reliable way to hide a webpage's URL from search. The robots metadata guidance also explains that a crawler must fetch a resource to discover its page-level or response-header indexing instructions.
Make a release-state matrix
The following is a planning example. Have the site's owner and maintainer approve the actual state and implementation before changing access.
| State | Access decision | Search decision |
|---|---|---|
| Private owner review | Require the intended access control | Keep the edition outside public acquisition plans |
| Public educational page | Accessible to its intended public audience | Allow intended discovery and review indexing directives |
| Public utility page excluded from search | Publicly usable | Apply an appropriate readable noindex instruction |
| Retired resource | Follow the approved retirement decision | Return the intended response or relevant replacement |
Do not treat “public but noindex” as equivalent to private. Anyone with access to the URL may still see publicly accessible content. Conversely, a private site's sitemap does not make its pages a functioning search acquisition channel.
Review a hypothetical launch mistake
Imagine a team building a product site behind owner-only access. During preparation it also uses disallow-all robots rules and noindex metadata. Those settings communicate the preview state, but access control is what restricts who can view the content.
Later, the owner approves a public launch. The release checklist should explicitly review access, canonical domain, robots rules, and indexing metadata. Changing only the audience may leave the public site excluded from intended discovery. Removing only noindex may leave the site inaccessible.
The owner should approve the launch state once; the maintainer then implements and verifies that concrete decision. A routine content update to a private edition should not silently become a public launch.
Inspect actual responses rather than configuration names
Check the effective access behavior, robots file, HTML metadata, and relevant response headers. A setting called “SEO enabled” is not evidence that every route has the desired behavior. Shared templates and hosting layers can each contribute directives.
Test representative route types: homepage, article, product, download, and utility page. Record the intended result and the actual response. A PDF may need a response-header rule because it does not carry HTML metadata in the same way as a webpage.
If directives conflict, trace their sources and resolve the conflict deliberately. Do not add another rule on top merely because it makes one testing tool report a different label.
Keep the decision in the release record
Record who approved the state, which routes it covers, when it was implemented, and the evidence checked. Preserve a rollback path for accidental exposure or unintended exclusion. Avoid changing a domain or audience as a side effect of publishing an article.
Use the technical launch checklist for the wider release. Revisit this matrix when access, domains, templates, or commercial availability change. A clear state model lets the team move from review to launch without guessing what the owner intended.