A SaaS pricing page or documentation site can look complete in a browser while its initial server response contains little more than a script shell. That difference can matter: Google documents JavaScript rendering, while a Vercel/MERJ analysis reported that the AI crawlers it tested did not render JavaScript. That vendor snapshot is not a permanent rule for every crawler. For critical public pages, check what each system can actually fetch and expose important content in crawlable HTML where practical.
Can AI crawlers render JavaScript pages?
Google documents that it processes JavaScript pages, but a Vercel/MERJ analysis reported that the major AI crawlers included in its test—including OpenAI search/user agents, ClaudeBot, and PerplexityBot—did not render JavaScript. The analysis is a provider- and time-specific observation, not an exhaustive or current census. Fetching a URL, seeing its meaningful content, and citing it in an answer are separate events.
What the supplied evidence reports | What a site team still needs to verify |
|---|---|
Google processes JavaScript pages and provides guidance for checking rendered content and crawling/indexing. | Whether Google renders and indexes a particular URL as intended at the time of testing. |
Vercel/MERJ reported non-rendering among the AI crawlers it tested, including the named OpenAI, Anthropic, and Perplexity agents. | Whether the relevant agent, product, request type, and current site response behave the same way today. |
A request to a URL can indicate a fetch, but does not establish that all page content was available or later cited. | Whether the request was from a verified agent, what response it received, and whether the page appears in repeatable answer checks. |
The Vercel/MERJ article is dated December 17, 2024. Its relative “past month” request counts belong to that historical snapshot and should not be reused as current benchmarks. A practical decision should therefore rely on your own dated HTML, rendered-page, and access-log observations—not a universal claim about every AI crawler.
How is Google's JavaScript rendering different?
Initial HTML is the server response before client-side scripts execute; the rendered DOM is the page structure after a browser processes scripts. Google documents how it processes JavaScript pages and recommends checking rendered content and crawling/indexing behavior. That guidance describes Google’s systems, not independent AI crawlers, so Google indexing cannot establish that another provider fetched or rendered the same URL.
Google’s AI features use its existing Search eligibility framework. A page must be indexed and eligible for a Search snippet; Google describes no additional technical requirements for those features. This is not evidence of third-party access. OpenAI, Anthropic, and Perplexity document their own bots and controls, so their requests and permissions need to be considered separately. Google Search Console reports Google’s systems; it is not a cross-platform crawler log.
Google status | Third-party fetch evidence | Citation observation | Cautious interpretation |
|---|---|---|---|
Indexed or eligible | Unknown | Not observed in a small sample | Google status alone says nothing conclusive about a third-party fetch or citation. |
Indexed or eligible | Verified request, response content unknown | Not observed | A request occurred; confirm the returned content and repeat citation checks before interpreting visibility. |
Indexed or eligible | Verified request with relevant content returned | Citation observed in a dated check | The page appeared in that observation; this does not establish a durable pattern or prove what caused it. |
Unknown or not indexed | No verified request | Citation status unknown | The available evidence is incomplete; investigate Google status and third-party access separately. |
Which SaaS page elements should appear in initial HTML?
Prioritize information a buyer or technical evaluator needs to understand without depending on a client-side interaction: core product facts, pricing, documentation, comparisons, supporting evidence, and crawlable internal links. Server-side rendering generates page HTML on the server; static rendering prepares HTML ahead of requests. Both can make that content available in the initial response, but neither is a universal prescription. Choose an implementation that fits the page’s freshness, functionality, and engineering constraints.
A focused initial-HTML review can ask:
Are the product’s purpose, audience, and core capabilities present as readable text?
Can a visitor or crawler reach public pricing and plan details without first triggering an interface action?
Do documentation pages include the substantive instructions, not only navigation or a loading shell?
Are comparison criteria and supporting evidence available in the response rather than hidden behind client-side loading?
Are important internal links represented as crawlable links with meaningful destinations?
Do canonical, redirect, and page-status signals point consistently to the intended page?
These are candidates to assess, not a claim that every page must be fully duplicated in HTML. For example, an interactive product configurator may need client-side behavior; its explanatory copy, key options, and links can still be exposed in the initial response where feasible. For the permissions side of the audit, see the AI crawler robots.txt policy. Robots permissions and content rendering are related checks, but they are not the same issue.
How do you test what a bot actually receives?
Compare the initial HTML, browser-rendered output, and observed requests for the same URL at a recorded time. A user-agent string alone does not prove that a request came from a provider’s crawler. A test fetch also does not prove that the provider processes content in the same way across products. Keep the evidence tied to the URL, response, agent, and test date.
Select representative public pages: for example, one pricing page, one documentation page, and one comparison page. Record why each represents a meaningful content type.
Fetch each URL and save the raw response HTML, response status, redirect chain, and timestamp. Note the user agent and test method.
Compare the raw response with the browser-rendered DOM. Check whether the core text, evidence, and links are present in both, and note content that appears only after scripts run.
Check robots.txt rules, canonical signals, response status, and relevant CDN or WAF decisions for each agent. A policy file alone does not show what the server returned.
Inspect verified bot logs where available. Record how the agent was verified, the requested URL, status, and any limitations in log coverage or bot identification.
Retest after a change using the same pages and comparable observations. Keep access outcomes separate from citation outcomes.
URL/content | Initial HTML | Rendered DOM | Verified bot fetch | Outcome |
|---|---|---|---|---|
Pricing page — record URL and test time | Present / partial / absent; note key facts | Present / partial / absent; note script-dependent content | Agent verification, request time, status, returned content | Access gap, no observed gap, or unknown; state limitations |
Documentation page — record URL and test time | Present / partial / absent; note instructions and links | Present / partial / absent; note script-dependent content | Agent verification, request time, status, returned content | Access gap, no observed gap, or unknown; state limitations |
Comparison page — record URL and test time | Present / partial / absent; note criteria and evidence | Present / partial / absent; note script-dependent content | Agent verification, request time, status, returned content | Access gap, no observed gap, or unknown; state limitations |
Keep crawler-policy implementation distinct from rendering checks. A policy file does not replace inspection of the actual response or verified bot requests.
Does being fetched guarantee an AI citation?
No. Crawlability can support some discovery paths, but a fetch does not prove that a system selected, indexed, or cited a page. Track access evidence separately from answer visibility: record target queries, dates, pages, and citations over a defined window, then compare repeated observations without treating a rendering change alone as the cause of any difference.
A useful measurement plan keeps the questions distinct: could the relevant agent access the content, and did the page appear in the answers being monitored? Neither result guarantees the other.
Does Google indexing prove AI systems can access a page?
No. Google indexing or eligibility for Google Search AI features does not prove that ChatGPT, Claude, or Perplexity fetched or indexed the same page. Google’s eligibility guidance applies to Google’s Search AI features. OpenAI, Anthropic, and Perplexity document separate bots and controls, so verify their access independently; even confirmed access does not establish citation.
Google’s status is useful evidence about Google, not a proxy for every external crawler. Check each relevant user agent’s robots policy, content delivery network (CDN) or web application firewall (WAF) decisions, HTTP response, redirects, and rendered content. Provider documentation describes declared behavior and controls; your logs and tests show what your site served under the observed conditions.
How do you test access without confusing it with AI visibility?
Check each relevant user agent’s robots policy, WAF/CDN decisions, and HTTP responses, then inspect rendering and redirect behavior. Separately record whether target pages appear in actual AI answers with citations; technical eligibility does not guarantee inclusion. The agent documentation from OpenAI, Anthropic, and Perplexity informs the provider-specific access checks, not a guarantee of citations.
Inspect applicable robots rules for the relevant agents and distinguish permission from successful retrieval.
Check logs and response statuses, noting verification limits and possible gaps in logging.
Test canonical and redirect destinations to confirm that requests and references resolve to the intended content.
Inspect JavaScript-dependent content by comparing initial HTML and rendered output.
Record dated queries, prompts, target pages, and citation outcomes over a defined observation window.
Bot identification and logs can be incomplete; treat an unverified user-agent string as an observation, not proof of provider identity. Keep unknown states explicit rather than converting missing logs into a claim that no fetch occurred.
What does a Google-indexed but uncited page tell you?
It establishes neither an AI-crawler failure nor a content-quality failure. Check access evidence first, then assess relevance and answer content, and repeat citation observations across an appropriate set of queries; absence from a small sample does not prove exclusion. Use a website migration SEO checklist to frame release checks, but not as a substitute for page-specific evidence.
When is rendering work worth prioritizing?
Prioritize a business-critical public page when meaningful text or links are missing from initial HTML and verified bot observations indicate an access gap. Balance that gap against engineering cost, content freshness, functionality, and the page’s existing Google performance. The available evidence does not identify one rendering framework as best or establish a guaranteed visibility lift.
Page importance | Initial HTML completeness | Verified bot access | Implementation effort | Practical priority |
|---|---|---|---|---|
High: important public product, pricing, or documentation page | Core content materially absent | A relevant, verified request shows an access or response gap | Feasible to expose content without undermining required functionality | Investigate and prioritize a targeted change; retest afterward. |
High | Core content present | No current access evidence | Any | Verify bot access and response before changing architecture. |
Medium or low | Some secondary content absent | Access is verified for the content that matters | High | Compare the likely value of a focused HTML change with its cost and freshness trade-offs. |
Any | Unknown | Logs or identity are inconclusive | Any | Improve the observation first; do not infer a failure from missing evidence. |
A migration can change rendering, redirects, and access behavior together. Use the website migration SEO checklist for release QA, then record which specific change corresponds to each observed outcome. Avoid attributing a later citation or traffic change to rendering alone when several factors changed.
What should teams conclude about JavaScript and AI discovery?
JavaScript-heavy delivery can create a discoverability risk for some crawlers, but the Vercel/MERJ result is a bounded vendor snapshot, not a permanent rule for all providers. Expose critical content in crawlable HTML where practical, then measure real access and downstream answer outcomes separately. Google rendering is useful for Google; it does not establish third-party fetching or citation. A verified request describes access under observed conditions, not selection; repeated query-level checks assess citations, and neither measure alone establishes causation.
Frequently Asked Questions
Can AI search tools render JavaScript content?
The supplied Vercel/MERJ analysis reported that the AI crawlers it tested did not render JavaScript, while Google documents JavaScript rendering for its own systems. Treat the vendor result as a time- and test-specific snapshot, and verify current access and returned content for each relevant provider; a fetch does not guarantee a citation.
Can AI crawl websites?
Yes, AI-related systems can retrieve public web pages, but access depends on the provider, crawler purpose, site rules, and the response the site serves. A request does not show that the page was fully processed or cited in an answer.
Is ChatGPT a web crawler?
ChatGPT is a product, not one single crawler. OpenAI documents different agents, including OAI-SearchBot for search discovery, GPTBot for content that may be used in model training, and ChatGPT-User for certain user-requested fetches; their purposes and controls differ.
How Nortav Can Help
For SaaS teams assessing JavaScript-heavy pricing, documentation, or comparison pages, the first step is to separate rendering, crawler access, and buyer-facing content checks. Priorities should follow what the initial HTML and verified requests show.
Our approach starts with understanding your current search visibility, identifying gaps against competitors, and developing a strategy focused on driving qualified traffic, generating leads, and delivering measurable business value.
Book a Discovery Call →
Sources
Google Search Central, “AI features and your website”:
Google Search Central, “Understand JavaScript SEO Basics”:
Vercel, “The rise of the AI crawler” (analysis by Vercel/MERJ ):
OpenAI, “Overview of OpenAI Crawlers”:
Anthropic, “Does Anthropic crawl data from the web, and how can site owners block the crawler?”:
Perplexity, “Perplexity Crawlers”:
