How to Audit LLM Answers About Your Company

Assess an LLM (large language model) by comparing repeated answers to a dated, verified company fact sheet across a fixed set of buyer questions and systems. Score checkable claims, omissions, outdated details and unsupported statements separately, and preserve the answer, citations and run metadata. This tests what a system says under specified conditions—not what it privately knows or whether it “understands” a company.

Can one fluent LLM answer show that it understands your company?

No. One fluent answer is a single observation, not evidence of stable factual accuracy or internal understanding. Compare repeated answers to predetermined buyer questions against a dated fact sheet, preserving source citations and omissions. Accuracy, brand mentions and citations are different measurements: a system can name a company while describing it incorrectly, or cite a source without accurately representing it.

The finite-sample name-recovery results in the 2026 preprint do not measure company factual accuracy. A separate preprint argues for treating visibility as a distribution across repeated queries rather than a one-time score.

A practical audit therefore asks whether observable, checkable statements match authoritative evidence. It does not try to infer a model’s hidden mental state. Keep visibility questions separate: if an audit asks whether a brand appeared, that is a mention or exposure measure, not a factual-accuracy score. For a related distinction, see why does ChatGPT mention competitors but not your brand; that is not a substitute for checking whether descriptions are correct.

What should a verified company fact sheet contain?

Before testing answers, create a dated reference for the company’s category, products, integrations, pricing, ideal customer and limitations. For each fact, record an authoritative public source, when it was last checked, when it applies and who reviewed it. A missing public source is a reason for human review; it should not automatically count as an LLM error.

For each fact, record the authoritative URL, last-verified date, applicability and reviewer. Prefer company-controlled documentation for product details, but use the best authoritative evidence available for other claims. Record disagreements between public pages and route them to an owner rather than choosing whichever version supports a preferred result. Assign a reviewer to adjudicate contested or time-sensitive facts.

Competitor comparisons need special care. A claim that a company is “the leading” option or is “better than” a named competitor is not an immutable company fact. It needs a defined comparison, current evidence and a clear basis. Do not score a model wrong merely because it omits an unsubstantiated superlative from marketing copy.

Google says Organization structured data can help Google understand organization details and disambiguate entities in Search. That statement concerns Google Search; it does not establish an effect on other LLMs or prove what any system knows. The source is Google’s Organization structured data documentation. The fact sheet should remain grounded in verifiable company information, not assumptions about markup’s reach.

How do you build a repeatable buyer-question audit?

Set the questions, engine and model configuration, time window, repetitions and scoring rules before examining outputs. Save every full answer, citation and run detail so disagreements and changes can be inspected later. One prompt or mention rate is not enough to characterize answer accuracy.

1. Select question categories. Cover category, products, intended customers, integrations, pricing, limitations and competitors. Use questions a buyer could reasonably ask, not only wording that favors the company.

2. Freeze the prompts. Save exact prompt text and any relevant context. These examples are illustrative prompts, not observed People Also Ask questions: “What does this company sell?”, “Who is it for?” and “Which integrations are confirmed?”

3. Define systems and runs. Record the engine, model or configuration when visible, date and time, relevant account or personalization conditions, and number of repetitions. If a system does not expose a version, record that limitation rather than guessing.

   With independent runs and fixed error probability p, use n = ceil[ln(1 − C)/ln(1 − p)] to calculate the run count. Here, n is the number of runs; ceil means round up to the next whole number, and C is the target chance of observing at least one error. At p=10%, C=95%, n=29; at p=5%, C=95%, n=59. This detects an error, not its rate; zero detections do not prove none exists. The calculation applies the independent-binary-trial model described in NIST Technical Note 2119. Runs may not be independent.

4. Capture complete outputs. Preserve answers as returned, including citations and links, not just snippets or selected claims. Keep the prompt and run metadata beside each response.

5. Score claims. Extract checkable statements and compare them with the fact sheet. Blind reviewers to the engine where feasible, so expectations about a particular system do not drive labels.

6. Reconcile disagreements. Have reviewers resolve ambiguous or disputed claims against the recorded evidence. Keep the original judgments and note the decision rule; do not hide disagreement by silently averaging incompatible labels.

7. Report denominators and dates. Show how many questions, answers and claims were examined, what counted as eligible, and the dates and configurations covered. A rate without its scoring unit and denominator is difficult to interpret.

Prompt wording, retrieval, personalization and model or system versions can vary. Treat them as possible sources of variation to document, not as effects with a known size unless the audit measures them. Keep the question set and method stable for comparisons, and annotate any changes that make runs less comparable.

How should you score factual accuracy and omissions?

Label each extracted, checkable claim as correct, incomplete, outdated, conflated or unsupported, and track missing required facts separately as omissions. Record the scoring unit, denominator, evidence URL and adjudication for each label. A citation is not proof that the associated claim is true: check whether the cited page actually supports it.

Set in advance which facts are material for each question. Do not count every absent detail as an error; an answer cannot include every fact about a company. Conversely, do not call an unsupported assertion a factual error unless evidence shows it is false. Preserve both the source cited by the answer and the fact-sheet evidence used to assess it.

A claim-level scoring rubric makes those decisions inspectable. If you also track brand appearance or citations, report them as separate measures rather than combining them into an “understanding” score. Measure exposure separately from claim accuracy; mention rate is not a factual-accuracy score.

What can repeated-query research tell us—and what can it not?

A 2026 preprint sampled 4,500 responses to 50 buying questions across six model configurations, with 15 calls per question/configuration cell. Within that study’s sample, estimated median ten-call recovery of names observed in each 15-call set was 92.6%–95.2% under a historical name dictionary and 89.5%–94.7% after re-adjudicating 45,683 candidate strings. These are conditional name-recovery results—not exhaustive discovery of all possible names, a benchmark of factual accuracy about a company, or evidence of internal understanding.

The denominator matters: the ten-call exercise was compared with names observed in the study’s 15-call set. It estimates recovery within that sampled set, not how many names exist beyond it. The re-adjudicated figures also reflect a changed name-review process. Do not generalize these percentages to every brand, model or future period. The work is a preprint; confirm its publication status and details when using these findings.

A separate 2026 preprint argues that visibility should be treated as a distribution across repeated queries rather than a one-time score. That is a proposal, not settled evidence that a particular rubric or sample size is optimal. Repeated runs can reveal variation within a defined audit, but they do not remove the limits of the question set, systems, dates or scoring process.

A useful research question is whether blinded human judgments align with automated fact-extraction judgments across models and versions, and whether clearer public evidence changes answer accuracy. Until such evidence is available, avoid treating mention rates, name recovery or citation presence as a proxy for accurate company descriptions.

How should you report the audit without overstating certainty?

Present dated, per-engine observations about claim accuracy and omissions alongside sample sizes, question mix, rubric and adjudication rules. Show examples of corrected and unverified statements, and make clear that the audit measures answers observed under specified conditions—not hidden model knowledge or guaranteed future citations.

For each rate, state which claims are included in the total and which claims form its denominator. Report visibility or citation observations separately, with their own definitions. Do not imply that a cited page caused an inaccurate answer merely because it was cited; inspect source ownership and support independently. For source ownership, see B2B SaaS AI source mix.

A useful report distinguishes the result from its limits. For example, it can say that a certain number of claims in a dated sample matched the fact sheet, while noting that the prompts, engine configurations and adjudication rules constrain interpretation. It should not claim that the company has—or lacks—visibility across all AI answers, or that a site edit will automatically improve answers across systems.

Closing perspective

To assess how accurately an LLM describes a company, audit repeated answers against a dated, source-linked fact sheet and score claims and omissions with explicit rules. Preserve the prompts, outputs, citations, dates and denominators. The result describes observable answers in that audit; it does not reveal a model’s private knowledge or predict every future response.

Frequently Asked Questions

What is the difference between LLM accuracy and AI visibility?

Accuracy measures whether an LLM’s checkable claims match reliable evidence. Visibility measures whether a company appears in answers or citations. A brand can be visible but described incorrectly, so the two measures should be reported separately.

How Nortav Can Help

For teams auditing how AI systems describe their company, Nortav can help assess the current measurement approach, identify gaps in how competitors are represented, and shape a strategy around business-relevant outcomes. Our approach starts with understanding your current search visibility, identifying gaps against competitors, and developing a strategy focused on driving qualified traffic, generating leads, and delivering measurable business value.

Book a Discovery Call →

Sources

• “Measuring Brand and Source Discovery under Repeated LLM Queries: A Finite-Sample Audit” (2026 preprint).

• Google Search Central, “Organization structured data.”

• “Don't Measure Once: Measuring Visibility in AI Search (GEO)” (2026 preprint). https://arxiv.org/abs/2604.07585 - NIST Technical Note 2119.