A&A INSIGHTS
When research documents disagree, what should AI summarize? Preserve grounds and open conditions, not averaged conclusions
When outside research reports and vendor comparisons reach conflicting conclusions, asking AI to summarize them often returns a bland average. Drawing on overseas cases, this article shows how owners can prompt AI to preserve disagreement, grounds, and unresolved conditions instead.
日本語で読むTHE STARTING POINT
It is not unusual for multiple research documents on the same question to reach different conclusions. If AI is simply asked to summarize them, the pro-and-con positions get averaged into a safe, characterless statement, and the grounds and open conditions needed for actual decisions get discarded. Drawing on Anthropic's parallelization workflow and Quillit's approach to citation traceability, this article outlines how to design prompts that keep disagreement and conditions visible instead of merging them. The intended reader is an owner who has to combine multiple outside reports into an investment or hiring decision.
Asking AI to "summarize" conflicting documents strips out the grounds for decisions
A&A perspective
It is common in management decisions to hold several outside documents that reach different conclusions on the same question. One vendor comparison report favors Product A, another industry survey favors Product B, and a third consulting document recommends both only under specific conditions. Asked to "summarize these three," AI will most often return a safe, hedged synthesis. The disagreement flattens, and what remains is a surface-level conclusion that treats every source as if it were equivalent. What is actually useful for a decision is nearly the opposite: a form that preserves why the conclusions differ, which premises are not shared, and which points remain unresolved. Under the label of summarization, the very information a decision-maker needs gets discarded. Before searching for a better summary prompt, the more basic problem is to name what AI should preserve, not what it should produce as a single answer.
From the sources
The Anthropic-published Quillit case study frames the core problem in qualitative research as one where "Critical insights risk getting buried in transcripts and notes." The case study is about interview transcript analysis; it does not directly address the reconciliation of multiple external research documents. The application of that observation to multi-document reconciliation in the sections that follow is Automate & Augment's reading, not a claim made in the Quillit case study.
The parallelization workflow view: preserve multiple outputs instead of merging them
From the sources
Anthropic's engineering post on building effective agents describes, as one form of the parallelization workflow, "Voting: Running the same task multiple times to get diverse outputs." The post notes that parallelization is useful when "multiple perspectives or attempts are needed for higher confidence results." Illustrative uses include reviewing code for vulnerabilities with several parallel prompts, and evaluating whether content is inappropriate using multiple prompts that assess different aspects, so that disagreement is preserved rather than collapsed at the last step.
A&A perspective
This pattern maps directly onto the situation of an owner reconciling multiple research documents for a decision. Instead of asking AI for a merged conclusion across three documents, run the same well-defined question against each document in isolation, and then present the responses side by side with their supporting grounds. The design change is not a matter of tuning the summary prompt. It is a change in what AI is asked to output at all: not a single conclusion, but a preserved set of positions with supporting grounds. The multiple parallel responses become the raw material the decision-maker then compares. Rather than asking AI to pick a conclusion, the owner asks AI to prepare the material from which the owner will pick a conclusion. The division of labor is what shifts.
Citations back to grounds: extending Quillit's approach to multi-document reconciliation
From the sources
The Quillit case study identifies one of Claude's key roles in the product as "Generating contextual citations: Creating precise references to transcripts and video recordings." The case study reports that citation accuracy improved from a 60-70% range with earlier models to a 89-98% range with Claude 3.5. The ability to trace conclusions back to the source utterance is positioned in the case study as central to converting research summaries into decisions.
A&A perspective
The same "trace back to the source" principle can be carried over when the primary material shifts from interview transcripts to external research documents. When summarizing multiple reports, what should be required of AI is not a single narrative conclusion but precise references from each claim back to the passage in the original document. When those references are present, any claim on which the reports disagree can be re-examined at the source: the sample size, the scope, the time period, and the underlying premises can be checked one by one. A summary without those references may read fluently, but it forces the decision-maker to re-read every document anyway. Quillit implements citation generation as a product feature on the Claude Platform. The point being borrowed here is not the product implementation but the design principle, which applies just as well to the prompt an owner writes when combining outside reports.
A prompt design pattern: separate positions, grounds, and open conditions
From the sources
The Anthropic engineering post also describes an evaluator-optimizer workflow, in which "one LLM call generates a response while another provides evaluation and feedback in a loop." The post lists complex search tasks as a fitting use case, noting that "the evaluator decides whether further searches are warranted." The two roles — generator and evaluator — are structured as separate calls with distinct responsibilities.
A&A perspective
Combining these two patterns yields a concrete prompt design shape for reconciling documents that disagree. In the first pass, each document is processed independently against the same question, and the response is required to break out four things: the claim, the citation passage that grounds it, the assumed premises, and the points the document does not address. In the second pass, a separate call arranges the responses side by side and produces a table that distinguishes points of agreement from points of disagreement, and, for the disagreements, states the reason (differences in assumed premises, in scope, or in the reference period). In the third pass, an evaluator call is asked to list the open conditions on which the final conclusion could still change and to enumerate what additional confirmation is required. The three passes are structural. Their point is precisely to keep AI from producing a merged conclusion. What the decision-maker reads is the annotated comparison table and the list of open conditions, not a conclusion that AI chose.
Hypothetical: reconciling three reports on whether to adopt retrieval-augmented generation
Hypothetical example
Imagine a fictional small company weighing whether to adopt retrieval-augmented generation (RAG) for its internal knowledge search. It holds three outside documents on the question. Document A concludes that RAG is effective for small-to-mid-scale data. Document B concludes that maintenance overhead is excessive and recommends fine-tuning instead. Document C concludes that it depends on the use case, and that adoption criteria should be defined first. Asked to "summarize the three," AI will most likely return something like: "Effective in some cases, but be careful about maintenance." That is a partial reflection of every document with the disagreement flattened. With the pattern above substituted in, the first pass produces, per document, the claim, its grounding passage, its assumed premises, and its unaddressed points. The second pass yields a comparison table that marks the points of agreement (all three mention the importance of operating capacity) and the points of disagreement (assumed data volume, update cadence, the acceptable margin of answer error). The third pass has the evaluator surface the open conditions that could still change the conclusion — for example, the number of expected queries per period, the agreed acceptable error margin, and the number of people available to maintain the pipeline. This hypothetical is illustrative and does not represent any real company, product, or research outcome.
A&A perspective
What this hypothetical illustrates is that when the prompt design changes, the artifact that ends up on the owner's desk changes. Tuning the summary prompt only produces variations on an averaged conclusion. Requiring an annotated comparison table plus a list of open conditions produces a different kind of artifact: one on which the owner can then layer their own knowledge ("for our situation, we can settle the expected query volume and the acceptable margin at this range") and re-evaluate the strength of each report's claim. The order matters. The owner is not adopting the conclusion AI chose. The owner is choosing a conclusion on top of the comparison material AI prepared.
A minimum prompt-design change you can try today
A&A perspective
Trying this approach in-house tomorrow does not require building any workflow tooling. Take three documents already at hand and rewrite the existing summary prompt with just three additions. First, state explicitly that a merged conclusion is not what is being requested. Second, require that each document be processed into four fields: claim, citation passage, assumed premises, and points not addressed. Third, when asking for a comparison across the three, require that points of agreement and disagreement be separated, and that the reason for each disagreement be recorded as a difference in premises, scope, or reference period. These three additions alone visibly change the shape of the output. Tool selection and additional automation can be layered in later, after the artifact shape has been confirmed as useful for the actual decision.
From the sources
The Quillit case study reports up to an 80% reduction in report writing time from adopting Claude, while also stating that "Anthropic's emphasis on data ethics and governance for Claude models aligned with our clients' expectations." Both speed of summarization and the shape in which results are delivered are treated as selection criteria, suggesting that fast summarization and decision-usable output are two separate design questions rather than one.
A&A perspective
AI producing summaries quickly and AI producing summaries in a form usable for management decisions are separate design problems. Pursuing speed alone yields a smoothed synthesis. Pursuing decision-usable output requires a prompt design that preserves disagreement and open conditions. The two are not in conflict — once the output shape is defined, the speed benefits still apply — but the shape has to be defined first. Owners re-examining how existing research documents are handled can start by rewriting the output specification of the prompts they already use. Automate & Augment advises on and implements this kind of prompt design and integration; we are happy to talk through what shape works, starting from the specific research documents and decisions you actually face.
When multiple outside research documents disagree, asking AI for a merged conclusion strips out the grounds a decision-maker needs. Requiring the output to preserve agreement and disagreement, the passages that ground each claim, the differing premises, and the open conditions — as an explicit prompt specification — turns AI from a source of averaged summaries into a support for the comparison work an owner has to do anyway.
Sources & editorial note
Primary pages read for this article. Publication dates below belong to the sources; access dates record our research.
- Quillit eliminates 80% of the time-consuming tasks of qualitative research using Claude
Anthropic · 2025
Accessed 2026-09-19 - Building effective agents
Anthropic · 2024-12-19
Accessed 2026-09-19
AI-assisted editorial production
A&A uses AI for research, writing, translation and editorial checks. Source facts, our analysis and hypothetical examples are labeled separately.
Editorial check: 2026-09-19