A&A INSIGHTS
Investigating unfamiliar code with AI: separating search evidence from change decisions
For engineers inheriting a system: use the Sourcegraph case to distinguish AI code-search results, impact evidence, uninspected dependencies and test coverage before editing.
日本語で読む
THE STARTING POINT
Finding a plausible edit location does not establish the impact of a change. Using the historical Sourcegraph case as a lens, this article follows a hypothetical CSV-field removal and proposes a revision-bound investigation record covering evidence, consumer expectations and unresolved dependencies.
Turn “Can we remove this field?” into an investigation
Hypothetical example
Imagine inheriting an order system and being asked to remove an obsolete CSV field. AI searches the field name, finds an export function and says that changing it will do the job. This is a hypothetical example. You have not yet established whether the function serves only the download screen, also supplies a nightly export, or produces a file consumed by another system with its own expectations.
A&A perspective
For an engineer investigating an unfamiliar system, A&A recommends requesting the scope of evidence behind a proposed change. Here, investigate where the field is written, which paths reach that code and what consumers require. Treat a coherent explanation separately from demonstrated impact coverage. The purpose is to expose gaps in the change decision, rather than simply increase the number of files read.
Use the Sourcegraph case as a starting point
From the sources
Anthropic’s Sourcegraph case describes codebase-aware questions and answers in Cody that help developers understand interactions between code. It also mentions an experimental effort to bring information such as monitoring metrics into the development environment. This is a historical account involving Claude 3, not a source for current plans or product availability.
A&A perspective
The useful lens is to avoid defining the necessary evidence solely by the code fragment already in view. Access to codebase context does not establish that every path relevant to your request was inspected. The investigation record below is A&A’s application of that lens to implementation work. We are not claiming that Sourcegraph used this format or reduced incidents through it.
Record the revision and the places you could not search
A&A perspective
First pin the repository, commit, behavior under investigation and accessible scope. A brief can request an impact investigation for removing a CSV field, with no edits in this phase, and evidence traceable to files, functions and relevant passages. Retain search terms and scope alongside results. When there are no matches, report that nothing was found within that scope, rather than concluding that no usage exists.
Hypothetical example
A hypothetical record might say: the screen’s download route calls the shared CSV writer; the nightly job configuration names the same destination, but its implementation is in an uninspected repository; and it is unknown whether the consumer reads columns by name or position. A tidy picture that omits the latter two findings is insufficient for this change decision.
A&A perspective
Pair each established relationship with supporting evidence, an unresolved question and the next place to check. Distinguish static reading from observed execution. If production configuration is inaccessible, preserve that limitation. Investigation does not require sending secrets or customer data to AI without authorization; begin with the scope that approved code and synthetic inputs can establish.
Follow the export and its consumers beyond the field-name search
Hypothetical example
In the CSV example, read the function containing the field name, then trace callers and destinations. If the screen and nightly export share a formatter, this is not merely a screen change. If evidence establishes that the nightly job produces a separate fixed format, it can be excluded with that evidence. Do not connect paths just because their names look similar.
Hypothetical example
Next, design a check that passes a CSV produced from synthetic input into the consumer’s import routine. For a name-based consumer, check whether it references the removed field. For a positional consumer, check that later values do not land in different fields. Leaving an empty column, separating formats and changing the consumer together have different consequences. Choose among them after establishing the consumer’s expectations.
A&A perspective
If runtime configuration might select a different path, configuration-aware verification becomes the next task. When that cannot be checked, retain the proposal as conditional on the examined path. Completing an investigation is a different state from being ready to implement and release. The expected output here is an explanation of covered paths and the evidence still needed to proceed, rather than an unconditional list of files to edit.
Separate passing tests from consumer requirements
From the sources
Anthropic’s “Building effective agents” explains that coding agents can improve using automated test feedback, while human review remains important for alignment with broader system requirements.
A&A perspective
In A&A’s proposed approach, first inspect what existing tests actually establish. A test that merely produces a CSV leaves correct consumer import as a separate check. Derive expected results from an agreed input/output example, rather than copying the modified output. Record tests that already failed before the change separately, so they are not prematurely attributed to this modification as either a fix or a regression.
A&A perspective
At handoff, check whether another engineer can return to the cited evidence and explain the path. An uninspected path can remain in the record if ownership of the next investigation is clear. Conversely, a local change with directly visible callers and consumers does not need an elaborate dossier. Scale the record to unseen dependencies, rather than the apparent size of the change.
Judge the investigation by the next decision it enables
A&A perspective
For a trial, choose a small past change whose impact was difficult to establish and compare the investigation records. Examine time spent retracing evidence, overlooked consumer paths and points where implementation proceeded despite unresolved questions. Include engineer review time, tool costs and the effort to keep indexes or configuration information current. A faster answer has not improved handoff if another engineer must repeat the investigation.
A&A perspective
A concise handoff can begin with the revision, established paths and evidence, uninspected dependencies, next checks and conditions under which the implementation proposal holds. When discussing an existing-system change with A&A, the behavior to preserve and unknown consumers are more useful starting points than a general description of the entire codebase. The related article on verifying business results after AI says it is done addresses verification after implementation.
Return the code you found together with what remains outside the investigation. Trace evidence from the examined revision to consumers and state the conditions for implementation. This turns an AI explanation into material an engineer can use for a bounded change decision.
Sources & editorial note
Primary pages read for this article. Publication dates below belong to the sources; access dates record our research.
- Sourcegraph Claude Platform (API) case study
Anthropic · Publication date not stated; historical Claude 3 case
Accessed 2026-09-16 - Building effective agents
Anthropic · 2024-12-19
Accessed 2026-09-16
AI-assisted editorial production
A&A uses AI for research, writing, translation and editorial checks. Source facts, our analysis and hypothetical examples are labeled separately.
Editorial check: 2026-09-16