There is a credibility gap sitting at the heart of every AI engineering review tool on the market today. Ask most platforms to check a P&ID or a pump datasheet, and the tool returns output. Plenty of output. Findings padded with hedging language, code citations that look plausible but cannot be traced back to an actual clause number, and location references so vague they are useless on a live drawing — "the heat exchanger area may potentially have a concern."
That is not a consultant deliverable. That is a first draft a junior engineer would be embarrassed to send.
The difference between an AI tool that generates findings and one that produces something a Discipline Lead will actually sign off on comes down to a single architectural decision: does the system review its own work before it ships?
SpecSense runs a two-pass critique protocol on every supervised review — the 48-hour pilot, project packs, and team workspace tiers. This article explains exactly what the second pass does, why it exists, and what changes in the report a Principal Engineer receives. Written for procurement teams evaluating AI review tools, and for Owner's Engineering leads who have been burned by tools that produce plausible commentary instead of defensible findings.
A single-pass LLM review runs one prompt: "act as a senior engineer, review this document against the following checklist, return findings." The model does its best. But three failure modes are endemic to this architecture, and every one of them shows up under audit scrutiny.
Large language models will confidently cite API 520 §7.4.2 or IEC 61511 §9.3.1.4 when those clause numbers are incorrect, do not exist, or refer to a different topic entirely. The model has seen enough real citations in training data to produce something that reads correctly to a non-specialist. It has not read the actual standard.
An engineer relying on a fabricated citation to justify a design decision faces real liability exposure. Under audit — whether internal QA, client verification, or third-party technical review — the citation will be checked. If it does not resolve to a real clause covering the finding, the entire deliverable's credibility collapses.
The default LLM register is cautious and passive. "It is observed that the pressure relief arrangement may potentially not be in full conformance with the applicable standard" tells an engineer nothing actionable. It cannot be assigned a severity. It cannot be closed out. It cannot be forwarded to the vendor with a clear expected response.
Every real engineering review needs findings that are declarative, specific, and closable. Passive hedging is worse than silence — it consumes review-cycle attention while carrying zero information density.
"On the compressor suction line" is useless on a P&ID with eight compressor trains across four drawing sheets. A finding without a precise sheet-and-zone reference, or a section-and-paragraph reference for a report, cannot be acted on in the field. The reviewer receiving the AI output has to re-do the location-finding work themselves, which defeats the point of the review.
These failure modes are not edge cases. They appear in every single-pass AI engineering review tested against real deliverables. The critique pass was designed specifically to eliminate them.
Every supervised review on SpecSense runs two sequential model calls against the uploaded document.
The first call runs the document against the full discipline-specific checklist. For a P&ID review, that checklist draws from ISA 5.1 (symbology), ISA 18.2 (alarm management), API RP 14C (surface safety systems for offshore), IEC 61511 (functional safety), and applicable process-specific codes. For a piping isometric review, it draws from ASME B31.3, project piping material specifications, and the discipline's line-list conventions.
Pass 1 is deliberately unconstrained. The model is instructed to find everything that could reasonably be a finding. Over-detection is acceptable at this stage — the output is an internal draft, not the client deliverable.
Typical Pass 1 output for a moderate-complexity P&ID: 28-35 raw findings across all severities.
The second call receives the Pass 1 draft and an adversarial critique instruction set. The critique model's role is explicitly antagonistic: its job is to push back on findings before they leave the office. Five explicit checks run.
Any finding that cannot be linked to a specific code clause, project specification requirement, or uploaded client standard is removed. Findings based on "best practice" alone, without a citable basis, are dropped unless the discipline checklist explicitly includes them. This alone typically removes 5-8 findings from the Pass 1 draft.
Every clause number in Pass 1 is re-checked against a curated clause-verification table. If Pass 1 cited ASME B31.3 §304.5 for a wall-thickness question that actually falls under §304.1, the critique pass either corrects the clause or downgrades the finding to Observation severity if the correct clause cannot be confirmed from the verification table.
This is the highest-leverage check in the entire pipeline. A single hallucinated citation in a delivered report is a chargeable defect. The critique pass exists in large part to close that risk.
Severity is one of the highest-consequence decisions in a review — it determines what gets escalated to the client, what appears in the executive summary, and what drives revised deliverables. The critique pass applies a strict floor: no finding rises above Observation unless a specific, real, verifiable clause requires the missing element.
The convention-before-critique rule sits inside this check. Before a "missing" finding on a drawing is retained at Major or Critical severity, the pass asks: is the drawing using a valid discipline convention that carries this information indirectly? If a PSV lives on the shared header per the client's piping material specification convention, the vessel-level PSV finding is either dropped or downgraded to an interface pointer.
Pass 1 tends toward severity inflation because LLMs bias toward the training data's most-common severity words. The critique pass corrects for this systematically.
Vague recommendations are rewritten to be specific. "Review and confirm" becomes "Confirm suction line diameter per attached hydraulic calc; if NPSHa margin < 0.5 m at rated flow, upsize from 6″ to 8″." A recommendation without a specific instruction, a measurable threshold, and a defined next action is not worth issuing.
Every finding must reference an explicit sheet, zone, tag, or section-and-paragraph number. Findings without a resolvable anchor are either enriched by re-inspecting the source document or downgraded to Observation.
For a typical P&ID review the numeric effect looks something like this:
| Stage | Findings | Major | Minor | Observation |
|---|
| Pass 1 (draft) | 32 | 14 | 12 | 6 |
| Pass 2 (critique) | 22 | 4 | 8 | 10 |
| Net removed | -10 | -10 | -4 | +4 |
Numbers are indicative of a moderate-complexity P&ID and vary by discipline and deliverable type. The pattern is consistent across reviews: fewer total findings, dramatically fewer Major findings, and a rebalance toward calibrated severity that reflects code-actionable risk rather than LLM-default overconfidence.
The client receives a shorter, sharper report with defensible severities. The Discipline Lead does not have to red-pen half the rows before forwarding to the vendor. And the review survives third-party audit because every citation is real and every recommendation is specific.
Prompt engineering means asking one model one question and hoping the output improves through better wording. The two-pass critique is a distinct pipeline stage — a separate model call with a different system prompt, a different role, a different objective, and full read access to the first pass's structured output.
The two objectives conflict inside a single prompt.
- Pass 1 optimises for completeness — do not miss anything a senior engineer would flag
- Pass 2 optimises for defensibility — do not ship anything that cannot be cited, calibrated, and closed
Ask one model to be both exhaustive and defensible and it will pick one, usually the one that generates more text. Separating the passes lets each optimise for its own goal. The pipeline architecture is why the critique pass is a product moat, not a prompt tweak.
If a review report has to survive scrutiny — from a client engineer, an auditor, a regulator, or a project close-out review — the tolerance for hallucinated citations, inflated severity, or vague recommendations is zero.
For teams evaluating AI review tools:
- Ask whether the tool runs a distinct second-pass critique. If the answer is "we use a strong prompt on the latest model," the output is a first draft, regardless of how polished the marketing looks.
- Ask what the critique pass verifies. Clause verification, severity calibration, convention-awareness enforcement, action-oriented recommendation rewriting, and location-anchor validation are pipeline stages, not prompt tweaks. They are the difference between plausible commentary and engineering findings.
- Ask for a sample deliverable from a real supervised review. Read it against your own project's code stack. Check the citations. Check the recommendations. If they fail your review standard, they will fail your client's.
- Ask who signs off. AI tools that ship without a Chartered Engineer's signature on the deliverable are handing you the audit exposure. AI-supervised review — where a CEng signs off on the pipeline's output — is materially different from AI-only review.
Every SpecSense supervised review — the 48-hour pilot, project packs, team workspace — runs through the two-pass pipeline by default. A Chartered Engineer signs off on every supervised review after the critique pass completes. The pipeline produces the draft; the engineer certifies it.
The self-serve marketplace tier at specsense.ai/services uses a single-pass review with an explicit "AI-generated, not for reliance" disclaimer, priced accordingly. It is a tyre-kicking tier for engineers who want to see what the tool can produce on their own deliverable before booking a supervised pilot. It is not a client-facing deliverable and is labelled as such.
That is the point of AI-supervised, not AI-only, review. And the point of running two passes instead of one.
SpecSense is the AI-supervised engineering review platform for oil & gas deliverables — traceable findings, comment registers, and human-supervised escalation in hours. Book a 48-hour supervised pilot or read how the supervised review pipeline works.
