The W-2 is often described as the simplest income document to extract. It has a fixed IRS-prescribed layout, standardized box numbers, and has been digitally generated by payroll software for decades. Compared to a 1003 with handwritten amendments or a 24-month bank statement PDF, the W-2 should be a solved problem.
It mostly is. But mostly is doing a lot of work in that sentence. When we ran Maestro's extraction pipeline across our internal test set, we found that the confidence-score distribution differs meaningfully across three dominant W-2 layout variants that lenders see in practice. The differences are predictable and systematic once you understand what generates them. Knowing where they come from is what allows you to flag them correctly rather than either silently accepting wrong values or flooding every W-2 with unnecessary review flags.
The three layout variants and what generates them
The IRS Form W-2 specifies which box numbers carry which values: Box 1 is wages, tips, and other compensation; Box 2 is federal income tax withheld; Box 3 and 4 are Social Security wages and withheld amounts; and so on. The IRS provides a reference layout, but does not mandate pixel-level positioning or font choices. Payroll software vendors implement the form to meet the specification while adapting it to their own rendering engines.
The three variants we distinguish are not official IRS categories. They are practical groupings based on what we see in loan files:
Variant 1: Large-payroll-platform digitally generated PDFs. These come from the major payroll platforms serving mid-to-large employers. They tend to use the standard IRS reference layout closely, with consistent box positioning, clean machine-readable fonts, and often machine-readable text layers embedded in the PDF itself. Extraction confidence on these is consistently high across primary income fields (Box 1, Box 3, Box 5) and secondary fields (employer name, EIN, employee SSN).
Variant 2: Small employer or accounting software W-2s. These come from smaller employers using desktop payroll software, regional accounting packages, or CPA-generated forms. The box positioning is more variable. Some software renders the boxes slightly wider, compressing the label-to-value spacing. Some places the employer copy header in a different position than the employee copy. The font choices are more variable, and some versions use proportional fonts rather than monospace for the numeric fields. Confidence on these drops modestly for Box 12 (deferred compensation codes) and the state tax fields, which are the most variable sections across software implementations.
Variant 3: Scanned paper copies. These are either the paper copies that employers send to employees (which borrowers scan and submit) or older paper copies from prior tax years that a borrower kept in a file. Scan quality varies from clean flatbed scans to phone camera photos. The challenge here is not the layout variant but the image quality: skew, shadow, and compression artifacts affect OCR confidence. The layout itself usually matches Variant 1 or 2; the degraded image is what drives the confidence difference.
Where confidence drops and why it is predictable
The fields where extraction confidence drops most across our test set are not the primary income fields. Box 1 extraction is reliable across all three variants except the lowest-quality scans. The confidence drop pattern clusters around three areas:
Box 12 coded entries. Box 12 carries multiple coded compensation items: code D for 401(k) contributions, code W for employer HSA contributions, code DD for employer-sponsored health coverage cost, and others. The codes are small-font two-character identifiers adjacent to dollar values. On Variant 2 forms, the box 12 section can have inconsistent spacing between code and value, and on scanned copies the small font is the first to degrade with image quality. Box 12 is also the field that matters most for calculating qualifying income: code D contributions reduce taxable wages in Box 1, and some lenders add them back for qualifying income calculation under agency guidelines. Getting this wrong has downstream DTI implications.
Multi-state W-2 state tax sections. Borrowers who live in one state and work in another, or who changed states mid-year, receive W-2s with multiple entries in the state section. The state section typically appears at the bottom of the form with smaller boxes for state abbreviation, employer state ID, state wages, and state income tax. With multiple state entries, the line spacing compresses and overlap can occur on some variants. We flag any W-2 with two or more state entries for human verification of the state income fields.
Employer identification fields on scanned copies. Employer name and EIN are used for cross-document validation: if a borrower submits both W-2 and paystub for the same employer, the employer name should match. On low-quality scans, the employer name field in the top-left box of the W-2 is sometimes partially obscured by scanning artifacts. We extract what we can and flag partial reads for confirmation when the extracted employer name does not match the paystub employer name within a reasonable string-distance threshold.
How the benchmark shaped our flagging thresholds
The benchmark exercise was not primarily about producing a number to quote. It was about calibrating where our flagging thresholds should be set relative to field and variant type.
Setting a single global confidence threshold for all W-2 fields produces two failure modes. Set it too low and you pass high-risk fields (like multi-entry Box 12) without flagging them for review, because the raw confidence score from OCR character recognition is high even though the semantic accuracy of the coded entry is questionable. Set it too high and you flag clean Variant 1 Box 1 readings for human review unnecessarily, which adds review workload without adding value.
The threshold we landed on is field-specific and variant-aware. For Box 1 on a Variant 1 or Variant 2 form, the threshold is set to minimize false flags on clean reads while catching the small fraction of cases where box-label confusion occurs. For Box 12, the threshold is more conservative and the flag-and-review posture is the default for any W-2 with code D or code W entries, regardless of extraction confidence, because the downstream implications of a miss there are significant enough to warrant human verification.
We are not claiming the model is correct at some specific rate across these categories. That kind of accuracy claim on a production extraction system requires ongoing measurement against a ground-truth labeled set that updates as new document variants appear in the loan file population. What we can say is that our confidence score distributions across the three variants align with where we would expect extraction difficulty: cleanest on digital-origin Variant 1, modestly lower on Variant 2, most variable on scanned Variant 3.
The field that surprises most loan officers
When we walk loan officers through the W-2 extraction output, the field that generates the most questions is not one of the income boxes. It is Box 14, the "Other" field that employers use to report miscellaneous compensation items: union dues, educational assistance, employer-paid disability premiums, state disability withholding in states that require it, and other items that do not have a designated box.
Box 14 is almost entirely unstructured from an extraction standpoint. The employer decides what label to use and what value to report. One employer's "CA SDI" is another's "State DI" or "Disability" or simply an unlabeled dollar amount. Some employers report multiple items in Box 14 as a comma-separated list. Some leave it blank even when they have reportable items.
Maestro extracts Box 14 contents as raw text with the note that the field requires lender interpretation. For pre-underwriting purposes, the main relevance of Box 14 is whether it contains state disability withholding that should be included in or excluded from qualifying income calculation depending on the lender's guidelines and the applicable state. We do not attempt to parse Box 14 into labeled categories because the label vocabulary is too variable. We surface it as raw text and let the processor or underwriter apply their lender-specific interpretation.
That is an honest representation of the extraction boundary. Knowing where the model ends and human judgment begins is not a limitation to hide. For pre-underwriting document intelligence to actually work in a production origination environment, the handoff from automated extraction to human review needs to be clear and trustworthy. Box 14 is one of the places where that handoff is not optional.