M19 — Redaction and sanitization

State: to do — Depends on: M14, M15, M16 — Active content read, reported and removed per ADR 37; pixels per ADR 42; conformance kept or its loss reported per ADR 17; the action.* and hidden.* families in the core's engine per ADR 36; key types from the Arlington model per ADR 44

Goal

Remove — not cover — what a caller marks or a detector finds: the glyphs, the vector paths and, where the core can decode them, the image pixels under each mark, and the same text wherever else the file keeps it; report everything a document carries beyond what its pages show, and remove it by category; always as a full rewrite, and verified by reading the result again.

A case file leaves the office: to the other side, to the court, to a publisher of decisions, to a client's auditor. Before it does, names are anonymized, account numbers removed, and whatever the producer left behind — an author's account name, a designer's C:\Users\… path, a script that runs on opening, a comment thread, an earlier revision — stripped. The failures this milestone exists to prevent are the famous ones: a black rectangle drawn over text that is still there to select; a "redaction" saved as an incremental update, the removed bytes still in the file; the OCR words of a scan removed while its pixels still show the name; a name gone from the page and still in the bookmark, the document title, the XMP history, a figure's alternative text, the thumbnail made before, the Factur-X XML attached to the invoice, or the font subset whose glyphs spell it; a sanitized file whose scripts are gone but whose open action still launches a program; white text on a white page telling a language model what to conclude.

Scope

In:

the content-stream editing pipeline — read, filter, rewrite — grown from M11's marked-content filter on M15's interpreter, shared by every later content change (M21, M23, M24's tests);
true redaction: marks as /Redact annotations, authored, read from other tools, reviewed, then applied — glyphs removed without moving the rest, paths clipped, shadings and patterns confined, form XObjects edited as copies, annotations and fields under a mark removed or cleared, image pixels replaced where a decoder is at hand, an overlay drawn, the tagged structure kept coherent;
scrubbing beyond the page: the removed text replaced in annotations, form values and XFA data, outline items, metadata (/Info and XMP), alternative, actual and expansion text, file names and descriptions, destination names; scripts that hold it removed; attachments that hold it refused, removed or kept by policy; thumbnails and page-piece data of marked pages removed; font glyphs that no longer draw scrubbed from the embedded programs;
always a full rewrite: earlier revisions, unreferenced objects and trailing bytes never written; signed inputs refused unless the caller insists;
verification after every application: the output reopened, extracted again, searched, and its decoded bytes scanned, the operation failing when anything survived;
search-and-redact: detectors for IBAN, the French NIR, SIREN and SIRET, e-mail addresses, telephone numbers and dates of birth, term lists and regular expressions, and a seam for the caller's own; hits become marks to review;
inspect-then-sanitize: a second built-in profile of M02's engine, disclosure, with the action.* and hidden.* rule families; sanitization by category — JavaScript, automatic actions, launch, submit and import actions, rich media, XFA, embedded files, hidden layers, hidden annotations, invisible text, comments, form data, metadata, image metadata, application data, thumbnails, others' watermarks, usage rights, encryption, signatures, earlier revisions —, with every loss of PDF/A, PDF/UA, Factur-X or a signature reported;
the command-line tool's redact and sanitize verbs.

Out, explicitly:

pixels of JPEG, JPEG 2000, JBIG2 and CCITT images — M22, whose AdCodicem.Pdf.Imaging decoders and lossless encoders plug into the codec set defined here; until then a mark over such an image is refused or the image removed whole, as the caller chooses — never left under an overlay (ADR 42);
OCR of a scan so that search can find text in it — M22; a search that finds nothing on image-only pages says so;
reflowing text to close the gap a removal leaves — never: the library does not lay out received documents, and the limit this leaves is documented below;
rasterizing a page to redact it — not planned; M25 could offer it, at the price of the text layer;
named-entity recognition — not in the library: a caller brings a model through IPdfTextDetector;
executing JavaScript, even to learn what it does — never (ADR 37); scripts are read, reported and removed;
rendering or redacting dynamic XFA — never (ADR 37): refused, and its XFA removable;
recursive sanitization of embedded PDFs — an embedded document is kept or removed; M07 unpacks a portfolio into one volume, which is the route to sanitizing its members;
re-signing a redacted document — M26; converting it to PDF/A — M21.

Design

Where it lives

In the core, with no dependency (invariant 1):

Content/Editing/            PdfContentEditor — M11's filter grown into the pipeline
Redaction/                  AdCodicem.Pdf.Redaction: marks, the plan, apply, the overlay, scrubbing, glyph
                            scrubbing, verification
Redaction/Detectors/        IPdfTextDetector, the built-in detectors and their data tables
Sanitization/               AdCodicem.Pdf.Sanitization: categories, profiles, the plan, apply
Validation/Rules/Disclosure the action.* and hidden.* rules and ValidationProfile.Disclosure

Type	Responsibility
`PdfRedactor`	`Mark(document, marks) → a change set of /Redact annotations`; `Plan(document, options) → PdfRedactionPlan`; `Apply(plan, output, progress, cancellation) → PdfRedactionReport`
`PdfRedactionMark`	A page, quadrilaterals (M11's `PdfQuad`) or rectangles in default user space, the glyph ranges when it comes from a hit, the overlay, a kind and a confidence, an author
`PdfRedactionOptions`	Immutable: overlay (`Box(color)`, `Text(text, style)`, `None`), replacement text, glyph rule, images (`Refuse`, `RemoveImage`), annotations crossing a mark (`Remove`, `Keep`), attachments (`Refuse`, `Remove`, `Keep`), scrubbing of removed text elsewhere and its minimum length, glyph scrubbing, verification (`Fail`, `Report`), signed-document and conformance policies, output version (ADR 40), document identifier, the image codec set
`PdfRedactionPlan`	Per mark: the glyphs, paths, images and annotations it removes and how; every place the text will be scrubbed; refusals; losses; the same text found elsewhere on the pages — inspectable before a byte is written
`PdfRedactionReview`	Accept or reject each mark by its `/NM`
`PdfRedactionReport`	Entries (code, page, object, message), exact counts by code, the verification's outcome; stable JSON
`IPdfTextDetector`, `PdfDetectors`, `PdfDetection`	Detection over M15's folded page text; the built-ins below
`PdfSanitizer`	`Inspect(document) → PdfDisclosureReport`; `Plan(document, categories, options) → PdfSanitizationPlan`; `Apply(plan, output, …) → PdfSanitizationReport`
`PdfSanitizationCategory`, `PdfSanitizationProfile`	Flags, and the named sets `ActiveContent`, `HiddenData`, `All`
`PdfImageCodecs`	The decoders and lossless encoders an operation may use: the core's — uncompressed, Flate, LZW, RunLength, the ASCII filters, predictors — by default; M22's satellite supplies the rest. An immutable value passed in options, never a static registry (invariant 8)

Every operation takes M03's IProgress<PdfProgress> then CancellationToken, with the stages Planning, Rewriting and Verifying, counted in pages. The permissions they demand, in the terms of M16's table, which names the operations of the milestones before it: applying a redaction demands Modify, and Copy besides when its marks come from search or a detector; sanitizing demands Modify; removing encryption stays owner authentication's.

Always a full rewrite, always verified

An incremental update keeps what it replaces. The bytes of every earlier revision stay in the file, readable by anyone who cuts it at an earlier %%EOF — M04 does exactly that. Apply therefore offers no incremental mode: the output is M03's full rewrite, which writes only what the current revision reaches — no earlier revision, no unreferenced object, no free-list remnant, no stale tail after %%EOF, no linearization hint.
The review copy is not the redaction. Mark adds annotations and removes nothing; the marked document may be saved in any mode, opened in Acrobat — which shows the marks and can apply them itself —, reviewed and changed. The report of Mark says it is not redacted (redaction.marks-not-applied), and the tool names its output -review.
The identifier does not link the copy to the original. The output's /ID is derived from its own content (M03) or supplied by the caller; the input's is never carried.
Verification is the last stage of every Apply (below). A redaction that cannot be verified is not reported done.

The content editing pipeline

One pipeline. M11 wrote the core's first content rewriter for layer removal and kept its operator table and state tracking apart from the layer logic for this milestone. PdfContentEditor runs M15's interpreter over a page's streams as one sequence — or over a form XObject, a Type 3 glyph procedure, an annotation appearance — with an IPdfContentEdit that decides for each operation: keep (its source bytes copied as they were, never reformatted), drop, or replace with operators written by M08's builder.
What an edit knows is what the interpreter knows: the graphics and text state, each glyph's box and its source location — the operator's index and the glyph's index in its string, the two integers M15 keeps for this purpose.
Hidden content still changes state (M11's rule): an edit that removes painting keeps cm, gs, colors, q and Q, clipping and text positioning.
Output: a page with an edit gets one new content stream, compressed on the fly; a page without one keeps its streams, copied by the full rewrite as they were. A form XObject edited for one placement is written as a copy, so every other placement keeps the original.
The pipeline is internal; M21 and M23 use it inside the core and M24's tests through the test support.

What a mark removes

Content	Wholly inside a mark	Crossing its boundary	Wholly outside
Glyphs — any render mode, visible or not, in any layer	Removed, their advance kept (below)	Removed whole when the glyph rule selects it, kept whole otherwise: a glyph is never cut in two	Untouched
A filled or stroked path	Its painting removed; its construction kept, as `n`, when it also clips	Its geometry clipped: fills subtracted from the mark, strokes cut against the mark grown by half the line width and the cap or miter extent	Untouched, byte for byte
A clipping path that paints nothing	Replaced by an empty rectangle at the same place, which clips as much	Kept	Untouched
An axial, radial or function shading (`sh`, shading patterns)	Removed	Wrapped in an even-odd clip that excludes the mark — its function is a gradient, not an image	Untouched
A mesh shading (types 4 to 7)	Removed	Removed whole, reported: a mesh can carry an image	Untouched
A tiling pattern fill	Removed	Clipped like a shading; the cell's own content kept and reported (`redaction.pattern-kept`)	Untouched
An image, XObject or inline	Pixels replaced, or the image removed	Pixels under the mark replaced (below)	Untouched
A form XObject	Its content edited in a copy, the mark mapped through its matrix and the CTM	Likewise	Shared, untouched
An annotation other than a widget	Removed, with its popup and replies (M11)	Removed by default; `Keep` leaves it and reports it	Untouched
A widget	Its field's value cleared and its appearance regenerated (M16), or the field removed by option	Likewise	Untouched

Which glyphs. A mark from a hit removes exactly the hit's glyphs, and anything else inside its quads — the second copy of a "fake bold" run M15 merged, a second text layer under an OCR'd image. A mark drawn by a person removes a glyph whose center lies inside it or whose box overlaps it by a fraction slice 2 fixes against PyMuPDF's rule and records.
Visibility is irrelevant. Text hidden by optional content, drawn in render mode 3 under a scan, clipped away or off the crop box is still text in the file: under a mark, it goes.

Text: removing glyphs without moving the rest

In Tj, TJ, ' and ", the string is split around the removed glyphs, and each removed run becomes a number in a TJ array equal to its displacement — computed from the font's widths with the font size, character spacing, word spacing on the single-byte code 32 only, and horizontal scaling in force, as ISO 32000-2 §9.4.4 composes them, in the writing mode's direction. Every kept glyph draws where it did, to a thousandth of a point.
' becomes T* and a TJ; " sets Tw and Tc first, as it did. A text object whose every glyph is removed keeps its state operators. A glyph in a clipping render mode (4 to 7) removed shrinks the clip, which hides more, never less.
What positions still say. Bland, Iyer and Levchenko (Story Beyond the Eye: Glyph Positions Break PDF Text Redaction, PoPETs 2023) showed that the width of removed text, and the fine positioning of the text around it, can narrow down what was removed. The overlay's width says as much; the kept glyphs stay where they were because moving them would change the document. The library does not reflow; PadOverlay widens the overlay to whole ems without touching the displacement, and the limit is documented, not hidden.

Paths, clips and shadings

Fills crossing a mark are subtracted from it by polygon clipping, the fill rule (nonzero or even-odd) kept; curves are flattened only on paths that cross a mark, at a tolerance and within a segment budget, reported (redaction.path-flattened). Whether the clipper is written here (the Vatti algorithm) or Clipper2's C# sources (Boost Software License 1.0) are vendored with their notice is decided by an ADR in slice 2, on robustness measured against degenerate inputs — coincident edges, self-intersections, zero-area rings.
Strokes are cut into sub-paths outside the grown mark; each piece keeps the dash pattern with its phase set to the arc length it started at, so that dashes outside the mark fall where they did; cut ends take the path's cap, which the growth keeps outside the mark.
A path whose every point lies outside the mark is copied as its source bytes — the ordinary case, and the one that keeps a document's rules and backgrounds untouched.

Images and pixels

Decodable images — by the options' PdfImageCodecs — are decoded a row at a time, the mark mapped through the inverse of the image's placement into sample space (a polygon, for a rotated or skewed placement), every sample whose square intersects it replaced, and the rows re-encoded with the same filter and predictor. Nothing but the samples under the mark changes, and the re-encoding is lossless.
The value written is uniform: the overlay's color where the space is device gray, RGB or CMYK, or Indexed over one of them (the nearest palette entry); the components' minimum in any other space, which the overlay covers anyway. What matters is that no sample under the mark keeps its value.
Masks are images too, and carry the shape of what was drawn: an /SMask or a /Mask stream is redacted under the same mark; an image mask's bits are set to paint nothing, through its /Decode; a color-key mask gets a replacement value outside its ranges.
Shared images are edited as copies; if the original is still shown elsewhere, the report names the pages (redaction.image-shown-elsewhere) — the caller marked a region, not the image.
Inline images redacted are written as image XObjects: rewritten inline data can contain an EI a reader takes for the end. Reported.
Undecodable images — DCT, JPX, JBIG2 and CCITT until M22, or any filter the codec set lacks — under Refuse, the default, are listed by the plan with their filter and the milestone that decodes them, and applying throws PdfRedactionException; under RemoveImage the placement is removed whole and reported (redaction.image-removed). Never an overlay over pixels that are still there.
M22's path: the Imaging satellite adds its decoders and lossless encoders to PdfImageCodecs — a JBIG2 or CCITT scan re-encoded losslessly in its own filter, a DCT image re-encoded as Flate by default, or edited in the DCT domain with only the blocks under the mark flattened, which is M22's to decide.

The overlay

Drawn after the page's content, inside q … Q from the page's initial state (M09's balance analysis), filling each mark's quadrilaterals — not their union —: /IC (black by default), /OverlayText with /DA's size and color, /Q alignment and /Repeat, clipped to the quadrilateral and upright in the displayed orientation (M09's placement); an /RO form XObject is drawn as given.
The face is the registry's, embedded and subset (M08); a /DA naming Helvetica from /AcroForm /DR keeps its size and color and gets the registry's face, reported — never a font left unembedded. Color and transparency follow M09's conformance table.
Tagged documents: an overlay without text is an artifact — of type /Pagination with the /Redaction subtype in 2.0 output, a subtype ISO 32000-2 §14.8.2.2.2 adds in PDF 2.0, and of type /Layout in 1.7 output. An overlay with text in a tagged document is real content: a Span with its own MCID, inserted in the structure element that held the first removed glyph, after its last surviving kid before it, with its parent-tree entry — so that a screen reader reads "M. [anonymisé] a déclaré", as M11 inserts annotation references into a received tree.

Tagged structure

A marked-content sequence whose painting is all removed loses its MCID's reference; elements left empty are removed bottom-up; the parent tree is rebuilt by the full rewrite (M11's rules).
A Figure left with no content is removed with its /Alt; /Alt, /ActualText, /E and /T are scrubbed like any text (below). An /ActualText span whose glyphs are all removed goes with them; one partly removed keeps its text with the removed run's text replaced where it is found in it, and is dropped and reported otherwise (redaction.actual-text-dropped) — ActualText exists where the glyphs do not spell the text, so it cannot be rebuilt from them.
Link and Annot elements of removed annotations go with them (M11).

Beyond the page: the same text everywhere else

What is scrubbed: for a hit, the matched string and the detector's canonical forms — an IBAN without spaces, a NIR without its groups, a telephone number in E.164 —; for a drawn mark, each run of removed glyphs as M15 extracts it, when it is at least MinScrubLength characters long (four by default).
Where: every text string the document reaches, found by the types the Arlington model gives each key (ADR 44) — outline titles; annotation /Contents, /T, /Subj and rich text /RC (its XHTML text nodes); field /V, /DV, /Opt, /TU, /TM; structure /Alt, /ActualText, /E, /T; /Info; file specifications' /UF, /F, /Desc; page-label prefixes; optional-content group names; collection items; thread information —; XMP through M14's model, every text property, xmpMM:History included; XFA kept in step through M16 when every occurrence is a bound value, removed otherwise (redaction.xfa-removed); name-tree keys, renamed and the actions that use them rewritten, as M06 renames. A string under a key the model does not type is scanned when it decodes as text, scrubbed and reported (redaction.scrubbed-untyped).
How: the occurrence replaced by the replacement text — the mark's overlay text, or "[…]" —, found with the folding the search used. Field appearances are regenerated after their value changes (M16), since an appearance shows the old value otherwise.
Scripts that contain the text are removed, not edited (redaction.script-removed).
Attachments: text-like ones — XML, JSON, text, HTML, CSV — are scanned as they stream. Refuse, the default, refuses a redaction whose text an attachment holds and names it — a Factur-X invoice's XML carries the buyer's name and IBAN —; Remove removes every attachment; Keep keeps them and reports each hit as a residual the caller accepted. An attachment that cannot be scanned is reported (redaction.attachment-unscanned).
Always: the thumbnails of marked pages and every XMP thumbnail are removed — they picture the page before —, and so is the page-piece data (/PieceInfo) of marked pages, where an editing application may keep its own copy of the page.
Found elsewhere: the removed text found again on other pages is reported with its page and quads (redaction.also-found), a suggestion to review; never removed without a mark.

Fonts: glyphs that no longer draw

A subset's glyphs, and its ToUnicode map, tell which characters a document draws. After removal, a glyph no page draws any more still spells what was there. With ScrubUnusedGlyphs (on by default):

the fonts a removal took glyphs from are written last, from numbers reserved ahead, once every page, form and appearance using them has told which codes it still draws — one pass, as M08 finalizes its subsets;
TrueType: the unused glyphs' glyf entries emptied and loca rewritten, glyph identifiers kept, so no content stream changes; CFF, bare or CID-keyed: their charstrings replaced by endchar, subroutines kept and reported; Type 3: their /CharProcs emptied; ToUnicode entries of unused codes removed; /Widths and CIDSet kept;
Type 1 programs are not rewritten — their charstrings are encrypted in eexec — and reported (redaction.font-not-scrubbed), as are fonts not embedded, which hold no glyphs.

Mark, review, apply

Mark writes /Redact annotations: /QuadPoints and their union as /Rect, /IC, /OverlayText, /DA, /Q, /Repeat, /RO from the options; /Contents naming the kind and confidence — never the matched text —; /NM a deterministic identifier; /T the author the caller supplies; an appearance from M11's generator, which gains the subtype. /Redact is PDF 1.7: marking an older document raises its version, reported (ADR 40).
Others' marks — Acrobat's, PyMuPDF's — are read by M11 and applied alike, /RO, /OverlayText, /Repeat and /DA honored.
Review: PdfRedactionReview accepts or rejects each mark by /NM; rejected marks are removed without being applied; every applied mark's annotation is removed by applying it.
Plan, then apply, as M05 does: the plan says what each mark will remove and what it cannot, before anything is written.

Detectors

Detector	Finds	Checked by	Confidence
`Iban`	Two letters, two check digits and a BBAN, grouped in fours or not	ISO 13616: mod 97 equal to 1, and the country's length from a table compiled from the published IBAN registry, its edition recorded	1 when both hold; below the floor on shape alone
`FrenchNir`	13 digits and a two-digit key, grouped as the carte Vitale prints them or not	Structure — sex, year, month, department with 2A and 2B, commune, order — and the key, 97 minus the number modulo 97 with 2A and 2B replaced by 19 and 18, as python-stdnum's `fr.nir` computes it	1; a number without its key only near a cue ("n° de sécurité sociale", "NIR", "n° SS")
`Siren`	9 digits, grouped 3-3-3 or not; within a SIRET; within a French VAT number `FRkk` + SIREN, whose key is (12 + 3 × (SIREN mod 97)) mod 97	The Luhn check	High near a cue ("SIREN", "RCS"); low alone — one random 9-digit number in ten passes Luhn
`Siret`	14 digits, grouped 3-3-3-5 or not	Luhn, except La Poste's SIRETs (SIREN 356 000 000), whose digits sum to a multiple of 5 instead	As SIREN
`Email`	An RFC 5322 address: a dot-atom local part, a domain with a dot, internationalized labels	Shape	1
`Telephone`	French numbers — 0 and nine digits in pairs, spaces, dots or hyphens —, +33 with or without the (0), and other E.164 numbers	The French plan's leading digits; lengths	High for French forms, lower for the generic one
`DateOfBirth`	A date, numeric or with a French or English month name, within a few words of a cue ("né le", "née le", "né(e) le", "date de naissance", "born on", "DOB")	A valid calendar date	Falling with the distance to the cue
`Terms`	The caller's strings — the parties' names —, folded as asked	—	1
`Regex`	The caller's pattern, `NonBacktracking` by default, M15's timeout otherwise	—	1

Detectors run on M15's folded page text and its offset map, so a hit maps to glyphs and quads, across line breaks and hyphens, through the spaces of U+00A0 and U+202F that French typesetting puts in numbers.
IPdfTextDetector receives that text and map and returns spans with a kind and a confidence: the seam for a named-entity model a caller brings.
Hits at or above MinimumConfidence (0.8 by default) become marks; the others are listed as candidates for review.
Deterministic and culture-free: folding through M15's generated tables, no CultureInfo.

Verification

Apply reopens its output — a seekable output, or a reopening factory the caller supplies; otherwise the report says redaction.unverified and VerificationPolicy.Fail throws — and checks, under the reader's limits:

The page: M15 extracts every glyph, visible or not, whose box meets a mark: there must be none.
The text: every scrubbed string and its canonical forms, searched through the whole document — pages, annotations, fields, outline, metadata, structure, file names —: no hit.
The bytes: every string and every decoded stream scanned for each scrubbed string of four characters or more in UTF-8, UTF-16BE, UTF-16LE, PDFDocEncoding and Windows-1252, streamed with a window the length of the longest encoding: no hit, but in an attachment the caller chose to keep.
The pixels: each redacted image decoded again, its samples under each mark uniform.
The file: one revision (M04), no byte after %%EOF, no /Redact annotation left.

A failure throws PdfRedactionVerificationException with the report; the output written must not be used, and the documentation says so where it shows Apply.

Inspection: the disclosure profile

ValidationProfile.Disclosure — disclosure, version 1 — is a second built-in profile of M02's engine (ADR 36), holding two families. The structural profile does not change, so no manifest findings move; the manifest gains expect.disclosureFindings, written from walks independent of our reader — pikepdf, pdfinfo -js, ExifTool, and Didier Stevens' pdfid over qpdf's uncompressed rewrite.

The rules judge facts others gathered: M15's inventory and glyph visibility, M04's revision history, M11's visibility evaluator, M16's form model. M19 adds one analysis, concealment: a device on M15's interpreter that flags a glyph wholly under an opaque rectangle or image painted after it (CoveredByPath, beside M15's CoveredByImage), a glyph whose fill color lies within a ΔE threshold of the opaque fill beneath it or of the page's white (LowContrast, judged only in device gray, RGB and CMYK), and a glyph drawn smaller than a threshold (Tiny); the thresholds fixed in slice 8 and recorded. The three join M15's visibility flags on PdfGlyph — facts like the others, which extraction may drop by option as it drops text optional content hides —, and are specified here rather than in M15 because the fixtures that calibrate their thresholds are this milestone's.

Rule	Severity	Checks
`action.javascript`	Warning	A script anywhere: the document name tree, an open or additional action, a link, an outline item, a field's format, keystroke, validate or calculate script, an XFA script
`action.automatic`	Information	An action run without the user: an `/OpenAction` other than a destination; catalog `/AA` (close, save, print); page `/AA` (open, close); annotation `/AA` on page open, close, visible, invisible
`action.launch`	Warning	A Launch action — a program run or a file opened
`action.submit-form`	Warning	A SubmitForm action — values, or the whole document, sent to a URL
`action.import-data`	Warning	An ImportData action
`action.remote-document`	Information	GoToR, GoToE to another file, a thread in another file
`action.uri`	Information	A URI action, with its host
`action.rich-media`	Warning	Screen, Movie, Sound, RichMedia or 3D annotations; Rendition, Movie, Sound, GoTo3DView and RichMediaExecute actions
`action.xfa`	Warning	An XFA form: an XFA viewer lays out its template and runs its scripts
`action.named`	Information	A Named action other than NextPage, PrevPage, FirstPage, LastPage
`action.state-change`	Information	Hide, SetOCGState and ResetForm actions — they reveal or conceal
`action.unknown`	Warning	An action type ISO 32000-2 does not define
`action.chain-cycle`	Warning	A `/Next` chain that reaches itself; read around
`hidden.superseded-revision`	Information	Objects an earlier revision defines that the current one redefines or frees (M04's comparison), counted by kind, those that carry text or content named
`hidden.unreferenced-object`	Information	Objects the index holds that nothing reaches
`hidden.trailing-bytes`	Information	Bytes after the last revision (M04)
`hidden.metadata`	Information	`/Info` and XMP properties beyond the title, language and conformance claims, each named
`hidden.local-path`	Warning	A string or XML text that is a local path through a user folder — `C:\Users\…`, `/Users/…`, `/home/…`
`hidden.object-metadata`	Information	Metadata streams on pages, images, fonts, forms
`hidden.image-metadata`	Information	EXIF, XMP, IPTC or comment segments in a DCT stream; metadata boxes in a JP2 stream
`hidden.image-location`	Warning	GPS coordinates in an image's EXIF
`hidden.application-data`	Information	Page-piece dictionaries, `/SpiderInfo`, web-capture name trees
`hidden.private-key`	Information	Keys the Arlington model does not define on the trailer, the catalog, the page tree and pages (ADR 44)
`hidden.thumbnail`	Information	Page thumbnails and XMP thumbnails
`hidden.embedded-file`	Information	An attachment at any level, with its name, type and size
`hidden.embedded-executable`	Warning	An attachment whose name, type or first bytes say it runs
`hidden.layer-content`	Information	Content in a group hidden in the default configuration for viewing
`hidden.annotation-not-shown`	Information	An annotation or widget Hidden, NoView, or outside the crop box
`hidden.invisible-text`	Information	Glyphs in render mode 3 or 7 that are not an OCR layer over a page image
`hidden.ocr-layer`	Information	A page whose text is an invisible layer over its image — kept by default, but it may differ from what the image shows
`hidden.text-outside-page`	Information	Glyphs wholly outside the crop box
`hidden.text-clipped`	Information	Glyphs wholly clipped away
`hidden.text-covered`	Warning	Glyphs under an opaque rectangle or image drawn later that is not a page image over an OCR layer — a failed redaction, or concealment
`hidden.text-low-contrast`	Warning	Glyphs the color of what lies beneath them
`hidden.text-tiny`	Warning	Glyphs drawn below the size threshold
`hidden.comment`	Information	Markup annotations with text, their authors and replies
`hidden.form-data`	Information	Field values and XFA datasets
`hidden.field-value-mismatch`	Warning	A field whose value differs from what its appearance shows
`hidden.redaction-unapplied`	Warning	`/Redact` annotations: whatever they mark is still in the file
`hidden.foreign-mark`	Information	Watermarks, headers and footers another tool added

No rule is an error: nothing here stops the reader reading a page as it was written, which is M02's bar (ADR 45). Rule identifiers are public from the first stable release and disjoint from diagnostic codes, which ValidationRuleIdTests checks.

Sanitization

Category	Removes	Profile
`JavaScript`	Every script and JavaScript action — the name tree, open and additional actions, links, outline items, field scripts (M16's recognized Acrobat formats kept under `KeepRecognizedFormats`) — and XFA scripts with the XFA	ActiveContent
`AutomaticActions`	An `/OpenAction` other than a destination, and every `/AA` entry	ActiveContent
`ExternalActions`	Launch, SubmitForm, ImportData; GoToR and GoToE to other files; threads in other files	ActiveContent
`RichMedia`	Screen, Movie, Sound, RichMedia and 3D annotations, flattened to their appearance by M11 or removed, their actions and media	ActiveContent
`Xfa`	`/XFA` through M16's `Remove`, and `NeedsRendering`	ActiveContent
`EmbeddedFiles`	Attachments at every level, file-attachment annotations, `/AF` entries	HiddenData
`HiddenLayers`	The content of groups hidden in the default configuration (M11's removal); visible groups kept	HiddenData
`HiddenAnnotations`	Annotations and widgets Hidden, NoView, or outside the crop box	HiddenData
`InvisibleText`	Glyphs certainly invisible — render mode 3 or 7 outside an OCR layer, wholly outside the crop box, wholly clipped, wholly covered —, through the pipeline; OCR layers and low-contrast or tiny text only by option	HiddenData
`Comments`	Markup annotations, popups and replies (M11)	HiddenData
`FormData`	Field values reset to their defaults or cleared, appearances regenerated, XFA datasets kept in step (M16)	HiddenData
`Metadata`	`/Info` but the title; XMP but the title, language and conformance claims (`pdfaid`, `pdfuaid`, `pdfx`, `fx`); object-level metadata streams	HiddenData
`ImageMetadata`	APP1 (EXIF, XMP), APP13 (IPTC) and COM segments of DCT streams — APP14 and ICC segments kept —, metadata boxes of JP2 streams: markers removed, entropy-coded data untouched	HiddenData
`ApplicationData`	`/PieceInfo` and `/LastModified`, `/SpiderInfo` and web-capture trees, keys the Arlington model does not define on the trailer, catalog and pages	HiddenData
`Thumbnails`	Page thumbnails, XMP thumbnails	HiddenData
`Links`	URI actions and web links	All
`Watermarks`	Others' watermarks, headers and footers: Watermark annotations, `/Artifact` content of subtype `Watermark` we did not write, and Acrobat's marks, recognized by the page-piece markers it writes on their form XObjects and the groups it puts them in — recorded from the corpus files in slice 10	All
`UsageRights`	`/Perms /UR3` and `/UR`	All
`Encryption`	The output written unencrypted — owner authentication, or `Ignore` declared	All
`Signatures`	Signature values, `/DSS`, `/Perms /DocMDP`; visible appearances flattened as they looked (M16's field flattening)	All
—	Earlier revisions, unreferenced objects, bytes after `%%EOF`: never written by a full rewrite, and listed	Always

Plan, then apply, each removal attached to the finding that justifies it: a removal nobody asked for is a corruption (ADR 22's rule for repair). Usage rights, encryption and signatures, which no disclosure rule reports, are planned from M04's and M16's models.
What keeps working: the structure tree, which no category removes; links unless Links is chosen; the recognized formats of a form under KeepRecognizedFormats; PDF/A and PDF/UA claims unless a removal breaks one.
What is lost is said: removing the attachments of a Factur-X invoice removes its fx claim and /AF entries (sanitize.facturx-removed, ConformanceLoss); removing a dynamic form's XFA leaves its placeholder page (xfa.placeholder-left, M16's); removing scripts from a form leaves its values as they were and its calculations stale; flattening a signature's appearance keeps what it looked like and nothing it proved.

Signed, protected and conforming inputs

Input	By default	When the caller insists
Approval-signed	Refused: M03's `PdfSignatureInvalidationException` — a full rewrite voids every signature	`AllowInvalidatingSignatures`: signature values removed, visible appearances flattened as they looked or removed where a mark crosses them; `write.signature-invalidated` and `redaction.signature-removed`
Certified	Refused, naming the permission	The same
Usage rights	Removed and reported, M16's `RemoveWhenBroken`	—
Encrypted, `Modify` granted or owner-authenticated	Applied; output unencrypted unless M16's options encrypt it	—
`Modify` withheld	Refused, `PdfPermissionException`	`PdfPermissionPolicy.Ignore` declared: applied, `permission.not-respected`
`Copy` withheld	Marks from search or detectors refused: extraction is `Copy`'s	The same declaration
A PDF/A claim	Kept: the overlay and replacements respect it (M09's table); an edit that cannot is refused, typed	`RemoveClaim`: written, `redaction.conformance-claim-removed`
A PDF/UA claim	Kept: structure coherent, overlay text tagged	The same
Factur-X	A redaction whose text the XML holds refused, naming the attachment	Attachments `Remove`: the XML, `/AF` and `fx` removed, `redaction.facturx-removed`
Dynamic XFA	Redaction refused: the page shown is a placeholder (ADR 37)	Sanitizing `Xfa` leaves the placeholder, reported
A marked page whose content cannot be decoded	Refused	—
An undecodable image under a mark	Refused, naming the image, its filter and M22	`RemoveImage`

Bounds (invariant 12, ADR 34)

Bound	Kind	Why
Marks per page	None: indexed in a grid over the page box, a glyph tested against the cells it touches	A file may carry thousands of `/Redact` annotations; the cost stays linear
Curve subdivision where a stroke crosses a mark: depth 24	Internal constant	A cubic crosses a rectangle's edges at most twelve times, so the pieces stay few; past depth 24 a piece is under a millionth of its curve, and is removed
Segments of a flattened curve	Internal constant, recorded in slice 2	Only fidelity near the mark is at stake: the subtraction is exact on the flattened polygon
Operations, glyphs and state depth on a page	M15's guards	The pipeline runs the interpreter
Samples of an image redacted	Checked arithmetic on width × components × bits, under `MaxDecodedStreamLength`; rows streamed	Dimensions come from the file
Attachments and streams scanned during verification	Streamed under `MaxDecodedStreamLength`; a stream cut at a limit makes verification fail, never pass	A limit is the reader's, not the file's
Glyph usage per font	A bit set over the font's code space, at most 65,536 codes	Format limit
A regular expression's match time	M15's `MatchTimeout` when backtracking	The caller's pattern on hostile text

Report codes

redaction.* and sanitize.*, public and stable from the first stable release, disjoint from reader diagnostics, validation rules and every other report family. sanitize.*-removed has one code per category — sanitize.javascript-removed, sanitize.embedded-file-removed, sanitize.metadata-removed and so on —, each entry naming the object and the finding that justified it.

Code	Severity	Meaning
`redaction.marks-not-applied`	Information	`Mark` added marks; nothing is removed yet
`redaction.image-removed`	Warning	An undecodable image under a mark removed whole, as the caller chose
`redaction.image-shown-elsewhere`	Warning	An image redacted here is shown unredacted on the pages named
`redaction.path-flattened`	Information	Curves crossing a mark were flattened
`redaction.pattern-kept`	Information	A tiling pattern's cell content kept, its fill clipped
`redaction.inline-image-externalized`	Information	A redacted inline image written as an XObject
`redaction.actual-text-dropped`	Warning	An `ActualText` span partly removed and not rebuildable
`redaction.scrubbed`, `redaction.scrubbed-untyped`	Information	Text replaced outside the page, typed by the model or not
`redaction.script-removed`, `redaction.xfa-removed`	Warning	A script or the XFA held the text and was removed
`redaction.attachment-unscanned`	Warning	An attachment that could not be scanned for the text
`redaction.residual-accepted`	Warning	The text remains in an attachment the caller chose to keep
`redaction.also-found`	Warning	The removed text appears on other pages, unmarked
`redaction.font-not-scrubbed`	Warning	A Type 1 program kept its unused glyphs
`redaction.pages-without-text`	Warning	A search or detector ran on pages that have no text (M15's page kinds)
`redaction.signature-removed`	Warning	A signature value removed after the caller insisted
`redaction.conformance-claim-removed`, `redaction.facturx-removed`, `sanitize.conformance-claim-removed`, `sanitize.facturx-removed`	ConformanceLoss	A claim the operation broke, removed as the caller's policy allowed
`redaction.unverified`	Warning	The output could not be reopened for verification
`sanitize.<category>-removed`	Information	One per category, as above
`sanitize.revisions-removed`, `sanitize.unreferenced-removed`	Information	What the full rewrite did not write

The command-line tool

redact    IN -o OUT ( --mark PAGE:X0,Y0,X1,Y1 … | --search TEXT … | --terms FILE
                    | --detect iban,nir,siren,siret,email,phone,dob | --regex PATTERN | --apply-marks )
          [--review OUT-review.pdf] [--overlay-text T] [--color C] [--min-confidence N]
          [--images refuse|remove] [--attachments refuse|remove|keep] [--allow-invalidating-signatures]
sanitize  IN [-o OUT] [--inspect] [--profile active-content|hidden-data|all] [--categories …] [--keep …]

--review writes the marked copy and stops; --apply-marks applies the marks a file already carries, ours or Acrobat's. sanitize --inspect prints the disclosure report and writes nothing. Both take --json and M06's exit codes; a refusal by policy is exit code 4, a failed verification exit code 1.

Slices

Each slice ends on a green commit, with the codes and rules it introduces documented and its benchmark, if it has one, recorded in docs/status.md.

The editing pipeline. Delivers PdfContentEditor from M11's filter, keep-drop-replace with source bytes kept, glyph removal with exact displacement, forms edited as copies, one new stream per edited page, and a test-only edit that removes chosen glyphs. Proved by FsCheck — for any page M08's builder writes and any subset of its glyphs, removing the subset leaves every other glyph at its position within 10⁻³ pt and the subset absent —; unit tests for ', ", Tz, Tw on code 32 in one- and two-byte fonts, vertical writing, a zero and a negative font size; integration: on every committed page MuPDF reads, mutool draw -F trace gives the kept glyphs the positions it gave before, pdftotext the original text minus the removed, qpdf --check no complaint. Leaves marks.
Marks, glyphs, paths and shadings. Delivers PdfRedactionMark, the glyph rule, the region index, fill subtraction, stroke cutting with dash phase and caps, clip replacement, shadings and patterns, the ADR on the clipper. Proved by FsCheck — no clipped path enters its mark, and everything outside every mark is unchanged —; unit tests on degenerate polygons; integration: MuPDF renders each marked page with pixels outside the marks equal to the original's within the visual threshold and inside them the overlay's color; pdftotext -x -y -W -H over each mark returns nothing. Leaves annotations.
/Redact marks, the overlay, annotations and structure. Delivers Mark, M11's appearance for the subtype, reading others' marks, review, apply as a full rewrite, the overlay in every form, annotations and widgets under marks, the structure rules, the signed and certified policy. Proved by PyMuPDF's apply_redactions on our marks removing the same glyphs we remove, each difference explained; our apply of marks PyMuPDF wrote; pdfinfo -struct-text reading overlay text in structure order; veraPDF's PDF/UA-1 profile on the three committed claims after redaction. Leaves pixels.
Pixels. Delivers PdfImageCodecs with the core's filters, row-streamed redaction at every bit depth, Indexed, /Decode, image masks, soft and explicit masks, color keys, shared images as copies, inline images externalized, Refuse and RemoveImage. Proved by unit tests per case; integration: Pillow on pdfimages -png before and after — every sample under each mark uniform, every other sample identical —; the undecodable scans refused, each image named. Leaves scrubbing.
Beyond the page. Delivers scrubbing by the Arlington model's types, XMP, XFA, name-tree keys, rich text, scripts, the attachment policies, thumbnails and page-piece data of marked pages, the found-elsewhere report, and glyph scrubbing with fonts written last. Proved by unit tests per place; integration: a pikepdf walk finds no scrubbed string in any string or decoded stream; ExifTool finds it in no metadata; fontTools shows the unused glyphs empty and every other outline identical; pdffonts lists the same fonts. Leaves verification.
Verification. Delivers the five checks, PdfRedactionVerificationException, redaction.unverified. Proved by unit tests that plant a survivor in each place — a glyph, a string, a stream, a sample, an earlier revision — and see each caught; integration: the acceptance's independent checks agreeing with ours on every redacted document. Leaves detectors.
Detectors. Delivers IPdfTextDetector, the built-ins and their tables, hits to marks, the confidence floor. Proved by FsCheck — ten thousand generated values per detector, valid and invalid, on whose validity python-stdnum and phonenumbers, in a container, agree with us; any value placed by M08's builder across a line break is found with quads covering exactly its glyphs —; integration: the hits on the invoices and the fixtures found where PyMuPDF's search_for finds their text. Adds the PII fixtures the corpus lacks (below) and expect.detections to the manifest and its schema. Leaves inspection.
Inspection. Delivers ValidationProfile.Disclosure, the action.* and hidden.* rules, the concealment analysis with its thresholds, PdfSanitizer.Inspect; adds expect.disclosureFindings to the manifest and its schema, and the concealed-text and wrong-redaction fixtures the corpus lacks (below). Proved by a triggering, a silent and a legal-but-unusual document per rule; integration: every committed document's findings equal to what the independent walks record. Leaves sanitization.
Sanitizing active content. Delivers the plan and apply for JavaScript, AutomaticActions, ExternalActions, RichMedia, Xfa, the ActiveContent profile, KeepRecognizedFormats. Proved by pdfinfo -js reporting nothing, pdfid over qpdf's uncompressed rewrite counting no /JS, /JavaScript, /AA, /OpenAction other than a destination, /Launch, /SubmitForm, /RichMedia or /XFA, a pikepdf walk agreeing, veraPDF's feature report listing none; the forms still fillable through M16. Adds the action fixtures the corpus lacks (below). Leaves hidden data.
Sanitizing hidden data. Delivers the other categories — attachments, layers, hidden annotations, invisible text, comments, form data, metadata, image metadata, application data, thumbnails, links, watermarks, usage rights, encryption, signatures —, the HiddenData and All profiles. Proved by ExifTool on the document and on every image pdfimages -all extracts; the DCT streams' entropy-coded bytes identical before and after, and Pillow decoding the same pixels; pdfdetach -list empty; qpdf and M04 showing one revision; the OCR layers still extracted by pdftotext; veraPDF upholding every claim a removal did not break. Leaves the whole.
Policies across both. Delivers the matrix above end to end — permissions, encryption on output, conformance, Factur-X, dynamic XFA —, and the permission demands above. Proved by the acceptance rows for signed, encrypted, PDF/A, PDF/UA and Factur-X documents. Leaves the tool.
The tool, budgets and the site. Delivers redact and sanitize, RedactionBenchmarks and SanitizationBenchmarks with MemoryDiagnoser, the memory checkpoints, the documentation. Proved by CorpusToolTests, the budget rows and a green Remote corpus run recorded in status.md.

Tests required

Unit — tests/AdCodicem.Pdf.Tests:

The pipeline: every text-showing operator; Tc, Tw, Tz, TL, Ts, Tr in force; one- and two-byte codes; vertical writing; a text object that spans streams; a form drawn on two pages, edited on one.
Geometry: FsCheck over random paths, marks and fill rules — no point of a kept fill inside a mark, the region outside the marks painted as before —; strokes with dashes, caps and joins; a clip-only path; each shading type.
Images: every bit depth, color space and mask kind; a rotated and a skewed placement; a shared image; an image whose declared size its data does not fill.
Overlay and structure: every overlay form; /RO; a /DA naming a font absent from the registry; a mark over a heading, a table cell, a figure, an ActualText span, a link.
Scrubbing: each place in the list; a term split across two strings of an outline item; UTF-16LE and UTF-8 strings; a name-tree key used by a link.
Fonts: TrueType, CFF, CID-keyed CFF and Type 3 scrubbed; Type 1 reported; a font shared by a kept and a marked page.
Verification: a survivor planted in each place, caught.
Detectors: each check digit, each exception — La Poste, 2A and 2B —, each grouping, the cues and their distance, a number split by a line break or a narrow no-break space.
Disclosure rules: a triggering, a silent and a legal-but-unusual document per rule; the concealment thresholds.
Sanitization: each category alone and with the others; every loss reported; nothing removed without a finding.
Hostile: content the interpreter cannot finish, a /Redact with ten thousand quads, a mark of zero area, an annotation array of absurd length, an XMP packet a megabyte deep, a JPEG whose segments lie about their lengths — each ends in a report or a typed exception within the time and allocation budgets.
Determinism: FsCheck over random marks and categories on corpus documents — two applications, identical bytes.

Integration — tests/AdCodicem.Pdf.IntegrationTests, every referee in a container (ADR 27):

poppler — pdftotext whole and cropped to each mark; pdfinfo -js and -struct-text; pdfimages -list, -png and -all; pdfdetach; pdffonts;
PyMuPDF — words, search_for with quads, text in a rectangle, and apply_redactions as a second implementation of applying /Redact marks;
MuPDF — mutool draw renders before and after, and -F trace for glyph positions;
qpdf — --check; --qdf --object-streams=disable for the byte scans; --json and --show-xref for revisions;
pikepdf — walks for strings, actions, annotations, structure and the parent tree;
pdfid (Didier Stevens) — keyword counts over qpdf's uncompressed rewrite;
veraPDF — PDF/A and PDF/UA-1 verdicts before and after; its feature report;
pyHanko — signatures after an insisted redaction;
ExifTool — document metadata, and the metadata of every extracted image;
fontTools — glyph outlines before and after scrubbing;
Pillow — sample comparisons;
python-stdnum and phonenumbers — the detectors' oracles.

Acceptance conditions

"Every committed document" means every document under documents/ and vendor/ the reader opens, the encrypted ones with their recorded passwords through M16.

Documents	Behavior	Verified by
The invoice through five producers — `documents/invoice/chromium-invoice-fr.pdf`, `libreoffice-invoice-fr.pdf`, `word-invoice-fr.pdf` (tagged), `word-print-driver-invoice-fr.pdf`, `pdf24-invoice-fr.pdf` —, `documents/contract/chromium-contract-fr.pdf`, searched for "Noailles" and redacted; `documents/report/chromium-report-fr.pdf` (whose table of contents links to the annex) and `libreoffice-report-fr.pdf` (whose outline item "8. Annexe — note de lecture" carries it too), redacted for "Annexe"	The term absent from `pdftotext`'s and PyMuPDF's text, from the outline, from the XMP and `/Info`, and from the bytes of `qpdf --qdf --object-streams=disable`'s rewrite in UTF-8, UTF-16 and Latin-1; every other `textContains` string still found; MuPDF's rendering unchanged outside the marks; one revision; every link outside a mark still resolving in pdf.js, the table-of-contents link under one removed with it and reported	`CorpusRedactionTests.Redacted_text_is_gone_from_every_extraction_and_every_byte` (new), `PopplerRedactionRefereeTests` (new), `PyMuPdfRedactionRefereeTests` (new)
The same marks applied by PyMuPDF's `apply_redactions`; marks written by PyMuPDF applied by us	The same glyphs removed by both, within the glyph rule's tolerance, each difference listed with its reason	`PyMuPdfRedactionRefereeTests.Our_marks_and_pymupdf_remove_the_same_glyphs` (new)
`vendor/us-federal/illustrator-irs-pub1-english.pdf` (an earlier revision draws text the current one does not), `vendor/uk-ogl/pdfmaker25-home-office-eia.pdf` (two metadata-only updates), `vendor/opf-format-corpus/pdfmaker5-distiller5-va-select-agents.pdf` (an `/Info` superseded in an update); remote `remote/govuk/print-to-pdf-border-force-poster-updated.pdf` (`/Info` replaced) and `remote/pdfjs/aspose-cad-stale-tail-issue9418.pdf` (a stale tail); a document whose earlier revision holds a term its current revision also shows — not in the corpus (below)	After redaction or any sanitization: one revision, for M04 and qpdf; every string and every text that only earlier revisions held — extracted from each input revision through M04 — absent from the output's bytes; the stale tail gone	`CorpusRedactionTests.Nothing_survives_from_earlier_revisions` (new)
`vendor/fr-licence-ouverte/fop-dictao-dila-signed-joafe-notice.pdf` redacted for "L'ATELIER DE LAURE"; `vendor/pyhanko/acrobat-reader-signed-twice.pdf` for "Testerino"; `vendor/us-federal/itext-govinfo-us-code-certified.pdf`; `vendor/lu-legilux/antenna-house-legilux-memorial-pades-lta.pdf`	Refused, typed, naming each signature and permission; insisted: one revision, pyHanko and `pdfsig` finding no signature, the visible appearances flattened as they looked, `write.signature-invalidated` and `redaction.signature-removed` for each; the term absent as in the first row	`CorpusRedactionTests.Signed_documents_are_refused_unless_the_caller_insists` (new), `PyHankoRedactionRefereeTests` (new)
The PDF/UA-1 claims veraPDF upholds — `vendor/pdf-association/pdflib-pps-kraxi-pdfa2a-pdfua1-invoice.pdf` redacted for "Kraxi Systems", `indesign13-pdfua1-german-book-chapter.pdf` for "Grundüberlegungen", `indesign15-pdfua1-form.pdf` for "PDF/UA form example" —; remote `remote/pdf-association/indesign-cs6-pdfua1-brochure.pdf`	veraPDF's PDF/UA-1 profile reports no failure it did not report before, and the Kraxi invoice's PDF/A-2a claim is still upheld; the overlay text read in its sentence by `pdfinfo -struct-text`; the parent tree still the inverse of the marked content (M13's pikepdf script); the title scrubbed and still present	`VeraPdfRedactionRefereeTests.Redacted_pdfua_documents_stay_valid` (new)
Every committed document whose PDF/A claim veraPDF upholds — the list M09's conformance row names —, each redacted for one of its `textContains` strings that no attachment carries and sanitized with the `HiddenData` profile	veraPDF upholds the same claim; the claims, title and language kept in the XMP; the Factur-X invoices' `fx` claims removed with their XML and reported	`VeraPdfRedactionRefereeTests.Pdfa_claims_survive_redaction_and_sanitization` (new)
Factur-X: `vendor/zugferd/mustang-zugferd2-en16931-invoice.pdf` redacted for "Bei Spiel GmbH" and by the IBAN detector, `vendor/docentric/aspose-d365-facturx-extended-invoice.pdf` for "A.Datum Corporation" and by the IBAN detector, `vendor/zugferd/itext-pdfbox-weclapp-facturx-en16931-invoice.pdf` for "Beispiel AG"; remote `remote/zugferd-corpus/fnfe-factur-x-basicwl-fr.pdf` for a name its XML carries	Refused by default, naming the attached XML — the Mustang XML holds its IBAN grouped with spaces, the D365 XML without, and both are found by their canonical form; with attachments `Remove`: `pdfdetach -list` empty, the `fx` properties and `/AF` gone, veraPDF upholding the PDF/A-3 claim, `redaction.facturx-removed` reported	`CorpusRedactionTests.A_term_in_an_attachment_refuses_the_redaction` (new)
Pixels the core decodes: `vendor/fr-licence-ouverte/word365-dila-text-drawn-as-images.pdf` (text drawn as soft-masked images, a mark over a word of it), `vendor/opf-format-corpus/openoffice32-writer-png-image.pdf` and `openoffice32-writer-tiff-image.pdf` (Flate), `vendor/uk-ogl/pdfmaker10-word-dh-care-bill-factsheet.pdf` (an image with a soft mask), `vendor/pdf-association/handwritten-inline-image-abbreviations.pdf` (inline Flate with a predictor and a `/Decode` array), `vendor/us-federal/distiller3-irs-ss4-1995-form.pdf` (LZW)	Under each mark every sample of the image and of its mask uniform, every other sample identical, by Pillow on `pdfimages -png` before and after; the inline image now an XObject; MuPDF's rendering unchanged outside the marks	`CorpusImageRedactionTests.Pixels_under_a_mark_are_replaced_and_no_others` (new)
Scans the core cannot decode yet: `vendor/us-federal/xerox-workcentre-treasury-imf-report-scan.pdf` (JBIG2), `acrobat3-import-irs-1040-1988-scan.pdf` (CCITT G4), `documents/scan/reportlab-scanned-receipt.pdf` (DCT), `vendor/opf-format-corpus/imagemagick-false-pdfa1b-jpx.pdf` (JPX), and the OCR'd scans `finereader8-frb-sr0115-examiner-guidance.pdf` (CCITT under its text) and `xerox-workcentre-5335-ocr-hud-fonsi-linearized.pdf` (JBIG2 under its text)	A search on the OCR'd ones finds their words; applying is refused, naming each image, its filter and M22; under `RemoveImage` the placements go, the OCR glyphs under the marks go, and both are reported; recorded `unsupported` for pixel redaction until M22	`CorpusImageRedactionTests.Undecodable_images_refuse_until_m22` (new)
The documents above, and the PII fixtures — not in the corpus (below) —, with terms in outline items, annotations, field values, alternative text, `/Info`, XMP and an attached XML	Each place scrubbed; the scrubbed strings found by no pikepdf walk over strings and decoded streams, and by no ExifTool field; the field appearances regenerated; the fonts' unused glyphs empty in fontTools and every other outline identical	`CorpusRedactionTests.Everything_else_that_carried_the_text_is_scrubbed` (new), `CorpusRedactionTests.Unused_glyphs_are_scrubbed_from_fonts` (new)
The five invoices and the Factur-X invoices above; the PII fixtures — not in the corpus (below)	SIREN 123 456 824 found beside "SIREN" and "RCS Paris" wherever an invoice shows them, and inside the VAT number FR40123456824, whose key it checks; the client's SIREN 552 041 319 and the IBAN FR76 3000 4008 2800 0123 4567 890, whose check digits fail, listed below the floor and not marked; the Mustang and D365 IBANs found; every planted value of the fixtures found at the quads PyMuPDF's `search_for` gives its text, and nothing else	`CorpusDetectorTests.Detectors_find_what_they_should_where_pymupdf_finds_it` (new), `CorpusDetectorTests.Invalid_check_digits_stay_below_the_floor` (new)
Generated values, ten thousand per detector	python-stdnum's `iban`, `fr.nir`, `fr.siren` and `fr.siret`, and phonenumbers, agree with our verdict on every value	`DetectorOracleTests.Detectors_agree_with_python_stdnum_and_phonenumbers` (new)
Every committed document and every remote one	The disclosure profile gives exactly the `action.` and `hidden.` findings `expect.disclosureFindings` records from the independent walks — among them the Launch action of `vendor/opf-format-corpus/pdfmaker9-word-distiller-external-link.pdf`; the local user paths in the XFA packets of `livecycle-irs-1040-2022-xfa-ur3.pdf`, `livecycle-dod-dd293-aes128-xfa.pdf` and `livecycle-es9-cerfa-14880-xfa-form.pdf`, in the XMP of `designer-distiller23-uscis-i9-javascript-form.pdf`, and in the Illustrator private data (`AIPDFPrivateData`) under the page-piece dictionaries of `illustrator-irs-pub1-english.pdf`; the hidden widgets of `omniform-usda-rd1924-5-hidden-widgets.pdf`	`CorpusValidationTests.Disclosure_findings_match_the_manifest` (new)
Active content: `vendor/opf-format-corpus/make-pdf-javascript-openaction.pdf`, `vendor/pdf-association/handwritten-compacted-syntax.pdf` (a catalog `/AA` script), `vendor/us-federal/omniform-usda-rd1924-5-hidden-widgets.pdf`, `designer-distiller23-uscis-i9-javascript-form.pdf`, `vendor/fr-licence-ouverte/pdfmaker-acrobat-cerfa-12156-form.pdf`, `libreoffice-cerfa-13983-form.pdf`, `livecycle-es9-cerfa-14880-xfa-form.pdf`, `vendor/uk-ogl/pagemaker-distiller5-hmrc-iht205-form.pdf` (RC4-128), `vendor/us-federal/livecycle-irs-1040-2022-xfa-ur3.pdf`, `livecycle-uscis-ar11-xfa-form.pdf` (AES-128), `vendor/pikepdf/livecycle-dod-dd293-aes128-xfa.pdf`, `vendor/opf-format-corpus/pdfmaker9-word-distiller-embedded-quattro-pro-spreadsheet.pdf`, `pdfmaker9-word-distiller-embedded-quicktime.pdf` (Screen and Rendition), `pdfmaker9-word-distiller-external-link.pdf` (Launch), `vendor/verapdf/pdfa2b-actions-fail.pdf`; remote `remote/opf-format-corpus/acrobat9-portfolio-signed-3d.pdf` (3D, a Flash navigator), `pdfmaker9-word-distiller-embedded-avi.pdf`, the dynamic XFA forms of `remote/canada/`	Sanitized with `ActiveContent`: `pdfinfo -js` reports nothing; pdfid over qpdf's uncompressed rewrite counts no `/JS`, `/JavaScript`, `/AA`, `/Launch`, `/RichMedia`, `/XFA` and no `/OpenAction` but a destination; a pikepdf walk and veraPDF's feature report agree; the report lists every removal, one for each inspection finding; the forms still filled by M16; the rule veraPDF reported for the fixture's additional action passes; the dynamic forms' placeholders reported	`CorpusSanitizationTests.No_active_content_survives` (new), `PdfidSanitizationRefereeTests` (new)
Hidden data: account names in `/Author` (`vendor/us-federal/print-to-pdf-excel-dod-fcf-rates-2021.pdf`, `vendor/uk-ogl/print-to-pdf-word-cspl-agenda.pdf`); page-piece data in the PDFMaker, InDesign and Illustrator files (`pdfmaker7-powerpoint-va-cancer-database-course.pdf`, `indesign-irs-pub1-arabic.pdf`, `illustrator-irs-pub1-english.pdf`); thumbnails (`indesign13-pdfua1-german-book-chapter.pdf`'s 21, `pdfwriter4-usda-dry-whey-standard-2000.pdf`, `indesign-acrobat-hmcts-n208-form.pdf`, `pagemaker-distiller5-hmrc-iht205-form.pdf`) and XMP thumbnails (`illustrator-irs-pub1-english.pdf`, `gnuaccounting-mustang10-zugferd-rc-invoice.pdf`); comments (`reader10-openoffice32-annotated-object-streams.pdf`); layers (`handwritten-utf16le-strings.pdf`, groups off; `pdfmaker21-ozev-sample-invoice.pdf`, Acrobat's watermark); remote `remote/ocrmypdf/photoshop-cc2015-pdfx3-cmyk.pdf` (text layers listed in its XMP), `remote/ocrmypdf/pdfpen-imprints-split-content.pdf` (private keys)	Sanitized with `HiddenData` (and `Watermarks` for the invoice): ExifTool finds no removed property; pikepdf finds no `/PieceInfo`, `/Thumb`, removed annotation or hidden group's content; the chapter still valid under veraPDF's PDF/UA-1 profile; the watermark "Sample" no longer extracted; the report lists each removal	`CorpusSanitizationTests.Hidden_data_is_removed_and_listed` (new)
The DCT images carrying EXIF in `vendor/us-federal/livecycle-uscis-ar11-xfa-form.pdf`, and in three signed documents sanitized under `AllowInvalidatingSignatures` — `vendor/lu-legilux/antenna-house-legilux-memorial-pades-lta.pdf`, `fop22-legilux-memorial-seal-renewed-timestamps.pdf`, `vendor/fr-licence-ouverte/fop-dictao-dila-signed-joafe-notice.pdf`	ExifTool finds no EXIF in any image `pdfimages -all` extracts; each stream's entropy-coded bytes identical and Pillow's decoded pixels equal	`CorpusSanitizationTests.Image_metadata_is_stripped_without_touching_pixels` (new)
OCR layers: `vendor/us-federal/hp-mfp-acrobat-ocr-nih-report.pdf`, the two Xerox files, `finereader8-frb-sr0115-examiner-guidance.pdf`	`InvisibleText` removes nothing from them by default, `hidden.ocr-layer` reported, `pdftotext` still finds every `textContains` string; with `IncludeOcrLayers` the text is gone and the pages `ImageOnly`	`CorpusSanitizationTests.Ocr_layers_survive_invisible_text_removal` (new)
White-on-white, tiny, off-page, clipped and covered text, and a black box over live text — not in the corpus (below)	Each flagged by its `hidden.*` rule; `InvisibleText` removes the certain cases and reports the others; the covered text gone after sanitizing, not merely under its box	`CorpusSanitizationTests.Concealed_text_is_found_and_removed` (new)
Permissions: `vendor/us-federal/pdfwriter3-copyright-office-dmca-summary-1998-rc4-40.pdf` (P = −12, modification withheld), `vendor/pikepdf/livecycle-dod-dd293-aes128-xfa.pdf` (P = −1052, modification and copying withheld), `vendor/opf-format-corpus/openoffice32-writer-rc4-128-no-copy.pdf` and `pdfmaker9-word-distiller-aes128-no-copy.pdf` (copying withheld); `documents/secured/qpdf-invoice-aes256.pdf`	Refused with `PdfPermissionException` naming the permission — search-based marks where copying is withheld; with `Ignore` declared, applied and `permission.not-respected` reported; the AES-256 invoice redacted, re-encrypted with the same password on request, opened by qpdf with it	`CorpusRedactionTests.Permissions_are_respected` (new)
`documents/stress/reportlab-journal-1000-pages.pdf`: "en attente" redacted — sixteen lines a page, sixteen thousand marks — and "Journal des operations" scrubbed from its title; sanitized with `All`; remote `remote/opf-format-corpus/jhove-hul-117-acrobat101-student-design-report.pdf` (247 pages, tagged) and `remote/govinfo/us-code-2023-title42.pdf` (9,302 pages, signed, insisted)	Retained memory flat between the 10 % and 100 % checkpoints, under budgets recorded in `status.md`; no allocation per glyph once warm, output excepted; throughput within a guard at three times the measured time	`CorpusRedactionTests.Redacting_the_largest_documents_holds_its_budget` (new), `RedactionBenchmarks` (new), `SanitizationBenchmarks` (new)
Every `damaged/*` document, `vendor/pdf-association/handwritten-type3-recursion.pdf` and the hand-written hostile files, marked and sanitized	A result, a report or a typed exception within each budget; no untyped exception, no hang	`CorpusRedactionTests.Hostile_documents_never_break_the_pipeline` (new)
Any document above, twice	Identical bytes	`CorpusRedactionTests.Redaction_and_sanitization_are_deterministic` (new)
The same operations through the tool	`redact` and `sanitize` give what the API gives, byte for byte, as a dotnet tool and as the AOT binary; refusals exit with 4	`CorpusToolTests.Redact_and_sanitize_match_the_api` (new)

The remote rows close only on a green Remote corpus run, recorded in status.md with its date. The rows that name documents not in the corpus close when the priority-1 gaps below are filled.

Corpus

What the corpus holds

Text to redact in every producer's form: our invoice through Chromium, LibreOffice, Word, the Print to PDF driver and PDF24 (whose box for every capital E M15 already reads through), the contract, the two reports with their outlines and tables of contents; Factur-X invoices whose XML repeats the buyer and the IBAN.
Tagged and claimed documents: seventy tagged committed documents, three PDF/UA-1 claims veraPDF upholds, the PDF/A claims M09 lists.
Signatures and permissions: six approval-signed documents and a certification; eighteen encrypted documents with permissions from open to modification and copying withheld.
Images: Flate, LZW, inline, soft-masked and Indexed images the core decodes; JBIG2, CCITT, DCT and JPX scans it does not, some under OCR layers.
Active content, found by a pikepdf walk over the committed files: scripts in the Cerfas, the USCIS I-9 and the HMRC form (104 JavaScript actions), a catalog /AA, an open-action script, a Launch action, a Screen annotation with a Rendition, XFA static and hybrid, attachments; remote, 3D, a Flash navigator, dynamic XFA, a video.
Hidden data: /Author holding a Windows account name; local user paths in the XFA packets of three forms, the XMP of a fourth, and Illustrator's private copy of its artwork under a page-piece dictionary; page-piece data in some twenty-five PDFMaker, InDesign and Illustrator files; page thumbnails in seven, XMP thumbnails in two, XMP histories in some fifteen; EXIF in the JPEGs of four; an annotated file with a highlight and a note; layers off by default; hidden widgets; an earlier revision that draws text the current one does not; metadata-only updates; a superseded /Info; remote, Photoshop's text layers in XMP, PDFpen's private keys, a stale tail.
OCR layers from Acrobat, two Xerox copiers and FineReader, to keep.
Scale: the 1000-page journal; remote, the 247-page tagged report and the 9,302-page United States Code.

What it lacks

Need	Why	Priority	Likely source
PII fixtures: fictitious IBANs, NIRs, SIRENs, SIRETs (La Poste's among them), e-mail addresses, French and international telephone numbers and dates of birth with their cues, all with valid check digits, and invalid twins — in body text, a table, across a line break and a hyphen, with narrow no-break spaces, and again in form fields, annotations, an outline item, `/Info`, XMP, alternative text and an attached XML —, from Chromium, LibreOffice and Word; `expect.detections` recording each planted value	The detectors' acceptance needs documents whose truth is known; the only real values in the corpus, the invoice's, fail their check digits, and personal data may never be committed for real	1	Generated here from our own sources, the values generated to pass or fail their checks and recorded in `build_corpus.py`; Word through `build_word.ps1`
A redaction done wrong: a black rectangle drawn over live text; a name replaced in an incremental update, the old revision still holding it; a document whose earlier revision holds a term its current one still shows	"Absent from every earlier revision" and `hidden.text-covered` need the failures they exist for	1	Generated here: recorded pikepdf constructions over our own documents
Concealed text: white on white, text of a size below a point, text off the page, text clipped away, text under an opaque shape	`InvisibleText` and five `hidden.*` rules have no committed case beyond render modes 3 and 7 (M15 lists it at priority 2)	1	Generated here: a recorded ReportLab construction; published prompt-injection samples examined before any is vendored
SubmitForm, ImportData, GoToE, Named actions beyond navigation, Hide, SetOCGState, and JavaScript in a link and an outline item	The roadmap names launch and submit actions; the committed corpus has a Launch action and no SubmitForm or ImportData	1	Generated here: recorded pikepdf constructions; a public source: government forms with submit buttons, examined first
`/Redact` marks left unapplied by Acrobat, with overlay text, `/Repeat` and `/RO`, and the same document after Acrobat applied them	Applying others' marks is in scope and only PyMuPDF's can be generated; Acrobat's applied result is the reference users compare with	2	A contribution; marks generated with PyMuPDF meanwhile
A lossless scan with text in its pixels — a Flate or LZW page image, with and without an OCR layer	Pixel redaction before M22 is proven on logos, photographs and Word's text-as-images; a scan is what users redact	2	Generated here: our own invoice rendered to an image and wrapped losslessly, with an OCR layer written from its hOCR by ReportLab
A JPEG with GPS coordinates in its EXIF inside a PDF	`hidden.image-location` has no case, and a phone photograph is the ordinary exhibit	2	Generated here: fabricated coordinates written by ExifTool into our own image, made a page through M07
RichMedia (Flash) and 3D annotations committed	The only ones are in the remote portfolio	2	A public source under an attribution-only license; otherwise stays remote
A tagged PDF 2.0 document	The `/Redaction` artifact subtype exists only in 2.0 output (M09 lists the same need)	2	The PDF Association's PDF/UA-2 and Well-Tagged PDF examples, remote if their license forbids committing
A Type 1 subset and a CID-keyed CFF subset whose glyphs a redaction leaves unused	Glyph scrubbing is proven on TrueType first; the corpus's Type 1 subsets are in two documents	3	The committed PDFMaker and InDesign files cover it partly; generated here otherwise

Traps

A box is not a redaction. Text under a rectangle is still text: selectable, searchable, extractable. Only removal counts, and hidden.text-covered reports the documents that got this wrong before they reached us.
An incremental update is an archive of what it replaced. Every revision is still in the file; a redaction written that way publishes the original. Full rewrite, every time.
The OCR layer is not the scan. Removing the words of an OCR layer leaves the pixels that show them; a mark over an image the core cannot decode is refused, not covered.
The same text lives in many places: a bookmark, the title, the XMP history, a figure's alternative text, ActualText, a field's value and its appearance, a note's popup, a named destination, an attachment, a thumbnail of the page as it was. Scrub by the model's types, then verify by scanning bytes, because the list is never complete.
Page-piece data can be a second copy of the document. Illustrator keeps its editable artwork — and a designer's local paths — in AIPDFPrivateData streams under /PieceInfo; the IRS publication in the corpus carries it on both pages. Redacting the page and leaving that is redacting nothing: page-piece data of a marked page always goes.
A subset spells what it drew. The glyphs a document no longer draws, and their ToUnicode entries, still say which characters it held.
Positions leak. A removed name's width, and the kerning around it, narrow the candidates; the overlay shows as much. Reflowing is not the answer for a received document — documenting the limit is.
Hidden content moves the pen. Removing a glyph must keep its advance; removing a painted path must keep the clip it set (M11's trap).
Word spacing applies to one byte. Tw adds to the single-byte code 32 only — never to a two-byte code, even one that draws a space (ISO 32000-2 §9.3.3); a displacement that assumes otherwise moves the rest of the line.
Inline image data can contain EI. A redacted inline image is written as an XObject.
Masks are pictures of the shape. An image's soft mask or stencil under a mark shows the outline of what was there.
A detector is a suggestion. One random 9-digit number in ten passes Luhn; a date near "né le" may be a signature date. Marks are reviewed before they apply, and the confidence floor says which to review first.
La Poste's SIRETs fail Luhn, and Corsica's NIRs have letters where the key's arithmetic expects digits.
French numbers are grouped with narrow no-break spaces (U+202F): a detector that splits on ordinary spaces misses most of them.
An IBAN in XML has no spaces. Scrub and verify the canonical form, not only the printed one.
Removing scripts can break a form: its calculations stop, its formats stop reformatting. Recognized Acrobat formats can be kept; the report says what else stops.
A dynamic XFA form without its XFA is a page saying "please wait" — reported, never presented as sanitized content.
Removing metadata can break a claim: PDF/UA wants the title, PDF/A the claims themselves. Metadata keeps them.
Sanitizing a signed document is re-making it. No signature survives a full rewrite; the flattened appearance shows what it looked like and proves nothing.

Documentation

docs/website/docs/concepts/redaction.md (new): removal against covering, the full rewrite, what a mark removes, verification, and the limits — positions, undecodable images until M22.
docs/website/docs/guides/redact-and-anonymize.md (new): mark, review, apply; detectors and terms; the overlay; tagged documents; attachments and Factur-X.
docs/website/docs/guides/sanitize.md (new): inspection, categories and profiles, what keeps working, what is lost.
docs/website/docs/concepts/validation.md and docs/website/docs/reference/validation.md: the disclosure profile beside the structural one.
docs/website/docs/reference/validation-rules.md: every action.* and hidden.* rule, its severity and its meaning.
docs/website/docs/reference/redaction-report-codes.md (new): every redaction.* and sanitize.* code.
docs/website/docs/reference/tool/index.md: the redact and sanitize verbs.
docs/website/docs/introduction.md: redaction and sanitization added to what the library does.
docs/architecture.md: the editing pipeline in Content/, the Redaction and Sanitization namespaces, the codec set M22 extends.
docs/adr/: the clipper decision of slice 2; NOTICE if Clipper2's sources are vendored.
docs/website/docs/guides/encryption.md (M16's): the permissions redaction and sanitization demand.
docs/corpus.md: expect.disclosureFindings and expect.detections; the referees pdfid, ExifTool, python-stdnum, phonenumbers and Pillow.
docs/features/features.json: the redaction entry brought to its state.
docs/status.md: the measurements and the calibrated thresholds.

Goal​

Scope​

Design​

Where it lives​

Always a full rewrite, always verified​

The content editing pipeline​

What a mark removes​

Text: removing glyphs without moving the rest​

Paths, clips and shadings​

Images and pixels​

The overlay​

Tagged structure​

Beyond the page: the same text everywhere else​

Fonts: glyphs that no longer draw​

Mark, review, apply​

Detectors​

Verification​

Inspection: the disclosure profile​

Sanitization​

Signed, protected and conforming inputs​

Bounds (invariant 12, ADR 34)​

Report codes​

The command-line tool​

Slices​

Tests required​

Acceptance conditions​

Corpus​

What the corpus holds​

What it lacks​

Traps​

Documentation​

Exit criteria​