Data Migration for AI Assistants: Getting Your SOP Corpus GxP-Ready

Most failed AI compliance deployments in pharma do not fail at the model layer. They fail at the corpus layer. A retrieval-augmented assistant is only as defensible as the document set behind it, and in a typical mid-size manufacturer or CDMO that document set is scattered across a validated EDMS, a legacy Documentum instance nobody wants to touch, three SharePoint sites, a shared drive labelled QA_alt, and a folder of scanned PDFs from a 2011 site transfer. Migrating that into a GxP-ready corpus is a data migration project, and it should be governed like one — under GAMP5 Second Edition Appendix D7 (Data Migration), Annex 11 clause 4.8, and your existing change control procedure.

Treat corpus ingestion as a regulated data migration

Annex 11 §4.8 is unambiguous: where data are transferred to another data format or system, validation should include checks that data are not altered in value and/or meaning during the migration process. Loading SOPs into a vector database is exactly this. The source PDF is converted to text, chunked, embedded, and stored — three transformation steps, each with the potential to silently alter meaning.

Practical examples of meaning-altering transformation we see repeatedly in DACH sites:

• Table collapse. A cleaning validation acceptance criteria table extracts as a linear string of numbers with no column headers. The assistant retrieves it and pairs the wrong limit with the wrong product.
• Header/footer bleed. The document ID, version, and "Confidential – for internal use" footer are interleaved into every chunk, diluting embeddings and polluting citations.
• Bilingual documents. German-language SOPs with English annexes — common at Swiss and Austrian sites — chunked without language detection produce retrieval that silently prefers one language.
• Scanned annexes. OCR of a scanned equipment log template yields "1,0 mg" as "10 mg". This is a data integrity event, not a formatting nuisance.

If your migration protocol cannot demonstrate that a numeric acceptance criterion survives ingestion unchanged, you have not validated the migration. You have simply moved files.

Step 1: Define the corpus scope in writing

Before any file is touched, QA must approve a documented corpus scope. This is the AI equivalent of a system boundary definition, and inspectors will ask for it. At minimum specify:

• Included document classes — SOPs, work instructions, validation master plans, URS/FS/DS, quality manuals, deviation and CAPA procedures, ICH Q7 Section-relevant guidance.
• Excluded classes — batch records, executed protocols, personnel files, anything containing GDPR-relevant personal data, and any document in draft status.
• Status filter — effective/approved only. Superseded and obsolete documents must never be retrievable as authoritative answers. If you retain them for historical queries, they must be flagged in metadata and in the rendered citation.
• Site and scope tagging — a CDMO running Basel and a contract site in Poland cannot allow cross-site SOP retrieval without explicit tagging. Site-specific cleaning procedures are a classic cross-contamination of answers.

Step 2: Build the metadata schema before the pipeline

Metadata is what makes a citation defensible under Part 11 §11.10(e) and Annex 11 §9. Every chunk should carry, at minimum: document ID, title, version number, effective date, review-due date, document owner/department, status, site scope, language, and source system of record. Where the document lives in a validated EDMS, the metadata must be pulled from the EDMS API — never re-keyed by hand, and never parsed out of the cover page.

This matters because the metadata drives three separate controls: retrieval filtering (only effective versions), citation rendering (the answer shows "SOP-QA-014 v7, effective 12.03.2025"), and periodic review triggers (documents past review-due are flagged). A migration that lands text without governed metadata forces you to rebuild all three later.

Step 3: Qualify the extraction, not just the file count

Conventional data migration verification counts records in and records out. For a document corpus that is necessary but nowhere near sufficient. Design your IQ/OQ evidence around extraction fidelity:

• Golden set sampling. Select 30–50 documents stratified by format (native PDF, scanned PDF, Word, Excel-embedded tables) and by risk (a temperature excursion SOP outranks a stationery ordering procedure). Compare extracted text against source, character by character for numeric fields.
• Numeric integrity check. Automated regex extraction of all numeric values with units from source and target, with a diff report. Any mismatch is a deviation.
• Table structure check. Confirm that tabular acceptance criteria retain row-column association in the chunked output. If your pipeline cannot preserve tables, exclude table-heavy documents from scope and say so explicitly.
• OCR confidence gating. Set a threshold below which pages are quarantined for manual review rather than silently ingested. Document the threshold and its rationale.
• Chunk boundary review. Verify that no chunk splits a numbered procedural step or separates a limit from its unit.

Step 4: Handle the legacy backlog honestly

Most sites have a tail of documents that will not migrate cleanly: 1990s scans, documents in a decommissioned format, SOPs whose only controlled copy is a wet-signed paper original in the archive. Do not force these into the corpus. Under a GAMP5 risk-based approach, the correct answer is a documented exclusion list with justification, plus a user-facing statement of what the assistant does not cover. An assistant that transparently says "no effective document found in scope" is inspectable. One that hallucinates from a garbled 1997 OCR is not.

Step 5: Establish the ongoing sync as a controlled process

Initial migration is a one-off; corpus currency is forever. Define, in an SOP, how a newly effective document reaches the index and within what timeframe — most teams commit to same-business-day for critical quality documents. Define the reverse path with equal rigour: when a document is superseded in the EDMS, the corresponding vectors must be deprecated within the same window. Log every ingestion, update, and deletion in an audit trail that satisfies Annex 11 §9 and Part 11 §11.10(e): who or what triggered it, when, which document version, and the outcome.

Finally, wire the sync into change control. A change to the chunking strategy, the embedding model, or the extraction library is a change to the data layer of a validated system and requires impact assessment under Annex 11 §10 — including regression testing against your golden set.

What good looks like

A GxP-ready corpus is one where QA can answer, without hesitation: which documents are in scope, which are deliberately out, how extraction fidelity was verified, where metadata originates, how quickly the index reflects EDMS reality, and what the audit trail records. Every one of those questions has a documented answer, and the answers are traceable to a migration protocol and report approved before go-live.

Corpus quality is not a preparatory chore before the interesting AI work begins. It is the interesting AI work — and it is the part an inspector will interrogate first.

See how ComplianceGxP handles SOP corpus migration and GxP-ready document ingestion for pharma and CDMO teams: See how it works →