AI INTEGRATION
Data engineering for AI is the work that decides whether anything above it can succeed. Pipelines, embeddings, vector stores, permission models and PII handling, built so agents read the right data at the right freshness and never see what they should not.
Why retrieval quality is a data problem first
Data engineering for AI is the practice of preparing, moving and governing data specifically for agents and models to consume: ingestion, cleaning, entity resolution, chunking, embedding, indexing, permission propagation and freshness.
It differs from conventional data engineering in what it optimizes for. A warehouse pipeline serves analysts asking aggregate questions of structured tables. An agent pipeline serves a system asking specific questions of unstructured content, under someone’s permissions, in milliseconds, and it has to be correct at row level rather than in aggregate. Approximately right is a rounding error in a dashboard and a wrong answer to a customer.
The other difference is that permissions become part of the data layer rather than the application. Once content is embedded into an index, the access control that governed the original document does not come with it unless somebody engineers that. This is the single most common way AI projects create a data-exposure problem.
You probably need this if you recognize these
- The same customer exists three times with three spellings and nobody knows which is right.
- Content lives in a shared drive nobody has audited since it was created.
- Your assistant answers from a document that was superseded last quarter.
- Access control is enforced in an application and nowhere else.
- Nobody can say which personal data would end up in an index.
- Re-indexing is a manual job somebody remembers to run.
What you get
Source inventory and classification
What data exists, who owns it, what is sensitive, and what is safe for an agent to read.
Ingestion pipelines
Scheduled and event-driven, handling the formats you have rather than the ones you wish you had.
Entity resolution
One identity per customer, supplier or product across systems that disagree.
Embedding and indexing
Chunking, embedding and vector store setup, with re-embedding when models or content change.
A permission model
Access carried from source to index and enforced at query time, so retrieval can never widen entitlements.
PII handling
Detection, classification, redaction or tokenization where required, and retention rules that actually execute.
Freshness and lineage
Defined per source, monitored, with every record traceable to where it came from and when.
How data becomes agent-ready
Data engineering for AI
Inventory
Sources, owners, sensitivity and volume, mapped before anything is moved.
Self-healed - retried with fallback tool. Human not required.
Ingest
Pipelines pull or receive content, with format handling and failure alerting.
Resolve
Entities are matched across systems so one customer is one customer.
Protect
Sensitive fields are detected and handled by rule before anything is indexed.
Embed
Content is chunked, embedded and indexed, with permissions attached to every record.
Refresh
Freshness rules run per source, and lineage records where every answer's data came from.
Agents that are only as good as their data
CRM Hygiene Agent
Entity resolution made visible: it exists because the same record is in three places.
Analytics Digest Agent
Degrades quietly when an upstream pipeline changes shape, which lineage catches.
Knowledge-Base Agent
Its accuracy is a direct function of indexing and freshness.
Where the data comes from
We work with the warehouse you have. Replacing it is almost never where the return is.
- Data warehouse and lake
- Object storage and shared drives
- Document management
- CRM
- ERP
- Ticketing
- Wikis
- Event streams
- Vector stores
- Identity providers, for permission propagation
Platform names are shown as examples of the categories agents connect to. They are not partnerships or endorsements.
How data access is governed
Autonomy boundary
Agents read from curated indexes with permissions attached, not from raw sources with broad service accounts.
Approval gates
Adding a new source, or widening what an index contains, is a reviewed decision with a named data owner.
What stays human
Classifying sensitivity, deciding retention, and resolving contradictions between authoritative sources.
Logging
Lineage on every record and an access log on every query, so both “where did this come from” and “who saw this” have real answers.
How the work runs
Typical ranges from our engagement model (doc 04 §5), not a quote.
| Stage | Typical | What happens |
|---|---|---|
| Pilot | 3-10 days audit, then 2-4 weeks | Source inventory, sensitivity classification and a proven pipeline for one high-value source. |
| Build | 3-8 weeks | Pipelines, entity resolution, PII handling, embedding, permission propagation and freshness. |
| Release | 1-2 weeks | Indexes released by audience, with permission checks verified before each widening. |
| Managed | ongoing, optional | Pipelines monitored, re-embedding on model changes, new sources onboarded. |
What we measure
- Query-time
- Permissions enforced per request, never flattened into the index (Target)
- Lineage
- Every indexed record traceable to its source and version (Target)
- Per source
- Freshness defined and monitored rather than assumed (Target)
- Your baseline
- Duplicate and stale-record rates, measured before and after (Yours)
“ Until then these are design targets and engineering commitments, not results.
What this looks like in practice
eCommerce catalog and ads hive
Reference scenario · eCommerce. Catalog enrichment across a large SKU count, which is an entity-resolution problem before it is anything else.
Reference scenario - a composite build illustrating our method. Figures are modeled and the model is shown.
Frequently asked questions
Do we need a data warehouse before we can do any of this?
No. Agents often read directly from source systems through governed integrations, and a warehouse is one option rather than a prerequisite. Where you have one, we use it. Where you do not, we would rather build the narrow pipeline your agents need than sponsor a platform program that delays everything by a year.
How do you stop an index from leaking data?
Permissions travel with the content and are enforced at query time against your identity provider. This is the failure everyone should worry about: embedding a document strips the access control that governed it, so one index can quietly flatten years of carefully configured permissions. We treat it as a security design problem and log every query with who asked and what was returned.
What about personal data and retention?
Sensitive fields are detected and classified before indexing, then redacted, tokenized or excluded by rule according to your policy. Retention rules are implemented rather than documented, including deletion propagating to derived indexes - which is the part most implementations miss and regulators ask about.
Which vector database should we use?
Whichever suits the workload, and it matters less than the surrounding decisions. Chunking, permission propagation and freshness affect answer quality far more than the store. We build behind an interface so the choice can change without a rewrite.
Our data is genuinely messy. Is this hopeless?
No, and it is the normal starting point. What matters is whether it is messy in a patterned way, which it usually is. The audit assesses that in days and tells you which sources are worth preparing first. We would rather clean one source properly than all of them badly.
How does this relate to RAG?
This builds and governs the data an index is made from; RAG and knowledge systems tunes how that index is searched. Retrieval quality problems are diagnosed across both, and the fix is frequently here rather than in the search configuration.
Related services
Self-Healing Infrastructure Agents
Agents that keep pipelines and services running.
AI Integration & Implementation
Reaching the systems this data comes from.
RAG & Knowledge Systems
Searching what this work indexes.