Dataset Autopsy: What Could 10 Years of Customer Support Data Teach an AI?

1. Understanding what the archive actually contains

Consider a customer-support organisation that has operated for ten years. Its systems contain emails, live chats, phone transcripts, CRM notes, knowledge-base articles, escalation records and ticket-status changes.

Management might describe the archive as containing 1.8 million customer interactions. That figure sounds substantial, but it is not yet meaningful for AI development.

The first question is what constitutes an independent observation.

A single support case might contain eight messages, three status changes, two internal notes and an escalation. Counting these as fourteen separate records would exaggerate the number of independent examples. Conversely, a database storing only the final ticket status may conceal valuable diagnostic reasoning that remains in its underlying messages.

The correct unit of analysis, or case grain, depends on the intended use. A summarisation model might require complete conversations with trustworthy summaries. A diagnostic assistant may require the sequence of symptoms, investigative actions and verified resolution. An evaluation dataset may require a self-contained problem, an authorised solution and a defensible scoring rubric.

This distinction has practical precedent. Feigenblat and colleagues introduced TWEETSUMM, a customer-service dialogue dataset containing approximately 6,500 human-annotated summaries. Its research contribution was not simply the quantity of conversations, but the connection between dialogue content and annotated summaries for a defined task ¹.

An operating company should therefore begin by identifying whether its systems preserve the following relationship:

Customer problem → operational context → investigation → decision or action → observed outcome.

The relationship is rarely held neatly in a single table.

Customer context may reside in a CRM, product versions in an asset-management system, attempted fixes in agent notes and financial outcomes in a billing platform. Combining these sources requires stable identifiers, compatible timestamps and reliable definitions.

Historical system migrations introduce further complications. Ticket categories may be renamed, customer identifiers replaced and resolution statuses consolidated. A category labelled technical fault in 2017 might encompass several distinct issues by 2026. If this transformation is undocumented, apparent trends can reflect administrative changes rather than real operational differences.

Exhibit 1: Anatomy of an AI-useful support case

Illustrative record structure. The example contains no real customer information.

This structure illustrates why context matters. The final instruction to install a patch is less informative without the symptoms, competing explanations and reasons earlier interventions failed.

It also highlights an important distinction: a case marked resolved is an administrative outcome. Independent confirmation that the problem stayed resolved provides stronger evidence, although even that may be incomplete if the customer subsequently sought support through another channel.

2. Signal density: separating routine volume from specialist knowledge

Customer-support archives are not uniform collections of expertise.

Routine requests can dominate volume: password resets, delivery tracking, account-access queries and repeated troubleshooting instructions. Such records may be useful for intent recognition, response drafting or identifying common language patterns. Their usefulness for teaching sophisticated diagnostic reasoning may be more limited.

Complex cases can contain richer information. An agent might evaluate competing explanations, consult multiple systems, recognise an exception, escalate to a specialist and revise an earlier recommendation.

These sequences reveal not merely what was said, but how the organisation approached a problem.

A useful analytical distinction is between four categories of evidence:

• Routine interactions: repetitive, predictable requests with standard responses.
• Diagnostic interactions: cases requiring investigation or interpretation of multiple information sources.
• Exception and escalation cases: situations in which standard procedures fail or an authorised decision-maker intervenes.
• Correction and outcome cases: records showing whether a previous action succeeded, failed or required revision.

These categories can overlap. A routine request may become an escalation, and an escalation may end without a verifiable outcome.

The commercial significance lies in their relative suitability for a particular buyer task. A vendor developing an automated first-line support assistant may value high-volume routine conversations. A developer evaluating complex troubleshooting behaviour may require rarer cases with expert-labelled outcomes.

In that sense, signal density is not a universal numerical property of a dataset. It is the concentration of evidence relevant to a defined task, given the quality, diversity and reliability of that evidence.

An archive that contains 200,000 repetitive conversations may be inferior to a smaller specialist archive for one task, yet more useful for another.

The distribution must also be examined for representativeness.

An archive may overrepresent customers who complain, users of older products or cases handled by a particular regional team. Its apparent success rate may reflect how cases were closed rather than whether customers received effective assistance. Expert notes may record shortcuts that were historically tolerated but are no longer acceptable.

Bender and Friedman argue that documentation of the populations and circumstances represented in language datasets is important for evaluating bias and generalisability ². This principle applies directly to customer-support records.

An AI system trained on a narrow operational history should not be presumed to perform reliably across different customers, languages, products or regulatory environments.

The practical objective is therefore to measure not only how much information exists, but which behaviours and outcomes the archive can legitimately represent.

3. Matching the archive to an AI application

The expression training data can obscure several fundamentally different uses.

A customer-support archive may support retrieval, fine-tuning, dialogue summarisation, classification, evaluation or the development of AI agents. Each use requires different preparation and produces different risks.

Retrieval-augmented assistance may draw on approved troubleshooting documents, current policies and verified solutions without incorporating every historical conversation into model weights. Here, freshness, document permissions and source attribution are especially important.

Supervised fine-tuning may use examples of customer questions and approved responses. However, an historical agent response is not necessarily a correct training target. A response may have been incomplete, outdated or subsequently overturned.

Evaluation datasets require independent cases with reliable expected outcomes or assessment criteria. Keeping evaluation examples separate from development data matters because exposure of test cases during training can inflate measured performance. Research by Sainz and colleagues examines this problem of benchmark contamination ³.

Agent development and evaluation introduce a more demanding requirement: the system must follow policies, interact with tools and carry out actions correctly. The τ-bench research programme evaluates simulated customer-service agents using tools, policy instructions and verifiable end states. Its design illustrates why completing an action correctly is different from merely producing a plausible conversational response ⁴.

Exhibit 2: AI use-case suitability matrix

This comparison prevents a common commercial error: proposing that an entire historical archive be licensed for unspecified AI training.

A more credible proposition identifies a narrower opportunity.

For example, a software company might offer a pilot involving complex enterprise troubleshooting cases, with verified resolutions and documented product versions. The buyer could test whether those cases improve the detection of faults that general-purpose systems mishandle.

Alternatively, if the archive contains excellent dialogue transcripts but weak outcome records, summarisation or intent classification might be a more realistic initial application.

The strongest opportunity is not necessarily the most ambitious one. It is the use that can be supported by available evidence at acceptable cost and risk.

4. Worked dataset autopsy: from 1.8 million messages to usable evidence

Consider a hypothetical customer-support business with a ten-year archive containing 1.8 million messages.

Its CRM identifies 240,000 historical tickets, implying an average of 7.5 messages per ticket. That average is not a measure of task complexity; a handful of lengthy escalations may account for disproportionate volume.

Following reconciliation, duplicate removal and spam filtering, 150,000 distinct cases remain.

The company then examines the quality of the outcome information.

Suppose 40% of those cases contain a resolution code, 18% record an escalation and 9% contain an outcome-related field, such as a recorded reopen status within 30 days. The existence of that field does not establish that follow-up was complete, that non-reopening meant a successful resolution or that an independent reviewer verified the result.

Exhibit 3: Illustrative archive-quality funnel

All quantities are hypothetical. The last three rows are separate attributes of the 150,000 cleaned cases, not sequential filtering stages. Their overlap has not been established.

The headline archive contains 1.8 million messages, but the quality review reveals a much narrower potential evaluation population.

The 13,500 cases with recorded outcome-related fields are not verified-outcome cases or a ready-to-license population. The company must establish whether the outcome was genuinely observed, whether the field means the same thing across periods, whether monitoring was consistent and whether associated actions were correct under the applicable policy. The 60,000 resolution-code cases, 27,000 escalation-flag cases and 13,500 outcome-field cases may overlap; they must not be interpreted as consecutive funnel stages.

A further quality audit might discover that certain products have reliable outcomes while others do not, or that older records lack essential context.

The commercial response should be to construct a task-specific sample rather than assume all cleaned cases belong in a single transferable product.

For instance, the company could select a stratified sample of 2,000 cases across product families, issue types, years and escalation levels.

The sample would be reviewed for completeness, factual reliability and permitted use. It would also test whether the archive contains enough difficult cases to distinguish one AI system from another.

A rigorous pilot would separate data used to develop or tune a system from data reserved for independent evaluation. Case-level separation is important: placing different messages from the same customer-support case in development and evaluation sets could leak the answer.

Temporal separation can provide an additional check. Evaluating on later cases may reveal whether a system generalises to changing products and policies. But a chronological split is not sufficient by itself if historical data and later cases share duplicated templates, leaked outcomes or recurring customer details.

The key question is what survives the quality audit, not merely the number of records retained.

What would an AI buyer learn?

A buyer might conclude that the archive is strong for summarisation, adequate for supervised classification and unsuitable for independent evaluation until missing outcomes are reviewed.

Another buyer, targeting complex troubleshooting, might find value in a small specialist subset despite the broader archive's limitations.

Both conclusions are consistent with the same underlying dataset.

The assessment becomes useful because it connects observable characteristics to specific technical applications rather than treating all records as commercially interchangeable.

5. Rights, privacy and transformation: preserving utility without creating unacceptable risk

Customer-support systems routinely capture information beyond what is required for an AI development task.

Free-text messages may contain names, email addresses, account identifiers, order references, screenshots, payment information, credentials, sensitive complaints and third-party documents.

A company may control the support platform without owning or possessing unrestricted rights to every element within it.

The rights review should distinguish the company's own records from customer-supplied material, licensed documentation, contractor contributions and confidential information.

It should also consider the original purpose of collection, contractual restrictions, retention obligations and whether external AI development represents a permitted secondary use.

The relevant answer will depend on the jurisdictions, contracts, data categories and proposed recipient.

Privacy transformation introduces a separate technical challenge.

Removing obvious identifiers is not sufficient if a unique combination of dates, locations, circumstances or free-text descriptions can still reveal a person's identity.

The UK Information Commissioner's Office explains that pseudonymised data remains personal data and that pseudonymisation reduces risk rather than automatically placing information outside data-protection obligations ⁵.

For AI development, excessive redaction can also destroy the relationships that make a case useful.

If a troubleshooting dataset loses product version, transaction sequence or device configuration during transformation, it may no longer support the intended task. Conversely, preserving unnecessary personal details can increase exposure without improving performance.

A proportionate strategy starts by specifying the minimum evidence the AI system needs.

Potential approaches include removing unnecessary fields, replacing identifiers with securely managed tokens, generalising certain values, filtering particularly sensitive cases or allowing controlled evaluation without transferring the raw archive.

The optimal technique should be tested against both privacy risk and task performance.

Documentation is essential. Gebru and colleagues' Datasheets for Datasets provides a framework for explaining a dataset's origin, composition, collection and intended uses ⁶. NIST's Generative AI Profile similarly addresses documenting data lineage, transformations and evaluation practices ⁷.

For an operating-company archive, this suggests a data inventory recording field definitions, source systems, transformation history, known exclusions, limitations and permitted uses.

A buyer should be able to understand what the resulting dataset represents without assuming that the supplier's historical operational knowledge is universally understood.

Sometimes the appropriate outcome is not to licence the data at all.

If customer agreements do not permit the proposed use, sensitive material cannot be safely separated or the cost of establishing permissions is disproportionate, the company should defer external commercialisation.

A controlled internal AI project may still be commercially attractive, but internal deployment also requires appropriate privacy, confidentiality and governance safeguards.

6. Preparation economics and the management decision

The technical potential of an archive is only one side of the commercial assessment.

The other is the cost of producing a reliable, documented, legally usable data product.

Historical records may require extensive engineering work to connect systems, reconstruct timestamps, harmonise categories and verify outcomes. Subject-matter experts may need to review cases individually. Privacy and legal teams must assess permitted use. A buyer may also require secure access, repeatable exports, ongoing monitoring and contractual commitments.

These activities can turn an apparently inexpensive data opportunity into a significant operating project.

Consider the hypothetical 2,000-case evaluation pilot described earlier.

Assume that each case requires six minutes of initial expert review, with additional adjudication for difficult examples. Engineering and legal teams prepare the controlled dataset and its delivery environment.

Exhibit 4: Illustrative pilot preparation budget

All hours, rates and costs are illustrative assumptions, not observed licensing benchmarks. The budget excludes taxes, financing costs, unpriced liabilities and later commercial support.

A further illustrative £5,000 of first-year support would bring the immediate cost requirement to £48,020.

That is not an estimate of the dataset's market value. It is a measure of the expenditure that the contemplated project would need to justify.

The economic question is therefore not simply whether a buyer might pay for the archive. It is whether a defined use can produce sufficient incremental value, under acceptable rights and risk conditions, to cover preparation and ongoing obligations.

If the prospective buyer cannot demonstrate a relevant technical improvement, the firm should avoid committing to full-scale extraction.

If a pilot establishes strong results, the company can assess whether further investment is justified and whether the commercial structure allows reuse of the preparation work for other permitted applications.

A non-exclusive arrangement may preserve future options, but only if its actual terms allow the relevant subsequent uses. Exclusivity could justify different economics while constraining future opportunities. Neither structure is inherently preferable without considering the buyer's task and the seller's alternatives.

The organisation should also compare external licensing with internal value creation.

Better support analytics, faster case retrieval, improved agent training and lower repeat-contact rates may offer measurable benefits without transferring the records to an external data buyer.

These internal benefits are not guaranteed either. They require baselines, controlled testing and a defensible way to attribute improvements.

Practical implications

A management team assessing a customer-support archive should be able to answer five questions before making a substantial commercial commitment:

1. What is the case-level evidence? Can conversations, actions, decisions and outcomes be reliably linked?
2. Which AI task is plausible? Is the archive suitable for retrieval, training, evaluation or agent workflows, and what would demonstrate improvement?
3. How trustworthy is the sample? Are labels, product versions, historical changes and outcome measurements sufficiently documented?
4. Can the proposed use be permitted and controlled? Are rights, confidentiality, personal data and transformation risks manageable?
5. Do the economics justify preparation? Is there a credible benefit large enough to warrant expert review, technical work and ongoing obligations?

A decade of customer-support history can be an important operational asset, but it is not automatically a commercially marketable AI dataset.

The defensible proposition is narrower and more useful: a defined collection of records, with known origins, interpretable actions, verifiable outcomes, appropriate permissions and a measurable application.

A million messages demonstrate activity. A carefully documented collection of problem-solving trajectories may demonstrate capability. The difference is established through evidence, not volume.