Metadata Enrichment
How IBM watsonx BI understands a dataset, rebuilt twice. IBM, 2024–2026.
IBM watsonx BI answers business questions about a dataset in plain language. Before it can do that, it has to understand every column: a readable name and description, what the column represents (a city, a date, a state), and how it should be used: as a number to add up or average, a category to group by, or an identifier. Producing that understanding is metadata enrichment.
I worked on both versions. The first was built around an existing enrichment service from another IBM product, with our own processing on top. The second replaced that service with our own LLM pipeline.
Java · Node.js · TypeScript · LLMs · Embeddings · RabbitMQ · Kubernetes
Why it matters
Almost every feature of the product reads what enrichment produces:
| What enrichment decides | Where it shows up | If it is wrong |
|---|---|---|
| Readable names and descriptions | Answers, chart labels, search | Users see cst_unt instead of “Cost per Unit”, and search cannot find the column |
| What a column holds: a city, a date, a state | Finding the right column for “sales by state”; time on a chart’s x-axis | “Jan 2020” is treated as plain text, so there is no trend over time |
| Number or category | Which columns are added up and which are grouped; which chart is suggested | Postal codes get added up, and the wrong chart is suggested |
| Sum or average | Totals in answers; which numbers can share a chart | “Total price per kg”, a confident wrong number |
| How strongly columns relate to each other | “Why” questions, such as why revenue fell: finding the columns most likely to explain it | The explanation points at a column that has nothing to do with it, such as an ID or a budget column |
| Prepared sample values | Matching the words in a question to the values stored in the data | “New York” does not find “NYC”, so the filter is missing and the answer covers everything |
A mistake in enrichment does not stay in enrichment. It shows up as a wrong answer, a wrong chart, or a search that finds nothing.
The problem
The first version had three problems: it was expensive, at $5.32 to enrich one data asset; it was slow; and it misread many columns. Most accuracy problems came from two sources: matching values against fixed lists and patterns, and relying too much on column names.
| Column | What it holds | First version | Correct |
|---|---|---|---|
zip5, plz_de | US and German postal codes | a number to add up | a category |
on_time_flg | 0 or 1 | a number to add up | a category |
salary_band | salary tiers | a number to add up | a category |
x_coord | x-axis coordinates | a number to add up | a category |
price_per_kg, speed_mph | rates per unit | summed | averaged |
city_code | NYC, LA, CHI | not recognized | a city |
opening_date | Jan 2020, Feb 2020 | not recognized | a date |
region_name | US states | a country or region | a state |
These matter because they change answers. Adding up postal codes, or summing prices per kilogram instead of averaging them, gives a user a confident, wrong number.
The solution
A model that reads meaning, not just names
The second version uses an LLM that understands what values mean. It knows NYC is a city and a price per kilogram is a rate, without a list or a pattern for either. It looks at more than the column name: the data type, sample values, and how many distinct values the column has. That is how it tells postal codes from quantities, even though both are numbers.
Smaller prompts instead of one large one
A model often does better when a large task is split into several smaller prompts, each asking for one clear decision. Deciding how a column should be used works this way: simple rules settle the obvious columns first, and the rest go through a few narrow questions in turn.
Wide tables
Some tables have hundreds of columns, too many to handle well in one request. They are split into batches that keep neighbouring columns together. A few batches run at a time, with a short pause between groups so the model service is not overloaded. A one-paragraph summary of the whole table, written from the column names alone, goes with every batch, so each batch knows what the table is about.
Reliable output
- The model must return JSON that matches a defined schema, written with Zod and requested through the Vercel AI SDK. The response is checked against the schema before anything uses it.
- Failed calls are retried with exponential backoff.
- Each step stops the run cleanly if it fails, and records how long it took.
- If sample values are not available, enrichment continues with a shallower analysis instead of failing.
Respecting people’s edits
Users can correct the names, descriptions and other details enrichment produces. Enrichment never overwrites a value a user has already accepted or defined.
Rolling it out safely
The first version was already serving customers, so the second could not simply replace it.
- A switch, not a deploy. Enrichment starts from messages on a queue. A configuration setting decides whether a new dataset goes down the old path or the new one, so traffic can move to the new version, or back, without changing code.
- Versioned results. Every result records which version produced it. Small improvements within a version can be applied when a result is read, if the caller allows it. A move from the first version to the second never happens automatically; it needs an explicit rebuild. A customer’s column names and settings never change without someone choosing it.
Results
| First version | Second version | |
|---|---|---|
| Cost to enrich one data asset | $5.32 | $0.009 |
| Speed | 1× | 6.5× |
| Accuracy | — | above 90% |
Cost fell 99.8%, which meant $19K in monthly customer savings. Accuracy is measured on a benchmark of test tables from many domains, with the correct answer written down for every column. Each step is scored on its own, and the benchmark is rerun as a regression check after changes.