Custom Instructions
Answers that follow a company’s own definitions. IBM.
Business users can write rules about their data in plain language, such as “junior roles means levels 6 and 7”, and IBM watsonx BI follows them when it answers questions. We made the rules part of how questions are answered, from finding the right data to writing the SQL.
Node.js · TypeScript · LLMs · Prompt engineering
Why it matters
An LLM can read a table’s columns, but it cannot know a company’s vocabulary. That knowledge is not in the data. Without it, the model writes SQL that looks right and returns a wrong number. From our tests on a workforce dataset, where both models we tried made the same kind of mistake:
| Question | Without instructions, the models | What the business means |
|---|---|---|
| “Junior roles” | guessed low level numbers, such as 1 to 3 | levels 6 and 7 |
| “Regular employees” | filtered on the word “Regular”, which is not in the data | employment type FT |
Neither is really a model failure. The model is guessing, because the answer cannot be found by reading the schema. Other rules, such as “current means the latest snapshot”, guard against mistakes a model makes only some of the time.
The solution
Stored once, read everywhere
Users write the rules in the product’s interface. When a dataset is enriched, its rules are added to the dataset’s shared description, the same description every service already reads. Every feature that answers questions follows the same rules, and a rule changed in one place applies everywhere.
Used at every step that makes a choice
A rule can matter before any SQL is written, so it is used at each step where the system chooses something:
- finding the dataset that can answer the question;
- choosing between similar columns or values;
- writing the SQL.
In each prompt, the rules are stated as requirements the answer must meet, not as background the model can weigh against its own guesses.
Tested, not assumed
For several datasets we wrote rules and test questions, each with the SQL we expected. Every question ran with and without the rules, on two different models, and the results were compared. The tests included questions that need several rules at once, such as the current headcount of junior roles.
Results
Business users can ask questions in their own terms and get answers grounded in their definitions, without rephrasing them in column names and codes. In the workforce tests, the questions both models answered wrongly without the rules came out right with them.