Suggested Questions

When someone opens a dataset in IBM watsonx BI, they see a few questions they can ask right away, written for that dataset. We built the service that writes them with an LLM and checks, before showing any of them, that each one actually returns an answer.

Node.js · TypeScript · LLMs · RabbitMQ · Docker · Kubernetes

Why it matters

Faced with a new dataset, most people do not know what to ask. Suggested questions are often their first interaction with the product. A good one shows what the data can answer. A suggested question that fails is worse than none: it teaches the user not to trust the product before they have asked anything themselves.

The problem

An LLM can easily write questions that sound right for a dataset. Many of them fail when you actually try to answer them:

• no query can be written for the question, because it asks for something the data does not hold;
• the query is written but is invalid and fails;
• the query runs but returns no data, for example because a filter matches nothing.

From the question alone there is no way to tell which ones will work.

The solution

Questions written from what the data means

Questions are generated from the dataset’s enriched description: readable column names, descriptions, and which columns are numbers and which are categories. A question then combines a meaningful number with a meaningful breakdown, such as revenue by region, instead of pairing columns at random.

Every question is answered before it is shown

Each generated question goes through the same path a user’s question would:

question → SQL → checked → run on the real data → answer

Only questions with a real answer are ever shown. The others are recorded with the reason they failed, which shows where generation needs to improve.

One query format for many databases

Datasets can live in different databases, and each one speaks its own dialect of SQL. Rather than teaching the model every dialect, the model writes SQL in one format, guided by examples in the prompt. Code then fixes common mistakes in its output, such as how limits, column aliases and date intervals are written. The result is converted to an intermediate query format, which another service translates into each database’s own SQL.

Bounded work

• Validation stops as soon as there are enough good questions.
• An overall time limit cancels whatever has not finished.
• The number of model requests running at the same time is limited, so a large batch of datasets cannot overload the model service.
• Datasets with too little in them to ask anything useful are skipped.

Prepared ahead of time

Questions are generated and validated in the background, once the dataset has been enriched. The results are stored and cached, so they appear instantly when someone opens the dataset.

Results

Users get a guided first interaction with every dataset, using questions that are known to return an answer. The service won an IBM Outstanding Innovation Award.