Creating Good Conversational AI Isn’t Easy

Anyone who works in market research will have noticed that over the last year a whole host of technology companies have launched conversational AI products promising ‘Qual At Scale’, ‘Qualitative Surveys’ or some such similar claim. You could be forgiven for thinking that creating a conversational AI product for market research is easy. In one sense it is; it is easy to get GPT or another LLM to write a probing question. If you're good with tech it's then relatively straightforward to build a platform on top of this and, hey presto, you have a ‘Qual at Scale’ offer for market research.

Large language models are basically predictive engines that have been fed massive amounts of text (the internet) and have been trained to generate text based on a prompt given to them. However, there are a lot of well-documented issues with LLMs, such as hallucinations and the perpetuation of biases based on the underlying data sources. So, if you ask an LLM to generate a probing question for market research based on a participant’s answer to an open-ended question, if you're lucky you will get a good probe. However you may get a probe that doesn't make sense, asks a bad question (e.g. a leading question), or is inappropriate or insensitive.

Framing this in terms of behavioral science, we can think of LLMs as System One. They are instinctive, and will say the first thing that the algorithm generates. They're like that person we've all met who seems to have no filter and blurts out the first thing that comes into their head. Sometimes this is wise, sometimes it's funny, but sometimes it's downright inappropriate. If you want to ask good questions for market research that not only elicit good insights but are also ethical, this is a problem. 

inca SmartProbe Uses Neuro-symbolic AI

Whilst the underlying technology for inca SmartProbe is GPT, the combination of our AI and market research experts have applied a lot of training on top to ensure inca SmartProbe asks good questions for market research. The approach we have used leverages Neuro-symbolic AI, which can be explained as applying a layer of System 2 reasoning to the System 1 output of Large Language Models.

To explain a little further, Neuro-symbolic AI recognizes that cognition encompasses both System 1 (i.e. fast, reflexive, intuitive, and unconscious) and System 2 (i.e. slower, step-by-step, and explicit). It does this by integrating neural and symbolic AI architectures to address the weaknesses of each, providing a robust AI capable of reasoning, learning, and cognitive modeling*. Both are needed for AI that can learn, reason and interact with humans to ask good questions. 

What This Means In Practice

The application of neuro-symbolic AI for inca SmartProbe has several profound implications versus less sophisticated approaches:

1. Quality control — inca SmartProbe has been trained by our in-house market researchers, using a huge number of best practice probe examples, to ask the right questions: probes which emulate market research principles and which are ethically sound. In practice, when a participant answers an open-ended question, inca SmartProbe generates several candidate probes based on its knowledge bank of market research principles and examples. Candidate probes are then flagged by our extensive in-house classification system, which helps to ensure ethical and professional probe questions by filtering out 20+ sub-categories of bad question types. These flags for bad questions are divided into leading questions (including giving examples or even mentioning the wrong brand name), unethical (such as probing for personal information or agreeing with toxic statements), and unprofessional (for example asking questions in the wrong dialect, or which don’t pertain to the research objectives). By leveraging the creative outputs of System 1 constrained with our rigorous System 2 principles, inca SmartProbe asks effective and ethical questions in real-time.

   
   Note: measuring and ensuring the quality of participant data is a separate but related challenge. inca SmartProbe’s high-quality probing questions have been shown to keep participants engaged and to elicit more accurate and actionable responses. In addition, inca SmartProbe provides a response demerit tool which helps to indicate participant responses which can be defined as low quality, i.e. that are made with low effort or not in good faith.

2. Steerable and interpretable — at Nexxt Intelligence our commitment is to keep humans at the center of research. With inca SmartProbe, this means elevating participants’ voices, but also giving researchers the tools to understand and control the types of questions that are dynamically asked to participants. This is analogous to the process a researcher would follow with a human moderator in a qual setting, briefing the moderator to focus in relevant ways for certain questions. Similarly, we have developed methods for the researcher to brief the AI. 

With inca SmartProbe, researchers are able to incorporate research context which helps to ensure that probes are sensible and tailored to the researcher’s objectives. By providing just a few examples of ideal questions to the trained models feature, researchers can ensure that generated probes will emulate the type and style of the questions they’d like to ask. Conversational targets allow researchers to control which aspects are probed on, and furthermore bring structure to how researchers design and analyze their inca SmartProbe conversations.

1. Eliciting information at scale for actionable insights — as market researchers, our ultimate goal is to understand humans at scale, and discover actionable insights based on our research objectives. inca SmartProbe has been designed to elicit relevant information in a structured manner**, and elevates researchers’ ability to understand the rich information collected in a number of ways. 

Conversational targets (described above) can also be used as a quantitative variable so researchers can easily count how often a key term is used by research participants in their answers and use the counts as cross variables for analysis. 

inca AI Coding (available on the inca Conversational AI platform dashboard or as a discounted bundle license with inca SmartProbe API) automatically codes the rich verbatim data from inca SmartProbe but puts the researcher at the center to be able to edit and optimize the AI generated coding. Like inca SmartProbe, inca AI Coding has been trained and optimized specifically for market research surveys. 

Custom probing frameworks (please contact us for details) can be developed for specific client methodologies and needs by representing the type and structure of information needed and then both probing and capturing information in a structure such that it can be readily analyzed at scale. For example, imagine a scenario where a client is interested in using probing to capture concrete physical details of a product, whereas for the packaging they are more concerned about probing for emotional associations. 

Combined, these elements of our unique approach to conversational AI make inca SmartProbe a compelling blend of qualitative and quantitative principles, purpose-built for market research interviews.

Summary Of Key Points

• Not all Conversational AI is the same.

• inca SmartProbe uses Neuro-symbolic AI which combines the intuitive questioning of System 1 with the reasoning of System 2.

• inca SmartProbe is the result of a partnership between leading AI specialists and experienced market researchers. It is technology that has been created by market researchers for market researchers.

• Consequently, inca SmartProbe will ask good probing questions for the market research context. Probes that:

• Follow best practice market research principles (are not leading, do not ask more than one question, do not ask closed questions).

• Follow a high standard of ethics (are not inappropriate, do not probe further to profanities or toxic content, do not ask for PII information).

• Can be easily steered based on the researcher's examples and objectives.

• Enable open-ended analysis at-scale.

*source = Wikipedia Neuro-symbolic AI - Wikipedia

** source = Flipping the Script: Inverse Information Seeking Dialogues for Market Research (Seltzer et al., 2022)