NTT Develops AI Technology That Visualizes Expert Knowledge for Business Succession Planning
- Leveraging LLM to model expert decision-making processes using dialogue data-

News Highlights:

Figure 1 Overview of Expert Decision-Making Process Extraction and Visualization

Background

As labor shortages worsen in areas such as security incident response and call center operations, there is a growing need to outsource tasks to less specialized staff and to automate query handling using technologies like large language models (LLMs). Meanwhile, expert staff who play a central role in query response rely on their own unique knowledge and thought processes, making their decision-making difficult to observe or formalize. As a result, the transfer of operational knowledge in query handling has become a significant challenge.

To address this issue, NTT has developed the world's first AI technology that leverages LLM to analyze dialogue data and accurately visualize expert decision-making processes. By applying these extracted insights, even novice staff can replicate expert-level responses, enabling more consistent and efficient operations.

Technology highlights

Step 1

Using LLMs, we extract questions and suggestions from all text-based dialogue data (Figure 2: Extraction). The extracted questions and suggestions are then grouped by content to create a merged question list and a merged suggestion list (Figure 2: Merged List).

Figure 2 Extraction of Questions and Suggestions and Creation of Merged Lists

Step 2

For each dialogue data, the LLM references the merged question list and merged suggestion list to trace which question in the list was addressed, what answer was provided, and how the conversation ultimately led to a particular suggestion.

This allows the entire process, from question to answer to suggestion, to be structured as a flow.
Applying this process across all dialogue data enables the transformation of unstructured dialogue data into structured flows.

Figure 3 Dialogue Data Structuring

Step 3

In each structured flow, transitions from one question or suggestion to the next are treated as a single step. The LLM counts the number of times each transition occurs across all flows. These transitions are then organized into a tree structure, with frequently occurring transitions placed higher in the hierarchy. This process generates a question and decision-making flowchart that represents common response patterns.

Figure 4 Construction of Question and Decision-Making Flowchart

Overview of the experiment

To evaluate the accuracy of this technology, we utilized FloDial¹, a publicly available dataset consisting of dialogue datasets and their corresponding flowcharts, commonly used in the assessment of natural language processing technologies. Using the dialogue data from FloDial, we generated flowcharts with our method and compared them with the ground-truth flowcharts included in the dataset.

By extracting all dialogue data from both sets of flowcharts and comparing them, we found that our method successfully reproduced approximately 90% of the question and suggestion tree structures present in the ground-truth flowcharts.

Future developments

This technology addresses the challenge of transferring operational knowledge in query response tasks such as security incident handling and call center operations. By leveraging this method, even novice staff can replicate expert-level responses. Having validated the approach using FloDial, an ideal publicly available dataset, the next step is to apply it to real-world dialogue data.

To ensure practical applicability, we aim to improve the accuracy of extraction and visualization so that the system can handle incomplete data and discontinuous dialogue, which are common in real-world business contexts.

[Glossary]

¹FloDial: Flowchart Grounded Dialogs Dataset
URL:https://dair-iitd.github.io/FloDial/

About NTT

NTT contributes to a sustainable society through the power of innovation. We are a leading global technology company providing services to consumers and businesses as a mobile operator, infrastructure, networks, applications, and consulting provider. Our offerings include digital business consulting, managed application services, workplace and cloud solutions, data center and edge computing, all supported by our deep global industry expertise. We are over $90B in revenue and 340,000 employees, with $3B in annual R&D investments. Our operations span across 80+ countries and regions, allowing us to serve clients in over 190 of them. We serve over 75% of Fortune Global 100 companies, thousands of other enterprise and government clients and millions of consumers.