Which products use generative chatbots

Chatbot study in the financial sector 2020: NLU technology as a success factor

The 2020 Chatbot Study in the Financial Industry by Prof. Dr. Peter Gentsch, AI expert and institute director for conversational business at Aalen University, shows surprising results.

Savings banks and Volksbanks beat fintech: The results show that savings banks and Volksbanks with an average hit rate of 70 percent rank well ahead of direct banks and fintechs with an average of 41 percent, while insurance companies achieve an average hit rate of only 35 percent.

It is astonishing that big players like Ergo or Arag come off with a hit rate of less than 5 percent. It is just as astonishing that in some cases the off-topic and personal questions were answered better than the technical questions. The chatbot study in the financial sector can be downloaded free of charge: https://chatbots.conversationalbusiness.de

Chatbots: artificial intelligence for customer centricity

Against the background of the current discussion about customer centricity and artificial intelligence, the sometimes poor performance of the examined chatbots is surprising. It turns out that chatbots are not a plug & play solution, but rather require dialogues that have been conceived and trained by the customer.

Results of the chatbot study in the financial sector

Who is leading the study ranking? Sparkasse Nürnberg, Sparkasse Köln, Volksbank Freiburg and VR-Bank Westmünsterland lead the ranking. This is followed by insurance companies, direct banks and fintechs. In the last places, Ergo and Arag bring up the rear in the study.

Differentiation of industry comparisons

However, the comparison between the industries has to be put into perspective. The relatively poor performance of the insurance companies can partly be explained by the fact that they often specialize in a few products, but the questions were based on a broad range of products in order to be able to cover all products.

Chatbots with NLU technology do the best

If companies use modern conversational AI technologies such as NLP, NLU and deep learning, then chatbots can bring high economic benefits and thus competitive advantages such as positioning and reputation due to their scalability and time and cost efficiency. It turns out that the companies that use NLU (Natural Language Understanding) technology do significantly better overall.

Reliable understanding of context only with NLU approach

Companies could benefit from this for general dialogues and larger industry topics. For company- and product-specific answers, however, the training volume is not comparable in terms of volume. Likewise, customer and company-specific contexts can only be reliably guaranteed with NLU (Natural Language Understanding) approaches, which require appropriate lexicons and extensive context modeling.

Research design of the chatbot study on the financial sector

In the study, 13 selected chatbots from the insurance and finance industries were analyzed at the Institute for Conversational Business at Aalen University for Technology and Economics. On the basis of qualitative and quantitative questions, the respective chatbots were examined from the consumer's point of view with regard to their ability (intents), i.e. to correctly recognize the inquiries and needs and inquiries of customers and to provide the correct answer.

Evaluation criteria

In order to evaluate the conversational ability of the chatbots accordingly, in addition to technical questions about products and conditions, “off-topic” and personal questions were asked. Five criteria were used to operationalize the quality of conversations and responses:
1. Accuracy of the answers to the respective question
2. Response speed
3. Ease of use / user-friendliness
4. Informative character of the answers
5. Entertaining nature of the answers.

Confirmed study findings "Conversational Business 2020"

In the previous chatbot study “Conversational Business 2020 - Status Quo and Trends” 600 consumers were asked about the acceptance of chatbots. The findings show that chatbots are often still developed too much from the perspective of cost savings and process automation and not from the customer perspective. Customers generally welcome chatbots if the response quality is acceptable. However, many consumers are frustrated with their chatbot experiences and therefore avoid them. A bad chatbot is therefore counterproductive for the customer experience. This suggests the recommendation: Better not have a chatbot than a bad chatbot. On the other hand, companies that use chatbots with a high response rate report very positive experiences.

When do chatbots get intelligent?

The latest success reports from Google’s Meena, Facebook’s Blender or OpenAI’s GPT-3 proclaim the next big AI milestone after the hype about deep learning. Indeed, the AI-based dialogues are fascinatingly human-like and entertaining. All three AI systems are based on huge amounts of training on the Internet. However, there is no monitored quality control here, and the bias that exists on the Internet is also learned. The question is what benefits this new level of conversational AI means for companies in addition to the entertainment effect.

The royal road to the future is a hybrid approach

According to Prof. Dr. Gentsch a hybrid approach that combines knowledge concepts and control logics with the generative approach. This allows the power of implicit learning through deep learning to be used in a quality-assured manner and, at the same time, company-specific and product-specific knowledge to be incorporated.

Potential of chatbots through customer centricity

This shows that chatbots in Germany are only gradually becoming more intelligent. However, the wide spread of the results also shows that it is of great importance which technology is used. If companies want to use the potential of chatbots in terms of their scalability as well as time and cost efficiency, they have to think and understand dialog lines from the customer's point of view on the one hand, and on the other hand they have to deal with conversational AI technologies such as NLP, which have become massively better in recent years , NLU and deep learning.

About Prof. Dr. Peter Gentsch

Prof. Dr. Peter Gentsch is a speaker, entrepreneur and scientist in one person and has been one of the pioneers and top experts in the field of digital transformation, artificial intelligence (AI) and big data since the 1990s. While others only talk about digitization and technology trends, he lives them. With numerous company foundations and investments and five successful exits, he is one of the most successful internet entrepreneurs in Germany: https://chatbots.conversationalbusiness.de

If you are interested in a Kauz chatbot demo, just send us a message.

Subscribe to our newsletter right away so that you are always up to date.