Using AI to improve maternal health chatbots in South Africa

Amahle is pregnant and soon expecting a new addition to her family. She has been seeking maternal care through South Africa’s national WhatsApp helpline for the past seven months where she frequently consults a help-desk team about pregnancy challenges she’s been facing. At one point she got really worried that she couldn’t feel her baby move and it took a while for the help-desk to get back to her. Soon, she will be able to get instant recommendations with the help of a technology developed by IDinsight that will automatically answer her questions.

Praekelt.org, in partnership with the national government, runs multiple WhatsApp-based chat helplines focused on public health for the citizens of South Africa. But often, information delivery and support over helplines have limited social impact. This is because human experts need to spend substantial time on individual cases before they are able to provide useful answers, especially when it comes to intricacies of sexual health, COVID-19, pregnancy, and other topics etc.

IDinsight is working with Praekelt.org to build an AI-powered plug-in, called “Ask-A-Question” or “AAQ”, for a national maternal and child care helpline that automatically forwards health content addressing the queries of mothers like Amahle. With an ability to correct misspelled words, process grammatically incorrect sentences and comprehend colloquial terms, AAQ stands at the forefront of modern technology deployed at a national scale. This plug-in is still being tested and piloted. Below we share our progress thus far.

Improving chatbot experience for new and expectant mothers in South Africa

South Africa’s mobile maternal health program has served more than four million women since its launch in 2014. Mothers are automatically registered at their first antenatal care visit and can then opt -in to receive stage-based information about pregnancy, labor, and the first several years of their child’s life. For more personalized support, they can connect to a team of help-desk operators responsible for any pressing questions they might have.

The help-desk team maintains a collection of standard responses to frequently asked questions (FAQs). When Amahle asks a question, the help-desk operator searches for an appropriate FAQ and forwards it to her. When no relevant FAQ is available, the question is forwarded to a trained midwife who offers customized support. Alternatively, Amahle can browse through the full list of FAQs on her own.

The Problem

Unlike search engines like Google, finding information on a WhatsApp-based service requires mothers to navigate through lengthy content menus. Instead, mothers end up sending their question to the help-desk team.

However, a user base of several million women can easily overwhelm even the most dedicated team of help-desk operators. In peak times, the team of 3 receives more than 1200 messages per day. And in the context of maternal health, where the situation could be life-threatening, it is very important for them to quickly and accurately answer the high volume of incoming questions.

We thus needed an automated way to respond to common questions with FAQs. This would allow the help-desk operators to focus on nuanced and urgent cases like Amahle’s.

So, how does AAQ help?

AAQ is an algorithm that matches a question, in free-form text, to stored FAQ content using Natural Language Processing (NLP) techniques.

The AAQ application returns top FAQ matches to any message sent to it. We have worked with the Praekelt.org team to configure AAQ for maternal health topics by collating expert-approved FAQs, defining contextual word meanings (like treating delivery as birth and not bringing) and then plugging the application into the chatbot. The chat flow has been configured such that mothers will receive top FAQ suggestions for their question, and then an option to forward their query to the live help-desk if they are still unsatisfied.

Integration of AAQ with the chatbot will ensure that at least the commonly asked questions are automatically addressed through FAQs; anxious mothers will get high-quality health information instantly rather than needing to wait hours for an available help-desk operator.

But…can we trust it?

We have experimented with several different approaches to increase the accuracy of AAQ’s FAQ predictions. Some of these require the program manager to “tag” each new FAQ content piece to direct the algorithm on how best to make matches, others (that rely on Google’s BERT algorithm) use a handful of example questions to train the algorithm, and others work by more directly comparing similarities between incoming questions and the full FAQ content (using an approach called “Word Mover Distance” estimation).

The latest deployment has a top-5 accuracy of ~65% when assessed on actual historical chat data, i.e. the correct FAQ appears in the top 5 matches 65% of the time – which is higher than Praekelt.org’s threshold of 50%. This made it worthwhile to deploy the algorithm for broad use on the platform. Considering the current workload of the help-desk, we are hopeful the adoption of AAQ will allow the operators to spend more time on tough and more urgent questions they receive from expecting women and new moms across the country.

Simultaneously, we are continuously upgrading the core AAQ algorithm and will eventually be incorporating user feedback to make further improvements.

Supporting workstreams and future chatlines

An alpha version of the AAQ approach was previously deployed for a COVID-19 hotline in 2021 and for the maternal health helpline, it has already shown promising results in a test setting.

The focus of Praekelt.org and IDinsight teams currently is to deploy AAQ for active use and test it on ~300 beta users before launching it to all registered mothers. In parallel, the help-desk operator team is being trained on the changes they can expect in their day-to-day work routine.

Apart from the core FAQ-matching algorithm, the IDinsight team delivered a number of complementary features to make the AAQ system (1) easier to work with and (2) more informative for program managers and help-desk operators. These include:

• An urgency detection module within AAQ to flag urgent messages based on keywords of medical urgency in the user message (like intense nausea during pregnancy). When active in full production, these messages will get flagged for priority for help-desk operators and trigger a template message to users about contacting their nearest medical provider.
• An Admin Web Application to add or edit the FAQs in the database and urgency detection rules.
• A feedback collection unit that captures user satisfaction with the FAQ recommendations made by AAQ that can be used to refine algorithms and content.
• A Tableau dashboard for project managers to monitor the traffic and feedback on AAQ, workload on help-desk and quality of the FAQ content.
• A Grafana dashboard for engineers to monitor the health of AAQ applications.

In the longer term, IDinsight is working with Praekelt.org to open source AAQ so it can be used (and contributed to!) by many other help-desk organizations in the social sector. In the interim, we are working with Praekelt.org to adapt and deploy this application to other helplines. This includes a separate national helpline focused on serving young adults with questions about sexual and reproductive health. We are also exploring the development of additional functionalities, such as user level personalizations and support for non-English language messages.

If you would love to know more about AAQ, or perhaps discuss how it might be useful for your context, please contact us at dsem@idinsight.org.