Welcome to the IDinsight Tech Blog
Check in regularly or sign-up to the IDinsight newsletter to find out about IDinsight's latest data science and engineering work.
All Posts
My board says do AI. Halp plz
Over the last year or so, the data science team at IDinsight has been busy building AI products like Ask-A-Question and Ask-A-Metric. A few months back, I was on a panel on GenAI for Social Impact and was asked if they should be investing in AI or not. I talked about how AI is a tool and we want to be problem driven. I talked about thinking about your use case and finding the tool that fits it best instead of starting with the AI hammer and looking for a nail. With all the hoopla around AI of late, these are questions on the minds of almost all social sector organizations. Below is how I see it. I’d love to hear your thoughts.
Why "Ask A Question"?
We’ve spent the last 8 months or so building Ask-a-Question, an AI question-answering service for direct-to-citizen helplines. We decided not to create yet-another-RAG solution. Ethan Mollick summarizes our concerns well. In short, the tech for a cheap, scalable, and guaranteed error-free AI is not there - the kind that you’d want to roll out to citizens in high risk and high trust use-cases like health. Instead, we wanted to lean on our experience building a question-answering for MomConnect to build something that is (a) trustworthy; and (b) provides actionable insights to support continuous learning and improvement.
Enhancing Maternal Healthcare: Training Language Models to Identify Urgent Messages in Real-Time
We have fine-tuned the Gemma-2 2-billion parameter instruction model on a custom dataset in order to detect whether user messages pertain to urgent or non-urgent maternal healthcare issues. Our model demonstrates superior performance compared to GPT-3.5-Turbo in accurately distinguishing between urgent and non-urgent messages. Both the dataset and the model have been made publicly available to support further research and development in this critical area.1
Comparing Ask-a-Metric and Vanna.ai
TL;DR: We are comparing Ask-a-Metric (AAM) and vanna.ai performance, on metrics that we find ourselves regularly testing for AAM use-cases. We find that Ask-a-Metric performs on-par with vanna.ai for straightforward queries, but struggles with more complex queries. Vanna.ai also struggles with complex queries and lacks guardrails, but has a greater range of features than AAM.1
Using Agents to Not Use Agents: How we built our Text-to-SQL Q&A system
Ask-a-Metric is a WhatsApp-based AI data analyst that uses LLMs to answer SQL database queries, facilitating data access for decision-making in the development sector (GitHub). Initially, we used a simple pipeline for rapid feedback but faced challenges in accuracy and building it for scale. We tested an agentic approach with CrewAI, improving accuracy but ending up with high costs and slow response speeds. We used these results to develop a pseudo-agent pipeline that combines the best of both approaches, reducing costs and response times while maintaining accuracy.1
