All Posts

A thumbnail image

Optimizing data loading using Airflow and Redshift

SurveyStream is a software product developed by IDinsight to support and streamline primary data collection operations. SurveyStream helps data collection teams manage survey operations more efficiently, freeing up their time to focus on other crucial activities both before and during a survey. SurveyStream functions as a web application backed by a robust data management system that survey teams can use to manage enumerator hiring, assign enumerators to respondents, send periodic emails to enumerators with their assignments and get timely insights into productivity and data quality.

A thumbnail image

Two New Principles for Humanising AI

AI has the power to free up human time and improve lives. Yet it can only do so if it is designed ethically and inclusively. There’s a significant risk that AI will primarily extract from, rather than serve, ordinary people. Just as with the industrial and digital revolutions before, people may become further alienated from the world around them. We want to forestall this, without adding drag to the full and remarkable potential for social impact from technology. We propose two new principles to drive AI development toward promoting dignity and human connection, ensuring that users - students, patients, citizens - are treated with respect for their humanity. In this fast-moving field we hope these will guide IDinsight’s own work, and we welcome feedback on how to improve them further.

A thumbnail image

ElectionGPT: How AI-powered tools supported India’s elections––and what may be next for AI in the public sector

In 2024, elections took place in some 70 countries, home to half the world’s population. In India alone, 642 million people cast ballots1 in the general election, making it the largest democratic exercise in history. Ensuring the smooth conduct of Indian elections for hundreds of millions of people was no small feat and required the dedication of millions of support staff. ] in the general election, making it the largest democratic exercise in history. Ensuring the smooth conduct of Indian elections for hundreds of millions of people was no small feat and required the dedication of millions of support staff. Our team worked with the government of Uttar Pradesh where millions of registered voters participated in the elections. To support this monumental task, IDinsight developed ElectionGPT–– an AI-powered tool designed to help election officers in India. The tool served as an indispensable resource, answering thousands of questions with speed and accuracy, while saving officials countless hours.

A thumbnail image

Let’s Make Sure the Right-Hand Rule is Left-Behind

Household surveys are a critical source of data for understanding the conditions, experiences and aspirations of families. Governments and social sector organizations use data from household surveys to inform program design, targeting, service delivery, budget allocations and more. Household surveys give families a voice – through data – in the policies and programs that affect their lives. But it is impossible to reach all households, and not all households are alike. So how do surveyors choose who to visit to ensure that their data are representative of the wide variety of families in a given place? That is a question about sampling.

A thumbnail image

Three Stage Sampling

An earlier version of this blog post appeared on my person blog. Household surveys often involve more than one “stage” of sampling – e.g. in the first stage, we might randomly sample villages and in the second stage we might randomly sample households within these villages. Most often, we use two stages when sampling. Accounting for two sampling stages is pretty straightforward. In some cases, we might want to consider using three stages. Unfortunately, to my knowledge, there aren’t a lot of good resources on how to account for more than two stages when sampling. In this post, I’ll try to answer four questions:

A thumbnail image

My board says do AI. Halp plz

Over the last year or so, the data science team at IDinsight has been busy building AI products like Ask-A-Question and Ask-A-Metric. A few months back, I was on a panel on GenAI for Social Impact and was asked if they should be investing in AI or not. I talked about how AI is a tool and we want to be problem driven. I talked about thinking about your use case and finding the tool that fits it best instead of starting with the AI hammer and looking for a nail. With all the hoopla around AI of late, these are questions on the minds of almost all social sector organizations. Below is how I see it. I’d love to hear your thoughts.