Who Gets to Define Success for AI in Development?
A critical review of a webinar proposing gender-aware evaluation of AI in India’s development sector, with questions for evidence, accountability, and local context.
The video
A Gender-Intentional Framework for Evaluating AI-Based Solutions in India's Development Sector — LEAD at Krea University.
Uploaded 2026-06-10 · Duration 01:26:33.
A better question than whether the model works
LEAD at Krea University’s webinar makes a valuable argument: an evaluation of AI in development should ask who can use a system, whose knowledge it recognizes, and who bears the consequences of its mistakes. The opening presenter questions whether accuracy and engagement alone reveal who is excluded; a later presenter argues for examining model, product, user, and impact together. Opening question, 0:39; Proposed framework, 13:12.
The contribution is a shift in the unit of judgment. A health chatbot can produce an accurate answer in a test and still be difficult to use in a working day. A person may need privacy to ask a question, time to check an answer, or a way to challenge advice that conflicts with experience. Those conditions are part of the intervention’s performance. The framework is persuasive as a set of questions; the transcript alone does not establish that applying it improves outcomes.
There is also a tension inside the proposal. Evaluation tailored to a community can uncover problems that a generic benchmark misses, but tailoring makes comparison harder. A project could celebrate its locally chosen indicators while leaving out harms identified elsewhere. A useful framework needs both a small, transparent common record and room for communities to add, dispute, or reject measures. The common record should make assumptions and disagreements visible, rather than force every experience into one score.
What the discussion supports
Urvashi Vattal describes questions being asked about an early-stage anemia tool, including accuracy against standard diagnostic practice and performance across populations. She presents these as validation questions, not completed proof of benefit. Model evaluation, 17:53. For a health worker chatbot, she reports difficulty framing prompts and says guidance was followed by better responses and greater likelihood of use. The account is useful, but the supplied cues give no comparison design or measured effect sufficient to establish that the guidance caused an improvement. Prompting difficulty, 23:25.
Kalika Bali argues that translated benchmarks may miss context and that evaluation should draw on domain knowledge and end-user input. She also acknowledges a role for simple tests during model development. Translated benchmarks, 39:36; Basic benchmarks, 41:47. This is a stronger position than simply discarding benchmarks: a basic test can identify a failure, while a context-specific investigation asks what that failure means in use. Neither by itself establishes a social benefit.
The discussion becomes particularly challenging when the moderator advocates evaluators working as partners with organizations. Evaluator partnerships, 1:08:42. Early collaboration can improve the questions being asked, but closeness can also make it harder to report failure or recommend stopping. The missing operational question is who can publish an adverse finding, who pays for the evaluation, and whether people affected by the system can contest its conclusions. Calling an evaluator independent does not answer those questions; neither does calling them a partner.
Proposed tests
Compare actual alternatives. Before deployment, define the problem with affected workers and community members. Compare the proposed AI tool with existing support and a feasible non-AI alternative. Record response quality, time, costs, access, and the ability to obtain help when an answer is wrong. This is a proposed study, not a result of the webinar.
Follow the route from answer to consequence. In a prospective evaluation, record which version of a tool was used, what the worker understood, whether advice was checked, and what happened to the person receiving the service. Report missed or abandoned interactions as well as completed ones. Where numbers are too small to support reliable subgroup comparisons, say so and supplement them with voluntary, confidential interviews.
Test disagreement and accountability. Invite differently situated participants to assess the same realistic scenarios, document where their judgments differ, and give them a way to challenge the final evaluation. Set a decision rule in advance for serious unresolved harms, including who can pause deployment. Pay participants for their time and report the evaluation’s funding and decision rights. These are recommendations to test, not claims that the webinar’s projects already use them.
The site’s AI Safety Needs Evidence Across the Whole System asks how monitoring, permissions, and human intervention connect across an operating system. This webinar adds a concrete question for that analysis: whose access and authority should those controls serve when a tool passes through a worker to a community? The Global AI Benchmark Becomes the Geographic Blind Spot examines how a benchmark’s reference frame affects what its numbers mean. The webinar extends that concern from country-level numeric questions to language, local context, and the choice of whose judgments count. Neither article validates the webinar’s proposed framework.
Limits of this review
This review uses supplied machine transcription from audio, not human-verified captions. Recognition errors can distort names, numbers, and technical terms; the supplied transcription also marks a passage as uncertain, which is not used as evidence here. The cues refer to slides, but their contents and any visual demonstrations cannot be assessed from the transcript. Timestamp links identify the relevant supplied speech cues; they do not independently verify the speakers’ claims or the underlying studies. The transcript is marked non-redistributable, so this review paraphrases selectively rather than reproducing it.
AI-use disclosure
This transcript-based review was drafted by GPT-6.1 Terra with high reasoning and checked in a separate GPT-6.1 Sol session with medium reasoning. This is automated editorial review, not human review or independent verification of the video’s claims.
Transcript
Machine transcription from audio · en · checked 2026-10-27. Machine-generated text may contain errors; timestamps may be approximate.
This review uses an automatic transcription of the audio. Open video on YouTube.