When Community Feedback Contradicts the Model: What Counts as Evidence to Retire or Revalidate?

@Charles Bawate you've put your finger on the fault line that runs through most evidence systems: frontline observation is treated as anecdote precisely because it arrives informally, not because it's weak. My view is that lived experience should count as legitimate evidence from the moment it is systematically collected and traceable, which is a design choice, not a status the observation lacks. A field officer's "this is not what we see anymore" becomes evidence when it's logged with who observed it, where, when, and what specifically changed, and when it carries an obligation to be reviewed rather than filed. On formalising it, the minimum I'd argue for is three things: a named intake route (a standing agenda item with an owner), a defined threshold for triggering review (e.g., three independent field reports converging, or one report corroborated by routine data), and a documented decision, even if the decision is "no change," with reasons. That last part matters because it closes the loop and tells field teams their sensing was heard. On documentation for handover, I'd keep it minimal and usable: full code, parameter provenance with sources, calibration data, an assumptions log, and an explicit statement of where the model should not be used if another team can't rerun and revalidate without the original modeller, the documentation has failed. My one pushback on your framing: "a retired model is not failure" is right, but I'd add that a revalidated model isn't automatically success either sometimes the honest outcome is that the model was answering the wrong question for the new context, and no amount of revalidation fixes that.

@Charles Frontline observations must be treated as formal sentinel data, not mere anecdote. Lived experience legitimately triggers revalidation when field teams see recurring patterns across districts, when workers must rely on operational workarounds, or when structural conditions like cash-transfer policies or connectivity shift, invalidating baseline assumptions long before statistical drift registers on lagging dashboards.

To institutionalize this, embed a mandatory variance question into routine field reporting tools (e.g., "Did the model's output reflect field reality?"). Crossing an alert threshold automatically triggers an "amber review" on the governance agenda, mandating an operational pause and widening confidence intervals while teams investigate. Appointing a designated Field Custodian ensures qualitative ground truth bypasses data backlogs and reaches decision-makers as formal evidence briefs.

For local teams to revalidate models independently, minimum documentation requires a plain-language model card outlining failure boundaries, containerized code environments, and a parameter dictionary translating abstract formulas into real-world variables. Local teams must know the behavioral assumptions behind the math to test whether the model still holds. Retiring an obsolete tool is adaptive management; relying on a blind algorithm because we fail to hear frontline reality is the true institutional failure.

Thank you Rhoda and Nonvicks for this rich pushback — you both moved my thinking forward.