Rethinking "Risk" in Algorithmic Systems Through A Computational Narrative Analysis of Casenotes in Child-Welfare

by   Devansh Saxena, et al.

Risk assessment algorithms are being adopted by public sector agencies to make high-stakes decisions about human lives. Algorithms model "risk" based on individual client characteristics to identify clients most in need. However, this understanding of risk is primarily based on easily quantifiable risk factors that present an incomplete and biased perspective of clients. We conducted a computational narrative analysis of child-welfare casenotes and draw attention to deeper systemic risk factors that are hard to quantify but directly impact families and street-level decision-making. We found that beyond individual risk factors, the system itself poses a significant amount of risk where parents are over-surveilled by caseworkers and lack agency in decision-making processes. We also problematize the notion of risk as a static construct by highlighting the temporality and mediating effects of different risk, protective, systemic, and procedural factors. Finally, we draw caution against using casenotes in NLP-based systems by unpacking their limitations and biases embedded within them.


page 1

page 2

page 3

page 4


Designing Human-Centered Algorithms for the Public Sector: A Case Study of the U.S. Child-Welfare System

The U.S. Child Welfare System (CWS) is increasingly seeking to emulate b...

How to Train a (Bad) Algorithmic Caseworker: A Quantitative Deconstruction of Risk Assessments in Child-Welfare

Child welfare (CW) agencies use risk assessment tools as a means to achi...

Algorithmic Harms in Child Welfare: Uncertainties in Practice, Organization, and Street-level Decision-Making

Algorithms in public services such as child welfare, criminal justice, a...

Unpacking Invisible Work Practices, Constraints, and Latent Power Relationships in Child Welfare through Casenote Analysis

Caseworkers are trained to write detailed narratives about families in C...

A Human-Centered Review of the Algorithms used within the U.S. Child Welfare System

The U.S. Child Welfare System (CWS) is charged with improving outcomes f...

Please sign up or login with your details

Forgot password? Click here to reset