The Political Nature of Software: Recognizing and Embracing Ethical Responsibility in Tech

DevOpsDays London

The Inherent Responsibility of Software [0:00:00]

• Kestral Gaian opens by thanking the DevOpsDays London team and BSL interpreters.
• She introduces the difficult topic that "code is never neutral" and challenges the audience to consider responsibility for harm, starting with analogies like atomic weapons and gun violence, then extending it to software that causes physical, psychological, or emotional harm.
  • Unlike long-standing issues like nuclear weapons, society has not had enough time to contemplate the harm software can cause.
  • Responsibility is never simple; it doesn't neatly sit with one group but is distributed across every step in the software creation chain.
• The presentation aims to be a rousing call to action rather than a "doom and gloom" session, emphasizing that every choice made in software development, no matter how small, adds up and influences real lives.
  • Software systems leak into how people work, travel, love, learn, and survive.

Real-World Consequences of Unethical Software [0:06:26]

The Post Office Horizon Scandal [0:06:26]

• The Horizon software, used by UK sub-postmasters for accounting, erroneously reported shortfalls, leading to over 900 post office workers being accused of theft and fraud.
  • Many were bankrupted, humiliated, and some imprisoned for crimes they did not commit. Several took their own lives before receiving justice.
• Horizon was not an isolated incident but followed two previous flawed systems, CAPTURE and ECCO+, spanning 40 years.
  • All three systems had fundamental software flaws, leading to wrongful convictions.
  • A toxic management culture pushed for convictions, but the underlying issue was a lack of rigorous testing, particularly in real-world settings.
• The speaker highlights that the tech maxim "move fast and break things" can have devastating human consequences, where "things" are not just systems but actual people.

  

Tesla Autopilot Fatalities [0:09:31]

• Tesla's "Autopilot" (an Advanced Driver Assist System, ADAS), released in October 2015, has been directly at fault in nearly 1,000 serious incidents, including 51 fatalities.
  • Tesla's PR described these fatalities as "acceptable margins of error."
• The speaker questions who grants companies the right to determine "acceptable margins" for human life, especially when users trust technology to ensure their safety.
  • Compares it to a doctor casually stating an amputation is "within acceptable range," emphasizing the lack of choice and the gravity of such decisions in software updates.

Optum Health's Biased AI [0:12:20]

• Optum Health, a US company, used AI to identify patients with the highest healthcare needs, a noble goal.
• However, their AI routinely underestimated healthcare needs for Black patients due to a critical missing variable and flawed testing.
  • The system used past healthcare spending as a proxy for need, failing to recognize that Black patients historically received less care and thus spent less, due to systemic inequality, not better health.
  • This programmer blind spot led the algorithm to learn the "wrong lesson," prioritizing Black patients lower.
• This incident highlights what happens when developers forget the real-world impact of code on people's lives, leading to class-action lawsuits.

  

Understanding the "Political" in Software [0:14:42]

• "Political" here refers to "politics with a small P," which is fundamentally about power.
  • It determines: Who gets seen? Who gets excluded? Who gets heard? Who gets silenced? Which communities receive access to resources? Which families get help? Which identities are represented, and which voices are amplified or omitted?
• Software is no longer an isolated technical tool; it shapes every aspect of modern life: healthcare records, housing applications, financial systems, border controls, elections, and even entertainment algorithms.
• Every developer, whether writing infrastructure scripts, database schemas, machine learning models, or even hobbyist code, is performing a political act that shapes the world's power dynamics.
• The speaker asserts that "The World Is Software" and that anyone who writes code or interacts with a computer is shaping the world.
  • Software is now a service to humanity, and every piece of it contributes to this overarching puzzle.

Personal Experience: Exclusion in Tech [0:18:35]

• As a member of the LGBTQ+ community, Kestral shares personal experiences of feeling betrayed and excluded by tech.
  • Forms often betray a lack of diversity in the originating team, offering limited options for relationship statuses or assuming traditional configurations.
  • There's a constant lack of appropriate options for pronouns and gender, often in mandatory fields for irrelevant purposes (e.g., buying a car).
• Despite being tech-savvy, she is routinely excluded and often told by companies that they are "aware" of such issues but lack the "effort," "budget," or "time" to fix them, or will forward it to a "compliance team."
• While these exclusions may seem minor compared to issues like nuclear war, they highlight that importance doesn't solely lie in existential threats; validation and acceptance for all users are crucial.
• Many users lack the privilege, time, or knowledge to seek alternatives or report issues, making it essential for developers to consider diverse needs.

  

Beyond Compliance: Cultivating Ethical Literacy [0:23:17]

• Ethical software development is not a matter of mere compliance or a one-time checkbox; it is an ongoing, daily practice, much like literacy.
  • It involves the continuous ability to understand the world clearly and to code responsibly.
  • Just as developers wouldn't ship code without testing or deploy without monitoring, they shouldn't ship systems without considering their impact on people's lives.
• The speaker acknowledges the challenges: deadlines, pressure, burnout, lack of managerial rewards, and the feeling that "doing the right thing" often requires extra, unpaid work.
• However, responsibility does not vanish simply because it is difficult. Every line of code has consequences, and every deployment changes something for someone. Developers cannot opt out of this responsibility.

A Code to Code By: 5 Steps for Ethical Development [0:26:26]

The speaker provides five actionable steps for developers to embed ethical considerations into their daily work:

1. Ask "Who Does this Exclude?" [0:27:09]
   • Exclusion is subtle; it hides in defaults, edge cases, and seemingly "fine" forms.
   • Explicitly asking this question before shipping or building can open doors for thousands of people. Even if a fix isn't immediately possible, knowing the exclusion is half the battle.
2. Make Diversity a Feature [0:27:59]
   • Diversity, inclusion, and equality are not just "nice-to-haves" but are critical for building systems that accurately reflect the real world and help catch blind spots.
   • Actively fight for diversity as an integral part of design, not just decoration.
3. Fight for Transparency [0:28:41]
   • This is especially crucial for AI systems: if a system cannot be explained, it cannot truly be trusted.
   • Demand clarity even when business pressures push for speed. Trust is an invaluable currency that technology cannot afford to lose.
   • Ensure documentation is thorough and avoid "black boxes" in the codebase.
4. Speak Up [0:29:48]
   • This is the most challenging step because it requires overcoming discomfort, but silence is also a decision that reinforces the status quo.
   • It's not necessary to have all the answers; simply asking the right questions can be enough to initiate change.
5. Normalize Reflection [0:30:16]
   • Integrate ethical reflection into routine development processes like retrospectives and stand-ups.
   • Ask questions like: "What did this exclude?", "How did we do?", "What went wrong/right?"
   • When reflection becomes a normal practice, responsibility becomes embedded in the team's culture.

Conclusion: Empowering Choice and Collective Impact [0:31:12]

• The speaker emphasizes that "It is choice, not chance, that determines destiny" (Jean Nidetch).

  

• Every code commit is a choice, every deployment is a decision, and every system built shapes the world.
• Developers are more influential than they might realize and are not powerless.
• By working together and consciously making ethical choices in the code they write, the systems they deploy, and the teams they work with, they can collectively make a significant difference in building a more thoughtful and inclusive world.