Responsible AI Bias Audit Project

Artificial Intelligence project 20

Responsible AI Bias Audit Project

Build a complete responsible ai bias audit workflow with reproducible Python code, transparent inputs, reviewable output, and responsible-use limits.

Explore AI training in VizagView all project ideas

AI objective

Compare selection rate, true-positive rate, and false-positive rate across groups and quantify simple gaps.

Data or knowledge source

Three thousand synthetic binary outcomes and predictions across three fictional groups.

Requirements

  • Python 3.10 or later
  • A terminal or command prompt
  • python -m pip install numpy pandas
  • About 45-60 minutes to build and review

How the system works

Build group confusion-count components, calculate rates with safe denominators, compare minimum and maximum selection rates, and calculate TPR and FPR gaps.

Validation checklist

  • Rates stay between zero and one
  • Every group carries a row count
  • Zero denominators return missing rather than false certainty
  • Metrics are not labelled a legal fairness determination

Complete Python code

Save the program as ai_responsible_bias_audit.py. The code runs locally and requires no paid API key or model download.

"""Audit synthetic binary predictions across groups."""

from __future__ import annotations

import numpy as np
import pandas as pd


def make_demo_predictions(n: int = 3000, seed: int = 42) -> pd.DataFrame:
    rng = np.random.default_rng(seed)
    group = rng.choice(["Group A", "Group B", "Group C"], n, p=[0.45, 0.35, 0.2])
    qualified = rng.random(n) < np.select([group == "Group A", group == "Group B"], [0.58, 0.52], default=0.55)
    score = 0.2 + 0.62 * qualified + rng.normal(0, 0.19, n) + np.where(group == "Group C", -0.07, 0)
    predicted = score >= 0.5
    return pd.DataFrame({"group": group, "y_true": qualified.astype(int), "y_pred": predicted.astype(int)})


def safe_rate(numerator: int, denominator: int) -> float:
    return numerator / denominator if denominator else np.nan


def audit(data: pd.DataFrame) -> tuple[pd.DataFrame, dict[str, float]]:
    rows = []
    for group, frame in data.groupby("group"):
        tp = int(((frame.y_true == 1) & (frame.y_pred == 1)).sum())
        fp = int(((frame.y_true == 0) & (frame.y_pred == 1)).sum())
        positives, negatives = int((frame.y_true == 1).sum()), int((frame.y_true == 0).sum())
        rows.append({"group": group, "rows": len(frame), "selection_rate": float(frame.y_pred.mean()),
                     "true_positive_rate": safe_rate(tp, positives), "false_positive_rate": safe_rate(fp, negatives)})
    summary = pd.DataFrame(rows)
    min_rate, max_rate = summary["selection_rate"].min(), summary["selection_rate"].max()
    metrics = {"selection_rate_ratio_min_to_max": float(min_rate / max_rate if max_rate else np.nan),
               "tpr_gap": float(summary["true_positive_rate"].max() - summary["true_positive_rate"].min()),
               "fpr_gap": float(summary["false_positive_rate"].max() - summary["false_positive_rate"].min())}
    return summary, metrics


def main() -> None:
    groups, metrics = audit(make_demo_predictions())
    print(groups.round(3).to_string(index=False))
    print({key: round(value, 3) for key, value in metrics.items()})
    print("Group metrics reveal disparities but do not determine legality, fairness, cause, or an acceptable policy.")


if __name__ == "__main__":
    main()

Run the project

  1. Create and activate a virtual environment.
  2. Install dependencies with python -m pip install numpy pandas when packages are required.
  3. Run python ai_responsible_bias_audit.py.
  4. Review confidence, fallbacks, sources, or error metrics rather than accepting output automatically.
  5. Test additional normal, edge, unsupported, and adversarial inputs.

Expected output

Group-level selection, TPR and FPR tables plus selection-rate ratio and error-rate gaps.

Accuracy, privacy, and responsible-use limits

Metrics reveal disparities but cannot determine fairness, legality, cause, or acceptable policy. Real audits need context, intersectional groups, uncertainty, data review, governance, and affected stakeholders.

Ways to extend the project

Add confidence intervals, threshold curves, calibration by group, intersectional analysis, missing-group review, documentation, and mitigation evaluation.

Continue learning Artificial Intelligence

Try the next project, return to the Softenant project library, or explore the AI training in Vizag for guided NLP, retrieval, evaluation, automation, and responsible AI practice.