The 2024 Foundation Model Transparency Index

Citation: Rishi Bommasani, Kevin Klyman, Sayash Kapoor, Shayne Longpre, Betty Xiong, Nestor Maslej, Percy Liang (2024/07/17) The 2024 Foundation Model Transparency Index.
Internet Archive Scholar (search for fulltext): The 2024 Foundation Model Transparency Index
Wikidata (metadata): Q135645196
Download: https://arxiv.org/abs/2407.12929
Tagged:

Summary

Updates the Foundation Model Transparency Index six months after its launch by scoring 14 developers on the same 100 indicators across upstream, model, and downstream domains, but with a key methodological shift: developers submit itemized reports (not just public web evidence). Average transparency rises from 37 → 58/100, with developers disclosing new, previously non-public information on ~16.6 indicators on average; yet sustained, systemic opacity remains around copyright status, data access, data labor, and downstream impact. The team also releases a public transparency report for each developer, arguing the Index itself likely contributed to improved disclosure.

Theoretical and Practical Relevance

Establishes a defensible, repeatable yardstick for model-developer transparency that regulators, procurers, and funders can operationalize (e.g., set minimum indicator baselines, require developer-submitted attestations, and reference per-developer reports). The results support policy interventions targeted at lagging areas (e.g., external data access, mitigations/evaluations) while preserving a comparison-ready dashboard that separates where transparency improved from where it hasn’t—useful for eligibility rules, oversight, and monitoring over time.