This paper assesses and tracks the transparency of leading foundation model developers in the AI industry. This is achieved by evaluating these developers against a set of 100 transparency indicators across various domains such as data, labour, compute, capabilities, risks, and impact. The paper aims to provide insights into the current state of transparency, identify areas of opacity, and suggest possible improvements and policy interventions.
Key takeaways
1.Purpose and methodology:
- The Foundation Model Transparency Index (FMTI) was launched in October 2023 to measure the transparency of foundation model developers.
- The initial version (v1.0) assessed 10 developers based on publicly available information, while the follow-up study (v1.1) involves 14 developers who submitted reports on the 100 transparency indicators.
2.Improvements in transparency:
- The average transparency score increased from 37 out of 100 in v1.0 to 58 out of 100 in v1.1, indicating a 21-point improvement.
- This improvement is largely due to developers disclosing more information during the v1.1 process, with an average of 16.6 indicators having new information that was previously not public.
3.Areas of opacity:
- Persistent and systemic opacity remains in areas such as copyright status, data access, data labor, and downstream impact.
- Developers are least transparent regarding upstream resources required to build foundation models, with a 46% score, compared to 65% on downstream indicators and 61% on model-related indicators.
4.Developer performance:
- Each developer assessed in both v1.0 and v1.1 improved their scores, with an average increase of 19 points. However, the degree of improvement varied significantly among developers.
- Developers can improve transparency by adopting practices from other developers, as at least one developer scored a point on 96 of the 100 indicators.
5.Policy recommendations:
- The findings suggest that transparency in the foundation model ecosystem can be improved and that the FMTI contributes to these improvements.
- Policymakers should consider interventions to mandate minimum transparency levels, especially in areas where little progress has been made, such as external data access and mitigation evaluations.
6.Publication of transparency reports:
- Transparency reports for each developer are published, consolidating the information disclosed during the v1.1 process, to enable further research and provide a resource for stakeholders to apply additional pressure for greater transparency.
The paper concludes that while there is significant room for improvement in terms of transparency in the foundation model ecosystem, the FMTI and other interventions are effective in driving these improvements. Nonetheless, persistent issues remain that require attention from policymakers and stakeholders to ensure accountability and transparency in the development and deployment of foundation models.
AI Advisory Boards 