Documentation can help bridge the gap between ethical AI principles and ethical AI in practice. By fully documenting machine learning systems throughout the development and deployment lifecycle, practitioners can better anticipate potential issues before they arise and create the artifacts needed to make these systems transparent. With our ABOUT ML (Annotation and Benchmarking on Understanding and Transparency of Machine Learning Lifecycles) initiative, Partnership on AI (PAI) has been working to coordinate efforts, bringing voices across the AI community together to develop and test best practices for the documentation of machine learning systems. With the completion of our first ABOUT ML Pilot, this initiative has entered an exciting new phase — one that attendees learned more about at ABOUT ML’s most recent quarterly meeting.

READ THE ABOUT ML PILOT SUMMARY

Designed to further improve our recommendations and resources by testing them in practice, our Pilots are real-world collaborations between PAI and organizations that seek to improve their processes. For our first ABOUT ML Pilot, PAI worked with a European startup building an AI product designed for use in the medical space. Together, we identified several opportunities to make the startup’s already extensive documentation framework even more effective in practice:

• Building a shared understanding on why documentation is needed
• Clarifying the intended audience and their needs
• Communicating and coordinating internal company processes

Our First ABOUT ML Pilot

Previously, PAI has published the ABOUT ML Resources Library, a library of resources designed to help organizations and individuals begin implementing transparency at scale. PAI is now putting these guidelines and resources into practice by conducting Pilots with a select group of organizations who are looking to improve their documentation approaches across the ML lifecycle. This work was shaped by our advisory expert group of Kasia Chmielinski, Michael Hind, Meg Mitchell, Kush Varshney, Jenn Wortman Vaughan, and Hanna Wallach, who provided guidance, support, and feedback on implementing ABOUT ML Pilots.

Our first ABOUT ML Pilot was with an Ireland-based startup building an AI product that collects drug safety information from medical journal articles. Prior to launching their first product, the team had developed an extensive set of documentation artifacts. The team put in significant effort building these documents from the ground up, using PAI’s ABOUT ML resources and other references as guides. Even while following these guidelines meticulously, however, the team still had many questions concerning what and why they were documenting and for whom. Here, PAI saw an opportunity to help the team more clearly identify the answers to these questions.

During the Pilot, the PAI team and the organization convened remotely for a series of biweekly workshops exploring current documentation processes and audience needs through exercises and facilitated discussion. Participants also completed an online diary study each week to share updates and capture insights. Additionally, participants completed an online survey at three points to reflect on the value of documentation and assess their own internal documentation processes and artifacts.

By thinking about the why, how, and who of ML documentation together, PAI believed we could change how the team thought about documentation and how they might implement it in their day-to-day work. We collected findings from this experience in the Pilot Summary “ABOUT ML in Practice: An Example From the Pharmacovigilance Field.” This Pilot Summary is part of the ABOUT ML Resources Library where it can serve as an additional resource for organizations that want to improve their documentation practices.

ABOUT ML’s Q3 2022 Meeting

At ABOUT ML’s most recent quarterly meeting, attendees heard from the startup’s CTO, Bruno Ohana, who spoke about the Pilot experience and what it’s like working at a small company that’s pursuing robust documentation while also working to release its first product.

Ohana described the process of creating specification that answered what and how the startup wanted to document and for whom. “I​​t was important to write it down and have a better picture of what we wanted to achieve,” he said. “Since then we’ve made a lot of progress, and it’s really great to see that, little by little, the body documents that we produce are more targeted and more relevant.”

Ohana also shared key learnings that came out of the Pilot and what’s next for his organization. He emphasized that documentation should be an active, adaptive process, that engagement models drive documentation requirements, and the importance of considering user perspectives as a starting point for adoption.

“It’s not just about having a manual that we can refer users to,” said Ohana. “We really need to bring users into this journey of taking advantage of AI, and what are the risks and […] how to bring in this new technology in a way that is safe and is going to be beneficial.”

Looking Forward

After the contributions of so many to develop and iterate upon PAI’s ABOUT ML resources, the first ABOUT ML Pilot represents an important milestone in the life of this initiative. In line with ABOUT ML’s iterative approach to developing highly usable guidance, feedback from this Pilot will inform future collaborations between PAI and others on ML documentation in practice.

PAI is eager to share our learnings with the wider AI stakeholder community and help additional organizations determine how they might improve their processes around documentation. As part of this effort, the ABOUT ML team recently presented a paper on the key challenges to ML documentation (and strategies for addressing them) at ACM’s conference on Equity and Access in Algorithms, Mechanisms, and Optimization.

As this work continues to evolve, we welcome your thoughts on opportunities to test documentation best practices. To learn more about participating in a future ABOUT ML Pilot, please fill out this Expression of Interest form or email jiyoo@partnershiponai.org.