Assessment:Cultural Assessment

DLF Cultural Assessment Working Group

This sub-group of the DLF Assessment Interest Group (DLF AIG) was formed in February 2016 to discuss ways by which we may assess our digital collections and their cultural impact. Members of the DLF AIG for Cultural Assessment aim to first identify institutional data and practices that may be relevant to building a robust understanding of “cultural assessment.” Then, the group will investigate and attempt to surface underlying assumptions within our data and practices to help the community better understand the social structures that both influence our work and result from it. Ideally, the group will develop helpful and nuanced rubrics for institutional measurement and analysis of cultural biases and assumptions. The DLF AIG Cultural Assessment group intends to raise awareness of cultural bias and institutional “blind spots,” as well as recommend a set of data points, to create more inclusive cultures within DLF member organizations.

We will explore whether and how cultural biases/assumptions are embedded in:

• materials we have available in physical collections - special collections, institutional archives;
• in librarians’ and archivists’ selections of what to digitize;
• in the requests their patrons and communities make for content;
• in choices about levels of digitization and preservation;
• in metadata-creation/descriptive activities;
• and in decisions about how/when/whether we publicize collections and make them discoverable.

with the understanding that biases and assumptions have concrete impact on digital library collections and services.

Mission

To raise awareness of cultural bias and strive for diversity, equity and inclusivity [1] in digital collection practice to create more inclusive cultures and to mitigate or expose collection bias. Where such practice is lacking, create new frameworks that uphold CAWG values.

CAWG Values

Pursuing ethics of care in our various institutions by identifying and assessing cultural biases and assumptions underlying digital collection data and practices. We will...

• Recognize implicit and explicit bias in current archiving and collection practice within Libraries, Archives and Museums (LAM) cultural heritage institutions.
• Seek to uphold practices that reflect the goals of diversity, equity and inclusion
• Encourage higher levels of transparency in digital collection creation processes and facilitate critical user engagement.
• Foster sensitivity in the development of access strategies that strive to involve community stakeholders
• Respect community stakeholders' dignity and their right to privacy while expanding access when appropriate.
• Identify methods for sensitive and respectful representation of collections that strive to involve community stakeholders and respect their dignity and their right to privacy.

Current Activities (in development)

Stay tuned as we plan for 2025!

Past Projects:

2024

Continued reading discussions and published the Inclusive Metadata Toolkit.

Inclusive Metadata Toolkit

Created by the Inclusive Metadata Task Force (IMTF) of the DLF Cultural Assessment Working Group (CAWG), the Inclusive Metadata Toolkit serves as a centralized guide to the range of inclusive metadata tools and resources currently out there, in order to equip practitioners to implement inclusive metadata practices in their day-to-day work.

The toolkit homepage can be accessed on DLF's OSF at https://osf.io/2nmpc/ and consists of two components:

• The Inclusive Metadata Toolkit guide document, which provides context for the listed tools and resources in order to make them easier to use and navigate. This document is static.
• The complete Inclusive Metadata Toolkit Resource Directory, which serves as a sortable and filterable directory of inclusive metadata tools and resources to help you wherever your institution is at. The version in OSF is a static snapshot of a living document.

We hope the Inclusive Metadata Toolkit Resource Directory can continue to change and grow, providing a living directory as more inclusive metadata tools and resources are created and published over time. Additional resources can be suggested through the Inclusive Metadata Toolkit Suggested Resource & Feedback Form. General feedback or questions are also welcome.

2023

Looking forward to 2023, the Cultural Assessment Group will continue to analyze and refine the components of the *Rubric of Reflection* including:

• Inclusive Metadata Framework
• Updating the Annotated Bibliography
• Refining the digitization workflow to address/assess/highlighting cultural bias in digitization processes
• Beginning to interrogate Discovery/UX and how libraries (which may include special collections and/or archives) make their collections discoverable and how they publicize collections

Annotated Bibliography

The Cultural Assessment Group published their first draft of an annotated bibliography reflects research that is helpful for framing the notion of cultural assessment in the cultural heritage sector with a particular focus on digital libraries. Resources include detailed legislation for cultural heritage stewardship, collaborative cross-cultural case studies, and articles focused on the current methods of digital collection creation. We drew much inspiration from the Social Inclusion Audit and the Balanced Value Impact Model among others.

The group is also soliciting feedback and additional resources in the running bibliography, in an attempt to make the resource more broadly applicable (i.e. not solely focused on North America). Any one can add to and comment on the bibliography. We only ask you do NOT DELETE any items.

Ongoing Public Bibliography

The DLF Cultural Assessment Interest Group welcomes feedback and comments on GitHub as well.

2021 - 2022

Continued to analyze and refine the components of the *Rubric of Reflection* including:

• Selection Workflow Framework
• Digitization Selection Framework
• Inclusive Metadata Framework

Outreach Task Force:

Hosted speaker Su Linn Blodgett to discuss Natural Language Processing and the harms associated with language technologies and her work with the Microsoft Research Montreal FATE group. Detail can be found here: https://www.diglib.org/aigs-cultural-assessment-working-group-presents-towards-building-equitable-language-technologies/

Inclusive Metadata Task Force (IMTF):

The Inclusive Metadata Task Force is working toward publishing the Inclusive Metadata Toolkit and is aiming to finalize it in 2024. The Toolkit is meant to provide a breadth of resources to support a wide range of practitioners interested in inclusive metadata, from folks who are just getting started and are looking for language to help with fostering conversations at their own organizations, to those who may be already in the midst of remediation projects, and are looking for more specific examples of workflows and tools for ongoing, iterative metadata maintenance. There has been a lot of work already done on this toolkit over several years and we are looking for more participants to help shape the toolkit to its final form for publication. In particular, we welcome contributors who are interested in sharing case studies and/or example workflows based on experiences within your organization, or who would like to review and provide analysis of published case studies. Additionally, we are currently seeking to increase coverage in two topical areas:

• 1) advocacy for inclusive metadata work within and across organizations
• 2) examples of approaches to DEI-focused metadata audits and assessments into departmental routines and standard workflows

2020

Even for all it's challenges, 2020 for CAWG saw some much needed growth and inter-group connection and collaboration.

Inclusive Metadata Task Force (IMTF):

The IMTF organized our efforts this year around two main threads:

• Discussions. Monthly discussions of a specific topic, resource, or reading. Recent discussion topics have included:

1. Review Draft of Cataloging Code of Ethics—linked from the Cataloguing Ethics Steering Committee site
2. Discussion of the documentary Change the Subject
3. Reading and discussion of statements on bias (gathered by Violet Fox in the Cataloging Lab’s List of Statements on Bias from Libraries and Archives)
4. Reading and discussion: SAC Working Group Report on Alternatives to LCSH "Illegal aliens"

• Toolkit. Ongoing content gathering for a public-facing "Inclusive Metadata" Toolkit. Our goal for the Toolkit is to create a resource that will support folks who want to begin and foster conversations at their own organizations, to promote inclusive metadata practices.

Through our monthly readings/discussions, we are increasing our awareness of some of the most issues and challenges across various organizations, and generating ideas for common challenges and relevant sources we want to include in our Toolkit. Along with these external sources, we are also keeping a running list of questions and challenges that Task Force members run into in their work. This list provides us with ideas for future discussions within our group, as well as potential case studies to explore in the Toolkit.

Digitization Selection Task Force:

The Digitization Selection group is a new group within CAWG this year, created to focus CAWG related efforts on selection as it relates to digitization. The group devoted initial resources to brainstorming possibilities, including discussing digitization and selection criteria. This resulted in numerous opportunities for conversations among group members from a diverse range of professional backgrounds. These conversations lead to several work products, including an initial draft of a visualization and 40 accompanying questions.

Work Products:

• Visualization. For its first project this year, the group created a visualization to outline the different workflows involved in digitization, especially as they related to the process of selection. The goal is to create a visual reference point for our future action items. Once the Digitization Workflow Visualization is completed, the group will share it with the rest of CAWG as so many aspects of selection relate to the other working groups.

• Questions. The group decided to compile 40 questions to accompany the Digitization Workflow Visualization to help illustrate the various aspects of each process of digitization selection. A secondary goal is to gather information about what steps various institutions are taking to make their assessment criteria more inclusive of digitizing collections from traditionally marginalized groups, and to incorporate these findings into the set of questions in order to foster inclusivity when assessing collections for digitization in our own documentation. Towards the end of this year, the group shared the draft questions with the CAWG Inclusive Metadata Task Force for feedback. Next steps include an environmental scan of related publications and digitization assessment policies.

Selection Task Force:

The CAWG Selection Task Force is conducting a survey of national, regional and local arts and humanities organizations whose collections reflect and represent the cultural diversity of their membership. We seek a greater understanding of the practices and policies that organizations use to build community archives or libraries, especially with regards to how materials are selected. The task force is interested in organizations with established collection activities and those that want to begin collecting materials. The goal is to incorporate the survey results into the CAWG Toolkit to be more inclusive of community cultural organizations. In addition, the task force will share the survey results and the toolkit with community organizations to assist their important curation efforts.

2019

This year saw the formation of a new committee and three task forces. Intended outcome of this work is:

• the formation of a group to help manage the outreach efforts of CAWG and to organize the burgeoning 'speaker series' of interest to the members
• draft 2 of the selection survey targeted to 'community groups' to help redefine the Selection Workflow Framework (part 1 of the Rubric of Reflection) to provide entry to all levels of cultural heritage stewardship
• remain on retainer to the UX group as they develop their annotated bibliography/recommendations to make the product as well rounded as possible with consideration for cultural awareness, competencies and sensitivity
• produce an annotated bibliography related to inclusive metadata practices providing a shared understanding to move forward with developing an Inclusive Metadata Framework (part 2 of the Rubric of Reflection)

Task Force Charges

Outreach Committee

Liaise with other like organization and GLAM institutions with direct ties with digital librarianship the creation of digital collections. Develop and maintain a virtual programing ‘clearing house’ for professional development opportunities related to the Cultural Assessment Interest Group Values. Solicit and coordinate the development of a CAWG Speaker Series. Create and test new workflows to sustain this work.

Selection Task Force

Develop a workflow related to the selection process that is articulated in broad strokes so that it may be used across many types of institutions, e.g. academic libraries, museums, historical societies, etc. Outline areas of potential cultural friction in the selection process and potential measures of successful navigation of said friction. These benchmarks will be expressed as a set of activities that accompany/augment the questions developed from previous development of the Selection Workflow Framework.

Personas (UX) Review Board

Collaborate with the DLF AIG User Experience (UX) Working Group to help develop their User Persona Best Practices project by collaborating on UX’s web resources task force through a lens of cultural awareness and sensitivity that informs digital library user personas are made. This work will result in a culturally responsive set of best practices that will be featured as part of the Cultural Assessment Toolkit and the User Experience Working Group’s corpus of resources and tools.

Inclusive Metadata

Collaborate with the AIG Metadata Group to conduct an environmental scan of current metadata practices related to the concerns of cultural and social justice. The group will identify avenues for intervention or augmentation possible in current metadata practice. Ultimately, the work will result in set of guidelines and recommendations for the creation and maintenance of equitable, inclusive, culturally respectful metadata that uphold the values articulated by the Metadata Task Force community.

2018

The Cultural Assessment group is moving away from the 5 subgroups in order to pursue an environmental scan of the use of metrics in Cultural Assessment. The group hopes to develop a set of metrics specifically for use in the cultural heritage sector to evaluate their digital collection cultural affinities and gaps, and how one might apply metrics to practices of digital collection creation (see previous projects for an idea on said practices).

The group will:

• Add Activities as benchmarks to the Selection Workflow Framework (to be the bases for Inclusive Curation and the overall Digital Collections Rubric for Reflection -previously called Cultural Assessment Toolkit)
• Continue to support the User Experience AIG in the development of a Web Resource for ethical Personal Creation
• Develop the collaboration with Metadata AIG to articulate values of Inclusive Metadata practices
• Conduct an environmental scan of 'diversity' related efforts and digital collection creation between 2016 and 2018

2018 Task Force Charges

Environmental Scan Task Force

Perform an environmental scan of current projects and work being done libraries, museums, and archives as it relates to cultural diversity, equity and inclusion since 2016. This will be the first step to determining goals for the CAWG Oct. 2018.

Selection Framework Task Force

Develop a workflow related to the selection process that is articulated in broad strokes so that it may be used across many types of institutions, e.g. academic libraries, museums, historical societies, etc. Outline areas of potential cultural friction in the selection process and potential measures of successful navigation of said friction. These benchmarks will be expressed as a set of activities that accompany/augment the questions developed from previous development of the Selection Workflow Framework.

Inclusive Metadata Task Force

Collaborate with the DLF AIG User Experience (UX) Working Group to help develop their User Persona Best Practices project by collaborating on UX’s persona gathering and evaluation task force, personas best practices guidelines, and web resources task force through a lens of cultural awareness and sensitivity that informs digital library user personas are made. This work will result in a culturally responsive set of best practices that will be featured as part of the Cultural Assessment Toolkit and the User Experience Working Group’s corpus of resources and tools.

Personas Task Force

Collaborate with the AIG Metadata Group to conduct an environmental scan of current metadata practices related to the concerns of cultural and social justice. The group will identify avenues for intervention or augmentation possible in current metadata practice. Ultimately, the work will result in set of guidelines and recommendations for the creation and maintenance of equitable, inclusive, culturally respectful metadata that uphold the values articulated by the Metadata Task Force community. Relevant link: http://dlfmetadataassessment.github.io/entries/about

Details on their charges can be found in the 2018 Project Scope Document.

There are possible collaborations here with several of the DLF Assessment Working Groups including Metadata, User Experience, and Analytics. The group is welcome and encourages collaboration with people and groups who may be impacted by this work. Feel free to reach out if you have ideas. As the year progresses, the Cultural Assessment group will reach out to groups who they see as having an interest and invite them to participate.

Such groups include (running list):

• Internet Archive / "IA/DLF Inclusive Curation Project"
• DLF AIG User Experience (UX) Interest Group
• IMLS Forum Grant recipients - Design for Diversity: An IMLS National Forum at Northeastern University Grant website
• Association of Tribal Archives, Libraries, and Museums association website
• Others

2017

The Cultural Assessment group is moving away from the 5 subgroups in order to pursue an environmental scan of the use of metrics in Cultural Assessment. The group hopes to develop a set of metrics specifically for use in the cultural heritage sector to evaluate their digital collection cultural affinities and gaps, and how one might apply metrics to practices of digital collection creation (see previous projects for an idea on said practices).

Over the year, the group will:

• Create an overarching workflow for cultural heritage digital collection creation
• Contextualize digital collection creation workflows within cultural heritage institution organizational structures
• Develop / articulate personas and narratives that would accompany an eventual Cultural Assessment Toolkit
• Discuss and articulate form and function of the toolkit

There are possible collaborations here with several of the DLF Assessment Working Groups including Metadata, User Experience, and Analytics. The group is welcome and encourages collaboration with people and groups who may be impacted by this work. Feel free to reach out if you have ideas. As the year progresses, the Cultural Assessment group will reach out to groups who they see as having an interest and invite them to participate.

Such groups include (running list):

• Internet Archive / "IA/DLF Inclusive Curation Project"
• DLF AIG User Experience (UX) Interest Group
• IMLS Forum Grant recipients - Design for Diversity: An IMLS National Forum at Northeastern University Grant website
• Association of Tribal Archives, Libraries, and Museums association website
• Others

Selection Workflow for Inclusive Curation

In 2017, the Cultural Assessment group drafted a selection workflow framework that is articulated in broad strokes to be used across many types of institutions, e.g., academic libraries, museums, historical societies. The workflow outlines areas of potential cultural friction in the selection process and creates a supporting bibliography (using the group's 2016 annotated bibliography for reference).

This draft framework document has two purposes:

1. To inform the Internet Archive’s book selection process for its Open Libraries Initiative (a MacArthur Foundation 100%Change semi-finalist project in which DLF has partnered).
2. To serve as a resource on best practices for Digital Library Federation member institutions and the broader digital library community.

The framework aims to define parts of a digital collection selection workflow for public cultural heritage and academic institutions, in alignment with the mission of the DLF Cultural Assessment working group.

Part of the framework included an open survey requesting digital selection workflows (graphic and text description) from information institutions. The number of responding institutions was small (13 total), but substantive results were obtained. The group plans to expand the breadth and scope of the survey in 2018.

Open Survey Results

• Selection and digitization decisions in responding institutions are made by a mix of donors or internal library staff, not by designated selection committees.
• The majority of responding institutions are not relying on codified, published policies that govern selection of cultural materials.
• A majority of responding institutions prioritize decisions based on staff or space capacity, not on collecting priority. Most consider the institution’s collection strengths and importance of the materials prior to accepting new materials.
• A majority of respondents indicate that selection and digitization decisions are made by individuals, not by appointed committees.
• Selection for digitization is prioritized by appraised value, fragility or rarity of materials, and materials that are already part of the institution’s collections.
• Most responding institutions are not actively seeking new materials.
• Collection agendas are dictated by collection development policies that reflect physical collections and do not stand on their own as born-digital or digital-only collection policies.

See the [Wiki files][2] for detailed documentation.

Published the 2017 Annotated Bibliography

2017 Published bibliography can be found here

Special Thanks to the following Authors and Contributors:
Diana Marsh
Hannah Scates Kettler
Kate Joranson
Liz Lorang
Lorrie McAllister
Rachel Trent
Ricky Punzalan
Sara Rubinow
Sharon Farnel
Susan Barrett

2016

The Cultural Assessment Group, in its first year, conducted an environmental scan about the cultural surrounding the creation of digital collection in the cultural heritage sector. The result of the first year of group work is a fully annotated bibliography concentrated on the formation of digital collections, the workflows required to create them, and the cultural responsibilities and consideration surrounding their selection, description, discovery and representation of communities. This is not an exhaustive bibliography, but represents the accomplishment of the first several months of work. The fully annotated bibliography can be found here, with full citation should you wish to use it. Additionally, feel free to add items of interest to the running bibliography here. Demarcate the ones you have added so they are not missed during annotation. Thank you in advance!

Annotated Bibliography

Define Cultural Assessment in collaboration between librarians, archivists and anthropologists in a way that applies to digital collection development and dissemination. The outcome will be an annotated bibliography that will help set the stage to further discussions about cultural and social responses of library digital collections and potential biases in information structuring and management. The project alos entails compiling the bibliographies and initial findings for the other sub-topics and coordinating the eventual creation of the Cultural Assessment Interest Group white-paper.

The Annotated Bibliography Group met biweekly on Mondays 12:00pm Central Time beginning Aug until Nov. 2016 (see working documents)

Metadata & Description Practices

Outline potential measures and standards for metadata and description activities that allow our digital collections to be culturally aware. Metadata, in this context does not only refer to Subject Headings (like those maintained by the Library of Congress) but may also include folksonomies and ontologies. The intent is to uncover ways metadata is currently being produced, identify ways in which one might determine the cultural awareness of metadata and descriptions (if possible), and by pointing to work that intends to expand the boundary of metadata, like the Mukurtu Project (http://mukurtu.org/learn/), develop a set of recommendations on how to generate metadata that is culturally responsible.

The Metadata Group met bi-weekly on Monday's at 11:00am CDT beginning on Aug until Nov. 2016 (see working documents)

Selection & Digitization

Investigate how institutions of higher education select materials for digitization and how they are prioritized. We will research metrics and decision-making workflows as well as the underlying assumptions regarding selection and decision-making throughout the beginning stages of the digitization process.

The Selection & Digitization Group met biweekly on Mondays at 12:00pm Central Time beginning Aug. until Nov. 2016 (see working documents)

Levels of Digitization and Preservation

Investigate how libraries (which may include archives) determine levels of digitization and preservation for digital collections. This may include a review of how groups are formed to make these decisions, how decision-making groups make determinations about levels of digitization and preservation, and how these decisions are communicated. We will think about how these decisions may be evaluated for cultural factors and influences and/or what impact these decisions may have on various communities.

The Digitization & Preservation Group met bi-weekly on Mondays at 11:00am Central Time /beginning Aug. until Nov. 2016 (see working documents)

Publicizing Collection and Discoverability

Investigate how libraries (which may include special collections and/or archives) make their collections discoverable and how they publicize collections. This may include how to identify constituents, stakeholders and customers; to craft custom communications; to apply descriptive metadata for searching and sharing in an effort to conquer the digital divide. Publicizing and discoverability should also include considerations for selecting discovery platforms and tools, defining methods to present collections, and user interactions with collections, and “usability”. Defining the standards for community demographics, marginalized and underrepresented groups, and representation of diverse communities within the collections is needed for qualitative and quantitative assessment.

The Publicizing and Discoverability Group meets bi-weekly on Tuesdays at 11:00am Central Time beginning Aug. 16 (see working documents)

Get Involved

CAWG All Hands Monthly Meetings

CAWG meets on the second Monday of each month at 2 pm ET.

Contact alexandra.provo at nyu.edu to learn more or join the Google Group for meeting announcements.

Joining the CAWG Google Group is the best way to get announcements about meetings and other activities.

The CAWG also has a publicly accessible calendar. Feel free to share relevant events and invites. DLF AIG - CAWG Calendar

---

Back to the Assessment home page