Alloy I 2019
Assessment of Data Catalog Taxonomy for Findability
Our client’s core data management system suffered from low functional usage. Specifically, users were struggling to effectively use existing tags to search and sort data. This was fueled by an ambiguous mix of system-generated and employee-created labels, which created severe Information Architecture (IA) issues.
I led the end-to-end UX research and strategic evaluation to diagnose the root causes. My focus was on quantifying user difficulty with the existing structure and translating the ambiguous labeling problem into a path towards improved findability and data accuracy.
This case study showcases my expertise in complex discovery, stakeholder alignment, and the translation of functional usage gaps into a tangible, actionable strategy. My work directly informed the requirements and direction for the subsequent IA and taxonomy implementation phase.
UX Strategist
I served as Lead UX Researcher and Evaluator, owning the evaluation, diagnosis, and validation of the IA/taxonomy.
Collaboration
I partnered with a UI/UX Designer to conduct the usability evaluations, and a UI Developer to build out the interactive prototype.
01. Summary
Challenge
A global consulting firm's data management system was suffering from labeling and organizing issues that made data difficult to find and use. This was evidenced by minimal utilization of core tagging and search functionalities. The problem was rooted in an ambiguous data structure: a complicated mix of system-generated and employee-created labels. Users frequently applied inaccurate labels, causing cascading issues in search accuracy and compromising overall data findability across tax and audit divisions. This lack of label organization presented a major hurdle to efficient analysis and decision-making for a critical, high-volume user base.
Opportunity
HMW support users of all levels across tax and audit divisions to effortlessly access data, improving accuracy and findability to support their tasks and increase productivity?
Approach
Research Activities:
Open and Closed Card Sorting: We facilitated card sorting exercises with users to understand how they categorized data and perceived the current label options.
User Testing: We conducted user testing to observe in-system behavior and evaluate the clarity and effectiveness of the label list.
Research Goals:
Evaluate Label Viability: Assess the effectiveness of the existing label list in supporting user tagging behavior.
Uncover User Mental Models: Identify the factors and considerations that influence user decisions when assigning labels to data assets.
Refine Label System: Develop recommendations to improve the label list for enhanced discoverability and tagging accuracy.
Impact
Enhanced Partnership: The project fostered a stronger partnership between Alloy's product team and the client's data management team. This collaboration aimed to improve the platform's search function, directly addressing user frustrations identified in the research.
Avoided Costly Mistakes: By validating that label tagging was valued and desired by users, we proved the need for this feature and that optimizing it rather than retiring it was the right step. We also steered leadership away from potential development costs associated with rehauling the search engine without building out labels that would resonate with users.
Users for Improvement: User participation in our label testing was positive and appreciated. Users actively volunteered to consult on further label list expansion, eager to contribute their unique needs, citing niche use cases, to the process. They also connected us to other opportune individuals, supporting our recruitment process and getting us buy-in.
02. Research
Label Validation Study
Study Goals
Identify trends in list classification
Identify pain points or content gaps related to Line of Service (LoS)
Address recommendations for next steps regarding list creation and user needs
Select Hypotheses
Common groupings of labels will emerge that align across the LoS and level of experience of users
Users will advocate for and/or prefer the option to create & add their own labels
User Testing
Our user participant criteria was primarily focused on three specific types of Lines of Service (LoS) across tax and audit divisions, and comprised of both junior and senior members of the firm. We conducted 60 minute testing sessions with a total of 9 users.
The first exercise, a Card Sort, was 2-part activity: the first stage was a committed guidance sort, followed by a no-guided sort. Users were first tasked with sorting a refined list of 70 active labels into categories of their choosing. The no-guidance stage of the exercise was an opportunity to learn how LoS and level of experience factor into the category delineations of existing labels generated from first stage Open Card Sort.
The last exercise was an interactive test where users were prompted to self-generate labels, which would be pulled from the full list of labels. This “live label” test was conducted through an interactive prototype built in-house.
Findings
Our user research and testing proved that users found the data tagging feature helpful. We validated that users did desire and value the functionality; they just were not using it. This communicated to my team that the problem was not inherent to the feature, but rooted in the labels themselves: the quantity of labels, the relevancy of the available labels, the lack of label categorization and definition, and more.
77% of users found tagging helpful and said they would use tags
77% of users wanted to apply 3-5 tags per asset minimum
44% of users found the labels lacking key elements in their LoS or did not find the list useful for their LoS
What we learned
Users saw the value in using labels, but the way labels were applied to overlapping categories created confusion and made it difficult for them to effectively use the system.
ALL users wanted more granularity in the labels
ALL users confirmed that categories would aid in their search and labelling process
What we found opportune
Clarity: Broad terms should be used as categories; for labels, the more specific the better
Findability: Potentially have views based on LoS/Engagement with View All option
Usable: Utilize ML for suggestions and backend filtering
Learnable: Consider Color-coded tags by category to help define broader or cross-over terms
What we recommend
03. Conclusion
Impact
The user research findings and recommendations from our work were highly valued by the client. This resulted in several positive outcomes:
Expanded Investigation Approval: Our initial research sparked client interest, leading to approval for further investigation of user tagging behavior. This allowed us to develop a deeper understanding of user needs and begin to test our recommendations before development builds.
Enhanced Partnership: The project fostered a stronger partnership between Alloy's product team and the client's data management team. This collaboration aimed to improve the platform's search function, directly addressing user frustrations identified in the research.
Actionable Recommendations: Client responsiveness translated into concrete actions. The client assigned a dedicated data librarian to work with our team on expanding the label list based on our findings. Additionally, they planned an internal audit of their machine learning capabilities, potentially paving the way for further improvement based on user data.
Next Steps
To ensure a smooth transition for the expanded effort, I undertook the following steps:
Knowledge Transfer: I prepared a comprehensive documentation brief for the Alloy product team members assuming responsibility for the next phase. This document covered the full scope of research and testing conducted, along with our backlog and recommendations for moving forward.
Project Handover: With the documentation in place, I effectively transitioned ownership of the project to the new team members within Alloy. This ensured continuity and facilitated a successful continuation of the research initiative.