Workramp is an online learning management system that enables company employee and customer learning. It aids in learnings anywhere from company education to personalized career development.
As an ongoing effort to understand our users’ pain points and needs, Workramp has a VIP Slack channel and feedback loops from customers to the product team that we evaluate every week. One of the highly noted pain points speaks directly to the difficulties that managers have when attempting to review their team’s learning performance. Therefore, in this project, we were tasked with designing a new team review experience that will not only answer the ongoing pain points (ie: not enough data) but also expand on its use case as the company migrates into a new product direction.
The Problem
All managers (if not most) are not admins of the account, but for a long time, the “My Team” performance page for managers to check their team’s learning performance was hidden behind the admin portal. This made it extremely difficult for managers to check their team’s performance at any given time. The alternative, however, was to either gain full admin access or request a data export from their admins.
The VIP Slack channel data (~33% of the comments) showed that admins were reporting disarray within the account due to large volumes of unwanted access, most of which are granted for managers to maneuver the performance page. On the other hand, we also received feedback from our internal customer-facing teams that many admins and managers have mentioned similar issues.
In addition to the accessibility issue, the overall user experience was simply outdated (Figure 1). Managers have mentioned the data not give them the full picture of what is needed to evaluate the team’s learning status. another team). As I further audited the user experience, I also discovered the lack of urgency to action items as well as the inconvenience when attempting to navigate between learners and content statuses. As it turned out, the long collection of action items was mainly due to actions for training assigned by other teams mixing in with required actions (ie: to grade “Product Test - 2024”).
Figure 1: Outdated manager team performance page
All in all, the problem boils down to:
Managers do not have an efficient way to monitor and evaluate their team’s learning performance
Managers stopped caring about action items as it became too cluttered
Workload for admins has increased due to manager requests
How might we ...
Provide quality insights for managers to better evaluate their team’s learning?
Assist managers to complete the necessary tasks on hand?
Reduce the need for admin access and/or request so that admins can refocus their priorities?
Business Goals
We help reduce friction between managers and admins and ensure managers are getting the insights they need to evaluate team performance. The business goal is actually much more than that. As a business, we also want to make sure managers are using this feature as it was truly intended. We wanted to make sure managers cared about the action items that are listed instead of choosing to ignore them due to irrelevance.
Goals:
Increase feature usage percentage compared to the previous “My Team” version
Reduce the number of active action items
Reduce the number of admin access
Reduce the number of at-risk accounts - about 32% of our total accounts
Business Challenges
Now that we have a stronger sense of what needs to happen, we did a high-level feasibility discussion to understand roughly what it will take to make this project happen. Some challenges that we noted:
Large data structure support
Data customization
Building on top of an old codebase
Competitive Research
Before diving into any concepts or solutions, I researched some of our top competitors and how they designed their version of manager dashboards or overall data tracking.
Figure 2: Competitive analysis of top competitors, Docebo, 360 Learning, and Articulate, on dashboard/metric design
User Goals
On the flip side, the user goals are much simpler. We have two types of managers here:
Managers with only one direct team of ICs (individual contributors) - 1 Level Manager (1LM)
Managers who manage other managers who then have their own direct team of ICs - Manager of managers (MoM)
Although 1LM and MoM are different roles, they do have very similar goals, most of which are the same. One high-level goal they have in common is that they want to get a deeper perspective on how their team(s) are doing.
Common Goals (1LM + MoM):
Get insights on team(s) learning performance (ie: completion percentage and number of overdue/failed)
Easily clear out the time-sensitive action items (ie: I need to grade this training so that it can be marked as complete)
Manager of Manager Goals:
Get insights on independent teams and compare (ie: Product has completed 10% more than Engineering)
By answering to the manager’s pain points on performance insights, admins by default will no longer need to take extra time in their day to help pull reports for managers. They also won’t need to grant additional admin accesses, maintaining the peace and organization of the platform.
User Research
As we have identified the major pain points managers have (and admins), I went ahead. I had my internal customer-facing teams (account executives and customer success) connect me with users who have mentioned similar issues so I could dig deeper. I scheduled five 30-minute interviews with them and discussed some of the points here:
Tell me about how managers currently evaluate the team’s learning performance. What is the process?
How does this (managers with admin access) affect admins’ day-to-day? What is the process?
What is some feedback on the (old) “My Teams” page? What is missing? What is helpful?
“I have to make my report because I can’t find the stats that I need on My Teams (old performance page).”
“I don’t have time to figure out what I need to do or see so I don’t even bother with it.”
“A lot of admin access were granted. I just don’t have the time to answer to every report requests. But now the account has too many people on it. It’s getting mess and I can’t tell who made changes.”
Personas
LiRen Zhu , 32
Customer Success Manager
Manager Type
A 1 level manager where I manage 5 individual contributors as my direct reports.
Objectives and goals
Check my team’s performances
Easily identify time-sensitive tasks (ie: compliance training is due but only 50% of my team completed)
Quick to take action
Frustrations
Not enough information to gauge team performance
Spending too much time figuring out how to use the platform
Spending too much time waiting for requested reports
Delays
Too reliant on admins
Dan Smith, 45
VP of Engineering
Manager Type
A multi-level manager where I have 4 direct reports and a total of 138 reports.
Objectives and goals
Check my team’s overall performance and compare
Easily identify which team is falling behind
Immediately identify the manager that needs to nudge the team that needs to pick up the slack
Frustrations
Not enough time to gather data on multiple teams
Unable to see my full team structure from top to bottom
Rely on admins heavily to help pull multiple reports and give me a summary of what I need
Spending too much time conversing with admins on requirements
Product User Challenges
After consolidating all the pain points and benefits from the user interviews, here are the user challenges when using the old “My Teams” performance page:
Managers find it quite difficult to get valuable insights from the page due to the lack of necessary data points (ie: total # of overdue training in my team)
Managers feel overwhelmed when attempting to review team performance due to the amount of steps required and the difficulty in figuring out what is urgent
About 2 to 4 hours are taken away from admins’ and manager's’ day-to-day tasks
Current User Flow
Using the old performance page “My Teams”, you can see that the user has to go through several steps and thought process before reaching to the final goal of obtaining data to evaluate the team’s learning performance.
Figure 3: Understanding the current pain points of a user getting the needed team data
Current User Journey Map
Let’s step into the shoes of my customer success manager, LiRen Zhu, to see the pain points of the old team performance page and why we are looking to change all of that.
Figure 4: Understanding what the user is trying to achieve and the specific issues that come along
Feature and Functionalities
To solve the user needs, here is what must be done:
Enhance data structure and new data points to provide stronger insights to the overall team performance
Proposed User Flow
Allow searchability so that managers can quickly and confidently look for what they need with ease
Design a self-served experience so that managers do not require additional help
Unique Features:
Team funnel structure that allows managers of managers to view their teams from the top down.
With the new proposed experience, the goal is to make it so that users can easily navigate the platform to obtain the data they need to conduct a team learning evaluation. On top of that, we want to encourage managers to utilize this tool to complete any learning responsibilities that a manager must do (ie: grade an assignment). Specifically for our manager of managers (MoM), we want to allow them the ability to have a top-down view of all the teams that report under them.
Figure 5: The new proposed user flow, eliminating the need to request additional assistance
Initial Design Explorations
Since the main dashboard page is where the bread and butter is, my thinking is that if I have the high-level direction down on the home page then it will help dictate how other components may come to be (ie: MoM funnel, visuals, stats, action items, searchability, and more).
User Feedback
Figure 6: The top 3 most well liked dashboard wireframes
As the wireframes became more prominent in detail, the need for certain components became more clear. I reconnected with my internal teams (sales and customer success) to help me set up interviews for various accounts. They didn’t have to be new or already interviewed members but fresh eyes are always good!
We successfully scheduled with 8 different accounts, speaking with 10 managers and 6 admins. Each session was about 30 minutes, discussing the initial design direction at a high level and whether or not they think the selected data was enough for managers to conduct a team evaluation. Using the main dashboard page as the interview discussion, here are some comments worth noting:
“If I filter for specific trainings, will the stats change? How will I see that?”
“The visuals are very helpful, would like to see how you would set the data up for those.”
“Amazing start, but how about managers who have hundreds of reports, how will you show that?”
Components Explorations
Even tho everyone gave really positive comments, it is clear what some of the primary components will be when designing for the entire team performance experience.
Table
Wireframes
Figure 7a: A table wireframe utilizing the matrix style
Figure 8: A simple table wireframe
Hi-fidelities
Figure 7b: High fidelity iteration of figure 7a. A lot more clutter than expected
Action Items
Wireframes
Figure 8b: High fidelity iteration of figure 8a. The table seemed to work fair for a single person
Figure 9a: Action items section wireframe with filter options
Hi-fidelities
Figure 9b: High fidelity of the action items with all training status as filters
MoM Funnel
Structure Mapping
Figure 10a: Mapping out the funnel to see if its feasibility
Hi-fidelities
Figure 10b: Although it looks good for a few layers, the complexity of too many layers is still unresolved
Visuals/Graphs
Wireframes
Figure 11: Wireframing bar graphs of all direct reports and active training statuses
Figure 12: Wireframing clickable bar graph
Figure 13: Wireframing a single bar that represents all the training statuses
Figure 14: Wireframing a pie chart of the training statuses
Review > Iterate > Review
Figure 15: A snapshot of the many iterations that was created throughout this project
The Prototypes
After many iterations and team reviews, it is now ready to be prototyped, ready for our full usability testing phase.
1 level manager walkthrough
The biggest difference between 1-level manager and manager of managers is that MoM may have an ‘infinite’ layer of teams that reports under a single person so it was important that we keep in mind how to structure this funnel without overloading our platform. As you can see below, MoM have an extra toggle under “Team Performance” that allows them to switch between a direct team video and a team roll up view, giving managers the flexible to investigate as pleased. In addition, each layer renders a new table. The reason behind this is that as managers evaluate each team, it is rarely that they will need to evaluate multiple teams simultaneously so keeping it simple and show one layer at a time.
The only difference with manager of managers is that they get the team roll up vs direct report view (funnel vs no funnel view).
Usability Testing
Seeking my internal teams again to connect with users, I gathered 10 accounts (12 managers and 3 admins) from previously spoken ones and conducted more 30-minute interviews but this time, giving users the space and freedom to just go through the prototype. Unfortunately I am unable to show videos or screenshots of the interviews but here are some highlighting comments worth noting:
“Oh, I did not think you can use the visual bar as a filter. That’s a bit odd.”
A clip of how the visual bar works - clickable filter
“I like how the team funnel is simple. But the visual bar is more confusing than helpful. Seems a bit distracting.”
“The legend (I assume) of the bar doesn’t match up with the bar itself. I like how the action items are so shortened. Not my favorite that it’s on top but much cleaner.”
Final Designs
As we come to the end of the Manager Dashboard MVP, it is clear that the visual bar added more friction to the users than was helpful.
Let’s revisit the initial user goals that we have discussed:
Managers want to have easy access to team data so they can check team performance at will
On top of that, manager of managers want to have a top-down funnel on all members that branch out from him/her
Easy check-in on specific trainings like compliance training that are time-sensitive
Remove admins from the equation
Data points that are useful for evaluation
Did we accomplish this?
In short, the answer is Yes. Let’s take a look:
Managers no longer need admin ‘permission’ to get access to team performance data/reporting because the new designs place the dashboard outside the admin portal
Admins are no longer needed, gaining their hours back
Managers can utilize the “Filter Trainings” button to filter through the team stats to only show the selected trainings
It added transparency for manager of managers to get a top-down funnel view.
New data points are added based on user feedback (ie: overdue and failed assignments being the most important to quickly recognize)
Team Performance Home (default)
Team Performance Home (action items expanded)
Filter Training Dropdown
Filtered Training
Training Performance
Path Training (scolled - multiple courses in one)
Individual Performance
Individual - Badge Modal (accomplishments)
Engineering Handoff
In short, the new design allows the manager to do their self-service team performance review with ease, anytime desired. With the designs wrapped up, it was time for an end-to-end engineering hand off. I created a handoff sync reiterating the goal of this project and what we are solving; more importantly, noting all the before/after that we will be changing.
Figure 16: A snapshot of the engineering handoff presentation deck
Launch and Metrics
As discussed earlier, the business goal of this project is beyond just ensuring managers can get the insights they need more so, we want to encourage managers to review their responsibilities (action items) and get them done. Translating all of this into metrics, we get:
Increase feature usage percentage compared to the previous “My Team” version
Reduce the number of active action items
Reduce the number of admin access
Reduce the number of at-risk accounts - about 32% of our total accounts
Launch Plan:
We launched the feature at a 10-50-100 rollout starting with the accounts that had the most reported pain points of this topic and then the greater 50% and eventually 100%.
Results
As we continued to monitor the feature and plan for the next versions, we did receive tremendous comments and usage on the experience.
Results based on the month following the full 100% release:
23% of our total accounts (71.8% at-risk accounts saved) renewed their contracts
Saved close to $1 million on those contracts
About 30% decrease in the number of active action items
The number of admin accesses (to non-admins) decreased by about 50% and increasing
Increased 20% more engagement than “My Team” (old)
Next Steps
Of course, as we continue to monitor the feature and gather insights from more users, more designs and ideas will come up. But I also did not forget there was a high number of requests for visuals. This is something that I plan to work on with my engineers moving forward to see what we can do based on our current component library and available data.
Unfortunately, I was not able to continue on this journey but implementing visuals would be my next big priority when it comes to enhancements to the manager dashboard. A big part of this discussion would include having discussions with the data analytics team and figuring out optimal ways to combine various data points to provide new insights for our users. In addition, I would like to better organize the action items, in a way that dives even deeper for managers to understand if it’s an action for training that I assigned or training from another team.
Another thing that needs attention is the assignment status, failed. At the time of the MVP, we could not figure out what happens to the manager when an assignment is failed. The thought, however, was to simply reassign the training for the learner and put the assignment back in the queue. Without the pressure to hurry the new dashboard, my engineers and I would then have more time to design a path for failed assignments.
What I Learned…
This was a very large-scale project and the process, although came with lots of headaches, was definitely an exciting one. The biggest takeaway that I learned from designing such a large-scaled project was having a more in-depth feasibility discussion with engineers shortly after the wireframing phase so that we could dig into any potential ideas. There were unforeseen issues that came up from engineering that I simply did not expect such as the inability to embed tables within tables but tables within an accordion is okay. The realization of time that it took for engineers to build certain simple components was unexpected as well because of the need to rebuild or build on top of old codebase. The short story here is, to ask more clarifying questions and not just take an engineer’s “yes, we can do that” as a signal that it will take a short time. Yes does not mean short, it just means yes.