Dates: Spring - summer 2016
My Role: UX Lead, User Researcher
Disciplines involved: User Research & Testing, Interaction Design, UX Management & Leadership
Deliverables: Test plan, Wireframes, Prototype for testing, Test results
CONTEXT
This case study illustrates how I led the effort to incorporate a round of user testing during the design phase of a site redesign. My design team was new to conducting user testing, so I used the opportunity to teach my team how to do it.
THE PROBLEM
Leadership Academy is an internal IBM website that provides training to IBM employees, managers and executives. The site also provides search and sign-up for classes, and provides assistance in preparing for annual reviews.
The website had recently undergone a redesign, which the team in charge of the site wasn’t happy with it. Some of the issues they had with the site were:
The site was hard to navigate
People were only visiting the site when required to for their annual review, and at no other time. The client wanted to make the site more relevant to users throughout the year
Users had difficulty determining what classes and topics were the best choices to advance their careers
The site had no impact, and lacked boldness
Search needed to be improved. Users were especially having difficulty with using the search interface.
There were other sites on the corporate intranet competing for employees’ attention from an education standpoint. The Leadership Academy team wanted their site to stand out both from a content and design perspective.
In addition to fixing the issues with the existing site, the client also wanted:
Introduction of the concept of personalization to the site, in which a user could enter information about themselves, and the site would populate personalized content
User research would be a part of the design process in some fashion.
My task was to prepare for an upcoming workshop. The goal of the workshop would be:
Discuss the issues the client had with the current site
Present my plan for incorporating user research
Present an overall project plan that showed how user research would be incorporated
Decide on key screens and features for the first MVP.
THE SOLUTION
Preparing for the workshop
For the initial site design, the client had side variety of documentation such as personas, traffic metrics, and surveys. I spent time studying the site and analyzing the documentation.
The client wanted to conduct a survey of the current experience with existing clients as part of the workshop preparation. She would be responsible for creating and administering this, but I would help out with the wording of the questions.
Recommending an approach to user testing
I realized that I was the only member of my team who had experience with designing and conducting moderated tests. If we were to incorporate testing, I would need to guide team members through the incorporation of user research into the design process.
After reviewing client materials and some initial discussions with the client and team, I chose to focus the design process on the creation of limited prototypes for user testing. Key components of this test would be:
Testing would be done during the design stage, to ensure that there would be enough time to implement any major changes
The focus would be on getting quick, actionable results as opposed to long test plans and extensive reporting
The aim of the test would be to get qualitative feedback via 5-8 moderated, remote, one-on-one user interviews
The client would be responsible for all recruiting and scheduling of participants. This would be critical because our team did not have the capacity to handle recruiting. Thankfully there was someone on the client’s team that was happy to help with this task.
Recommending a project approach for design and user testing
In addition to recommending how the testing would be done, I also had to think about how where it fit into the design phase. The high-level approach would be:
Take an initial stab at the test plan. This was to make sure everyone was on the same page regarding what we were working towards. I would start on this, then come back to fill it out further later in the project.
Create designs for a prototype in InVision of a redesigned site using key screens identified from the workshop. I would start with wireframes and iterate on them with our visual designer. Then, a Jr UX person would create the interactive prototype in InVision under my supervision.
While the prototype was being created, I would then peel off and focus on flushing out the test plan further with the client.
Conducting the test and delivering the results
Implement changes from the results, and creating visual comps for the screens to be delivered to development.
Getting buy-in at the client workshop
At the workshop, I presented the project approach to client. I explained how the test would work, benefits of utilizing this type of testing, and how it would be incorporated into the project. I walked the client team through the test plan template to give them an idea of what to expect.
We chose key prototype screens for the initial preferences setup and the personalized home page. We would also test key concepts of the site design, such as the Leadership Journey.
In addition, the client presented the results from the survey I worked on with her. The results basically confirmed what we already knew.
Post-workshop tasks
After the workshop, I delivered a week-by-week schedule of tasks, goals and role responsibilities.
Starting WIREFRAMES
I started by creating high-level wireframes of key screens for the prototype. These were based off of the whiteboard sketches that were done at the workshop.
WORKING IN PARALLEL WITH VISUAL DESIGN
While I was working on the wireframes, the visual designer began working with the client on branding and the look & feel. I staggered by work in batches, focusing on the site nav and home page first. These were passed on to the visual design once they were good enough, who in turn got started on exploring branding and other visual options. While he was working on the site nav and home page, I began working on the remaining sites.
Working out remaining wires and incorporate visual design work into sketch file
Once a basic look-and-feel was worked out, the wireframe document was updated to incorporate higher-fidelity designs and annotations. I used the visual designer's designs for the wireframes where possible. The purpose of this stage was to achieve client sign-off on site designs in order to move forward to creation of a prototype that could be tested.
I presented my wireframes to the client, and after some updates, they were delivered to the Visual Designer and Jr. UX Architect for prototype creation. Some of these include image work that was done alongside the Visual Designer.
Unfortunately, Squarespace doesn’t have very good options for zooming in on images. I have included my final wireframes in PDF format for convenience.
Test planning
Once the designs for the prototype were approved, I began detailing the test plan, serving as an ad-hoc representative from the client side. We created a detailed schedule with tasks for each week in order to make clear who was responsible for what.
PROTOTYPE
While the test plan was being worked out, I simultaneously initiated the effort to create the prototype for testing. I used InVision for prototype creation, coordinating the efforts of a junior UX Architect on our team and the visual designer in building out the prototype.
Select screens from the InVision prototype:
RECRUITING & SCHEDULING
While the prototype was being created, I split my time between supporting that effort and assisting the client with recruiting and scheduling participants for the test. This process included:
Creation of the screener with the client
Providing dates and times to the client for scheduling
MODERATOR GUIDE
As the test approached, I created a moderator guide. Since there were going to be several moderators, I saw this sub-project as a way to ensure everyone was on the same page when moderating the test.
PILOT TEST
Once the moderator guide and prototype was built, I had the team conduct a pilot test. I learned from previous experience that it’s necessary to do these in order to stress-test both the prototype and the moderator guide. In addition, it let the UX team members who didn’t have experience with moderation a chance to stretch their wings. We used members of our design team as participants for this.
After going through some pilot tests, and making adjustments to both the moderator guide and pilot test, it was time to conduct the test.
CONDUCTing the TEST
Eight sessions lasting around 45 minutes each were conducted. While a moderator was conducting a test, we had at least two observers available to take notes. I scheduled at least 30 minutes between each test session so the observers could discuss what they saw immediately after the test, as opposed to later after all the sessions were completed.
Screen grab of one of the user test video sessions
RESULTS
The day after testing was completed, the moderators met and discussed the overall results. As we were doing so, we filled out the Rainbow Spreadsheet accordingly.
The following day, we met with the client and discussed our findings and recommendations for improvement.
Images from the Rainbow Spreadsheet
FINDINGS
As expected, there were shortfalls in the test findings:
Some users had issues filling out the profile setup form
Self-identifying using the list of choices provided was difficult for many
Nobody was particularly dazzled by the home page design, with reactions ranging from “meh” to negative
The My Leadership Journey arc was problematic, as participants had problems understanding the difference between “My Leadership Journey” and “My Leadership Development”
Higher-level executives and especially aspiring managers didn’t find the site particularly relevant to them
Issues, for certain. But uncovering these issues—and crafting real solutions—was the fundamental purpose of the test.
Final design comps
Once testing was completed, the design comps were updated to reflect changes from the test results.
Here are some selected design comps created by the Visual Designer:
Outcomes
In the end, what began as a rudimentary, unengaging site was transformed into something that had much more real-world impact on those who visited. The redesign offered giant leaps in functionality and relevance that simply didn’t exist upon initial launch. Though the new version was not perfect, it exposed gaps and incongruities through testing that could be addressed and creatively resolved—an ongoing process that continues to this day.
Unfortunately, after doing all of this great work, the site redesign didn’t launch. The project sponsor, who was very involved in the visual design and had specific demands for it, ended up moving to another department. A new manager took his place, and hated it - especially the visual design.
However, the project was an excellent opportunity to mentor my team on how to incorporate user testing into the design process. They obtained a better understanding of what is involved in doing small-scale qualitative user testing, and what you can (and can’t) get from it.