User testing is an important phase in the designing process, but some find it a waste of time and money. Find out why skipping on user testing could cost more than you think.

What is User Testing?

User testing is one of the most valuable tools to ensure product expectations are met for both stakeholders and end users. It consist of observing the user experiences a customer will have with a product, prototype, or feature. This can include their responses, feelings, perceptions, and actions in response to using a product from when they start to when they stop.

At KRUTSCH, we believe in the importance of user testing. It's a cornerstone of our process and of user experience design, or UX. For both an existing and new product, we will user test a revised version of the product that has gone through our design process. We will take an interactive prototype of the product and administer guided walk-throughs of potential workflows end users will encounter. Pain points, issues, and successes are recorded in a summary of findings. We're generally surprised by at least one finding produced during user testing, and will integrate it into further iterations of the design. It's what we do.

Why Some Skip User Testing?

Although user testing is a valuable investment in product creation or redesign, some clients may choose to forgo it. Reasons generally range from cutting costs to making assumptions in knowing how users will react to a product. While eliminating user testing may seem like a cost-saving tactic, it could cost in the end. Nielsen Norman Group posits 10 percent of a development product's budget should be spent on user testing because the figure spent will ultimately double usability. Better usability equals more sustained use of the product and ultimate end user satisfaction.

What Happens When You Skip User Testing?

We have established that user testing is a valuable piece when it comes to making a design that works efficiently. But what happens when the decision is made to not user test?

Case Study: Cyberpunk 2077

On December 10th, 2020, a highly-anticipated video game was released for both PC and console play. Cyberpunk 2077-designed and released by CD PROJEKT RED (CDPR)-had already been subject to some criticism due to years-long delays of its release.

Cyberpunk 2077 is a dystopian first-person shooter set in a labyrinthine, layered future world called Night City. CDPR developers knew the intricacies of Night City's environment would present challenges for the hardware the game runs on, specifically older generation-or old-gen-consoles. This includes Xbox One and PS4 machines. Developers also knew simultaneous graphics loading in Night City would take time to successfully complete. Multiple delayed releases hinted at issues like this.

The Problem

Generally, video games implement a more common approach to game design by creating expanded, stretched worlds which can easily accommodate less powerful console hardware. Graphics are loaded as they become visible, akin to cresting a hill and seeing distant objects which weren't originally visible. However, for Cyberpunk 2077 adapting the intricate and layered world of Night City to be playable by older hardware was a problem CDPR spent much time trying to solve.

By December 1st, 2020 PC reviews had begun to emerge and were largely positive. While sixty percent of Cyberpunk 2077 pre-sales were for PC, over 40% of pre-sales were for the console-version of the game. Review copies for the console were delayed and not sent out until December 8th, 2020, a mere two days before the game was to release to the public. Console users had issues with the fact that reviews were limited to PC users and that CDPR needed an extra week to smooth out issues with old-gen consoles.

Not User Testing

Important information became clear during a CDPR conference call with key management board members. The call occurred December 14th, 2020, four days after initial product launch. The 44-minute call soothed investors' worries and addressed reasonable concerns regarding bug fixes. One passage stood out:

"Maybe one thing which didn't help us is COVID as internal testers are able to test Cyberpunk 2077 working at homes because we provide them with all connection machines and so on. But external testers, working for external companies, were not able to test game from homes because they have test centers and they are not there, they're not able to work. So we had some decrease in the number of testers, but I wouldn't point at it as a major source of problems."

Not only did CDPR have a "decrease in the number of testers", they had a decrease in the number of external testers.

PS4 gameplay, courtesy of Gamer Sarukesh. Please note that this video contains violent themes.

Consequences to Not User Testing

According to Metacritic, a website that amasses professional and user reviews for various types of media, the PC version of Cyberpunk 2077 maintains a score of 85 out of 100. The PS4 version, considered an old-gen console, has a score of 57. Concerns with the old-gen console reviews noted graphic issues, bugs, and crashes.

While Cyberpunk 2077 pre-sold about 8 million units, over 500 million USD, only around 30,000 games were refunded, 1.8 million USD. This number doesn't include returns to retailers such as GameStop and BestBuy. It's also a nearly negligible amount of units returned at less than one percent. CDPR did fairly well financially with returns, all things considered. But user satisfaction and trust in their product was compromised, factors that cannot be financially measured.

Could this have been avoided? Maybe. Cyberpunk 2077 was officially announced by CDPR in May of 2012, a time when the new-gen consoles weren't even out yet; PlayStation 4 and Xbox One were released in 2013, both of which were considered old-gen by Cyberpunk 2077's release in 2020. Remote work was prevalent throughout 2020 due to an unprecedented pandemic, which made external user testing either impossible or highly improbable. Wishful thinking did not produce a fully functional game, and delayed old-gen review release introduced doubt into users' minds.

Why User Testing Matters?

While we are not subject matter experts on game user experience and game user testing, we know user testing. We also understand the importance behind internal teams taking a step back and contracting external individuals to help with design problems. A fresh set of eyes from outside of the organization and having an unbiased perspective are the most value tools to utilize when deciphering user experience problems. External testers do not feel the same pressure from executives to deliver the right results, which is why they are crucial to user testing. Failure to externally user test, will likely produce more headaches than necessary. And who wants unnecessary stress?

End Note

Follow us on LinkedIn to see the next article in your feed.

The Cost of Not User Testing