Net Now - Valuing UX Design
Friday, October 17, 2003
The Red Herring of Usability ROI

Review: BayCHI October Program Meeting
Seven Myths of Usability ROI
Daniel Rosenberg, VP User Interface Design
Oracle Corporation

According to Daniel Rosenberg, VP of User Interface Design for Oracle Corporation, there is no tangible return on investment (ROI) for user interface design in software. In his talk at the October BayCHI program meeting at the Palo Alto Research Center, he presented seven myths of usability ROI. They are:

1. Generalization is valid
2. Usability cost is best calculated from the producer perspective
3. You can ignore other factors (functionality, performance, marketing, network externalities)
4. Analog comparison is not required
5. Usability $ is always spent effectively
6. Executives believe voodoo economics
7. Usability investment shortens software development time

Fundamentally, I agree with Mr. Rosenberg’s perception that these are common myths in the UXdesign community. In particular, myths 1, 3, 5, and 6 ring especially true based on my own experience developing usability project valuation metrics for web sites. It seems to me that the usability community is seeking a general statement like: “investment of x dollars in usability will have a return of y dollars” – the thinking is that this will prove value to executives and will result in increased development resources. However, my own belief is that seeking a macro level assertion of this kind is a red herring (most likely instilled by years of reading reports by Jakob Nielson).

In real-world finance, no executive or analyst expects such a generalizable formula for any ROI calculation. Whether building a new factory or opening a new line of business, investment decisions are made by comparing and adjusting the multiple factors that make up a good ROI estimate. In this way, ROI is a project valuation tool and a means for executives to choose between multiple projects given their access to scarce resources (capital, people, etc.). It is not a means to say: “investing $1 billion in a new operation in China always yields a $5 billion return” – that would be a ridiculous assertion. Therefore, what the UXdesign community needs most is not a general panacea to demand more funding, but rather a standardized methodology for calculating a project-specific ROI estimate given the needs of their customers and the current and future goals of their business.

In this light, I found Mr. Rosenberg’s generalization that usability investments have no tangible return very troubling. However, his inclusion of myth 2 illuminates why nine years at Oracle may have given him this view. Mr. Rosenberg talked at length about why Oracle’s cost and return on usability investment is not “important” for enterprise software firms. Instead, total cost of ownership (TCO) for the customer is a much more meaningful metric in assessing competitive advantage.

Now why is that?

First, enterprise software usually requires a great deal of customization, and firms have been able to get away with pushing the cost of UI development on to the customers. As a result, customers have notoriously low expectations for usability in enterprise software. Mr. Rosenberg himself said that Oracle mostly views usability investments as a means to mitigate competitive risk. Therefore, their only financial incentive to make significant usability investments is to keep their TCO metric low compared to the competition – not to develop an optimal UX.

Second, let’s talk about competition in enterprise software. Basically, there is little or none. In fact, since customers typically don’t know how broken the UX is until after they are in the process of implementation, competition doesn’t really matter because all the providers have adopted a standard lazzes faire attitude toward usability. That attitude can be summed up in 3.5 words: “it’s your problem.” To their credit, PeopleSoft is already starting to realize the market opportunity in developing more usable software and has convened meetings of customers to proactively address common problems in future releases.

Finally, let’s talk about pricing. Enterprise software is enormously expensive, both in terms of nominal costs and costs of ownership. Once the decision has been made to go with a certain vendor, the customer is essentially making a commitment to stay with that software platform for 5 to 10 years before any benefits can be fully realized. Unfortunately, because the up-front cost is so high (both price and TCO), the cost of switching to another vendor is also high – firms have little incentive to compete with each other on the basis of UX, and it is the customer who ultimately becomes responsible for ensuring that the tools are usable.

So I’ve spent three paragraphs explaining why enterprise software isn’t a good example of where a usability ROI calculation would be meaningful for the field. To me, it seems there is a logical continuum where the ROI of UX can be expected and where natural competition will ensure that UX plays a vital role in customer satisfaction:

Enterprise software Consumer software Online software

Price / switching cost High Medium Low
Barriers to entry (by competition) High Medium Low
Need for customization High Medium Low

Expected UX ROI Low Medium Potentially High

The chart above may be biased toward my own research, but I think it is a good illustration of why the expected ROI of usability should first be analyzed for online software. Since barriers to entry, price / switching costs, and need for customization are lowest for online software, it is the space where good user experience has the potential for a strong competitive advantage, and likewise, where the most meaningful examination of its financial impact should occur. In addition, analysis of metrics affected by user experience interventions is much more instantaneous online than in packaged software (e.g., you don’t have to wait until the next product release to see impact), so you can better control for other market factors (myth 3).

Once a methodology has been identified and the relevant metrics and their cofactors analyzed, there is potential for realistic application to packaged software. If like Mr. Rosenberg, you believe that the ROI of usability for software is negligible, a decade-old example comes to mind that may change your mind:

As a 22 year old print designer, I was a die-hard WordPerfect user and only very grudgingly switched to MSWord after my responsibilities became more focused on writing. I liked WP because of the control it offered, but I changed to MSWord because of its usability (and because it became more ubiquitous – whether this was the result of monopolistic anti-competitive behavior or true market forces is another debate). As control became less important to me, I appreciated the UI of Word because I didn’t have to remember the key strokes and menus necessary to do very simple layouts. Of course, Word was (and still is) often wrong when it assumes how I want to format an outline, chart, or numbered list, but it is still much “easier” to work around Word’s assumptions than opening and remembering how to adjust WP layout under “reveal codes.”

To sum up the whole ROI debate, I think that Dan Rosenberg offered a very intuitive (and humorous) anecdote that captures the true nature of the beast (or the red fish) in calculating an ROI for usability. As an example of myth 6, he provided the following quote:

“There are 1 billion users on the internet and half of them could come to your site. If the average cost of an abandoned shopping cart is $20 you will lose $10 billion a year in sales of your designer pet food”
-- Rosenberg (2003) Parody of J. Nielsen

Like all of Rosenberg’s observant myths, the misguided belief that statements like these can be made (and more importantly believed!) is the great red herring of usability ROI research. Let’s rid ourselves of these top-down, macro-level assertions and get down to the real work of analyzing specific usability interventions at the project level. Only through rigorous and in-depth analysis can larger patterns emerge and applications be developed.

Friday, October 10, 2003
Web Channel Conflict

It seems to me that at the enterprise level, channel conflict is one of the biggest impediments to fully realizing the business value of a web-based line of business. This problem is most apparent when the success of the offline business is viewed as being compromised by the online business (and vice versa). Generally speaking, it is a problem of managerial accounting and organizational structure – afterall, it is in the firm’s best interest to meet the needs of customers, not to bolster an unproductive asset (whether it be website or storefront).

Over the next few posts, I will present some thumbnail case studies of firms where the dynamics of cross-channel conflict are well illustrated. But first, here’s an example of a company that has limited its trouble with cross-channel conflict

Design Within Reach
Channels: web, catalog, store

I think that the main reason DWR has been so successful in avoidingchannel conflict is because the company was founded as a purely catalog/web business – the design studios are a recent development. As a result, DWR owns the whole distribution system and more importantly, thinks of each of their sales channels (web, catalog, and store) as a fully integrated system. It’s like one big store with three doors: the catalog, the website, and the design studios.

Value of website – DWR values the website because it is their cheapest means of order fulfillment. The catalog and stores funnel orders to the website to the greatest extent possible. In fact, you can’t really buy anything in the store – rather, you place an order with a sales associate. Likewise, the catalog pushes people to order on the web because it is cheaper than having a customer call in an order or place one by mail. Despite the growth of their network of design studios, which doubled in the last year, the size and cost of their call center has remained constant because they continue to improve their ability to funnel orders to their web site.

Value of catalog – As with any catalog business, the biggest value in this channel is their ability to do targeted mailing and response analysis. Generally speaking, e-commerce can learn valuable lessons from the catalog business and its focus on sales metrics, including response rates, sales per page, sales per square inch, etc. Of course, web sites offer marketers even more control of the tools to influence these metrics. And, changes can be tracked and measured instantaneously. Web natives, like Amazon.com, understand the value of this analysis, but it has yet to be comprehensively applied at any of the non-native firms I have worked with.

Value of studio – DWR originally did not plan to open physical stores. I can hypothesize why: Its founders and management team come from specialty retail (in particular, Williams Sonoma / Pottery Barn), where asset management is a major challenge. It’s hard to stay nimble and responsive to a changing marketplace with a huge inventory to manage and storefronts to invest in. However, DWR realized that if they choose their storefronts wisely and don’t over-saturate the market, the return on investment is high and manageable. In addition, they made the very wise decision not to carry inventory. So for DWR, the value of the studio is purely to serve the function of an interactive catalog and 3-D website.

So here are my take-aways from this case:
1. Channel conflict is minimized when the various channels are viewed as equal parts fo one integrated system.
2. Although part of one system, channels make different contributions to the value chain, and should be invested in and evaluated accordingly.
3. Old school contribution: hey, let’s steal from two centuries of catalog history to figure out better ways to evaluate our website.
4. New school thinking: what do you mean retail success doesn’t have to be measured by numbers of stores, inventory turnover, or same-store sales?

Powered by Blogger