Net Now - Valuing UX Design
Wednesday, February 18, 2004
What Is an MBA?
Harvard Business School prides itself in its ablity to train students to make difficult decisions and then to tenaciously defend them. I think this emphasis is an interesting failure given the decisions and rationalizations of some recent alumni:

George W. Bush, POTUS (HBS 75)

Jeffrey Skilling, Enron CEO (HBS 84)

Aren't we all a little better off involving others in our decisions, making the process more transparent, and being willing to admit when we are wrong?
Monday, February 09, 2004
ORKUT.COM Party - Friday, Feb 6, 2004

The orkut launch party was really fun -- we partied like it was 1999! The space was beautiful and the crowd nerdilicious (even if a little too straight for my taste).

A few thoughts:
- wow! google staff are really paranoid -- you can come to our party, BUT DON'T ASK ANY QUESTIONS!;
- orkut, the man, is just as adorable as his site (and feeling a little E-cstatic about his birthday! ... if you catch my meaning);
- San Francisco is still home to lots of fun geeky-freaks. Hooray for us!
Wednesday, February 04, 2004
Six Sigma and Web Development
HOT TOPIC: Control Planning

As the resident MBA at Adaptive Path, I often get asked to explain those businessy phrases you hear all the time (and can fake understanding in context, but still don't really know what they mean). You know, terms like NPV or hurdle rate from corporate finance -- in context they are fairly easy to figure out, but an exact definition is not always obvious. So recently, a colleague at work asked for a one-page primer on Six Sigma ... and specifically how it can be applied to web development processes.

Here's my initial response to the request; i'll update it later.


WOW - that's a tall order. Here's a quick stab at how Six Sigma is being used (but first, ignore anything you read on alertbox or useit about sixsig because I think that Jakob is misapplying this tool -- and I chatted with a few baychi folks who agreed (most notably john zapolski who is pretty smart about this stuff)):

Six Sigma is a design process for manufacturing that was first pioneered by Motorola and GE. The ultimate goal of sixsig is to use intensive quality control to ensure that defect tolerances (or for UX, error rates) are limited to outside six standard deviations from the mean (in other words extremely unlikely). This is especially important for manufacturing things like computer chip equipment, where errors that cause rework are super expensive. Of course, website errors could never achieve this level of quality control because there is no way to account for user error in sixsig. (ed., actually I was wrong about this -- more later. --scott)

It is being applied to UXdesign because the benefits of a six sigma design process extend beyond ensuring a high quality product. Because the error tolerances are so slim, designers must get really really meticulous about their process to ensure that they are accurately measuring all the available data to make design decisions (here is where usability and other user research come in). You often see the abbreviation DMAIC to describe the user-centered design process in sixsig -- I can't remember for sure, but I think DMAIC is: design, measure, analyze, improve, control.

Also, you hear about Management by Fact (MBF), which is another component of sixsig that I think is very applicable to UXdesign. MBF ensures that metrics are identified in the design process to measure the success of the possible outcomes. MBF metrics break the process into smaller pieces so that discrete elements of the process can be analyzed separately, and therefore better controlled. MBF metrics are identified in what is called a process control plan. I like MBF because it emphasizes that design decisions should be made with the support of data (rather than just intuition -- I guess you could say that it is the antithesis of heuristics).

Ultimately, sixsig is about identifying sources of error, developing ways of measuring these errors, and controlling those errors through iterative design. In this regard, another tool you sometimes hear about is a pareto chart -- this is a fancy term for a histogram that is organized by error type so that designers can see what the most common errors are.

In sum, I think there is a lot to learn from TQM in developing better web design processes. Interestingly six sigma has its own UCD component in its adherence to Voice of Customer (VOC) as a source of control planning data and for error identification. Another great thing about TQM is that there is a wealth of literature on the subject, much of which has already been applied to use in software design. Of course, such a rigorous analytic process carries the risk of "designing to metrics" which some view as limiting creativity and focusing too much on incremental changes ... rather bigger-picture strategic design issues. But then, aren't those design projects intiated in a different way? more later ...

Short glossary of Six Sigma terms (google for more info):
Control plan
Management by Fact
Design tolerances
Pareto chart
Hoshin planning
Voice of Customer
Wednesday, January 28, 2004
More on accounting (can you tell i just finished reading "The Smartest Guys in the Room"?):

I was looking for a business definition of the term “pro forma” and I came across this site – it’s kinda crappy, but I’m really intrigued by the idea: www.ventureline.com. I was looking this up because in latin pro can mean “before” or “as good as,” and I found it funny that the debate surrounding the value of pro forma analysis is around whether it is “as good as” SEC filings, but I was pretty sure that it meant “before” SEC filing – BTW, in the accounting glossary at venture line, it means “before” (e.g. financial statements in their “before form,” pre GAAP and pre closing of the books). Ironically, I think it has come to be perceived as meaning “as good as” since the heyday of the 90s (e.g. a lot of people think that SEC fillings are limiting because of strict adherence to GAAP).
As in:
- What do you mean I have to charge my $10 million restructuring costs to Net Income?! (anyfirm.com)
- I know; let's say that our tech investment writedowns are non-recurring, even though we called the run-ups recurring. (Enron circa 2001)
- Let's recognize the revenue of a 10 year software contract today, even though we haven't written the software yet (Oracle circa 1992)
- Let's capitalize and depreciate marketing expenses, it's sort of like an asset (AOL circa 1995)

Monday, January 12, 2004
Reported earnings follow the rules and principles of accounting. The results do not always create measures consistent with underlying economics. However, corporate management's performance is generally measured by acounting income, not underlying economics. Therefore, risk manamgent strategies are directed at accounting, rather than economic, performance.
--Enron in-house acccounting manual
Friday, December 12, 2003
Here's the info for the IA Summit in Austin (my old hometown of 2 years: 1998-2000):
I created this blog to test how good I'd be at posting to it -- in reality, i think i shot myself in the foot. because it's just a test and contains just-for-me content and i'm not telling anyone about, then i don't post to it.
Anyway, it recently came to my attention that a few people had found this content on their own and had read it. So if you are reading this, please drop me an email to say "hi" and let me know what you like.
I'm speaking at the Information Architecture Summit in Austin, TX -- February 27 - 29, 2003:

ROI of User Experience

“ROI” has been the buzzword of the year. Whether justifying budgets or deciding how to allocate resources, design management professionals are struggling to define the real business impacts of high-quality user experience. Despite this growing interest, little research has drawn a clear link between user experience and business results.

In 2003 Adaptive Path partnered with researchers at UC Berkeley’s Haas School of Business and a distinguished panel of independent advisors to conduct a study that would identify best practices and common obstacles in linking user experience investments to business returns.

This session will unveil the results of this 6-month research study. This session will give you:
• 7 case studies, including KQED, Cathay Pacific Airlines, and Bank of America
• Best practices for gathering value metrics and using them to guide user experience management decisions
• Proven methods for prioritizing projects based on value criteria
• A framework for understanding how ROI and other value metrics can get you credibility, resources, and ultimately better designs
• 10 techniques for measurement that can be used immediately by managers

Advisory Panel: Sara Beckman, PhD, Professor, Haas School of Business; Harry Rich, Director, UK Design Council; Peter Morville, Semantic Studios; Suzanne Van Cleve, Director, PeopleSoft.com; Andrew Anker, August Capital.


Janice Fraser, Partner, Adaptive Path
Janice Fraser is a founding partner of Adaptive Path, a premier user experience firm. She has worked in high-tech media for over a decade, and pioneered consumer Web applications for Netscape in 1996. Other clients include The United Nations, PeopleSoft, Intel, and Weight Watchers.
Janice is a featured speaker for the Nielsen/Norman User Experience World tour, the founder of four startup companies, and was previously managing editor for IDG Communications.
In addition to her work with Adaptive Path, Janice teaches interaction design at San Francisco State University's Multimedia Studies Program, where she instructs students in high-level interaction design, the business side of user experience, ROI, and politics management.
Her credentials include leading sites in the U.S. and overseas, such as WellsFargo.com, Verisign, NewsCorp, BarnesandNoble.com, SGI, and LineOne.
Janice is a frequent contributor to industry publications such as Boxes and Arrows, and has spoken at conferences including Seybold, South by Southwest, AIGA’s GAIN, CMP Media's WEB shows.

Scott Hirsch, MBA, UC-Berkeley Haas School of Business
Scott Hirsch is a consultant specializing in project finance and development processes. He is also the lead author of "Return on Investment of User Experience Design: Case Studies in Business Analysis and Project Valuation,” a collaborative study sponsored by Adaptive Path and researchers from UC Berkeley's Haas School of Business. Scott's recent clients include Google, Wells Fargo, Weight Watchers, and Network Associates.
Scott helps clients understand and measure the business value of their online user experience by examining their entire Web development process with an emphasis on project selection, financing, evaluation, and accountability. This analysis provides clients with project-success criteria and financial metrics designed to capture valuable user experience data and prioritize future Web development projects.
During his graduate studies, Scott extensively researched emerging techniques for using financial information analysis to value product design and development processes. He has presented on his research and client experience at the DUX 2003 conference and was a featured speaker for BayCHI's monthly program in December 2003.
Scott previously developed and evaluated AmeriCorps programs at the Texas Commission on Volunteerism and Community Service in Austin, Texas. Scott has an MBA from UC Berkeley's Haas School of Business and a double BA in English and Geology from the University of Virginia.

Friday, October 17, 2003
The Red Herring of Usability ROI

Review: BayCHI October Program Meeting
Seven Myths of Usability ROI
Daniel Rosenberg, VP User Interface Design
Oracle Corporation

According to Daniel Rosenberg, VP of User Interface Design for Oracle Corporation, there is no tangible return on investment (ROI) for user interface design in software. In his talk at the October BayCHI program meeting at the Palo Alto Research Center, he presented seven myths of usability ROI. They are:

1. Generalization is valid
2. Usability cost is best calculated from the producer perspective
3. You can ignore other factors (functionality, performance, marketing, network externalities)
4. Analog comparison is not required
5. Usability $ is always spent effectively
6. Executives believe voodoo economics
7. Usability investment shortens software development time

Fundamentally, I agree with Mr. Rosenberg’s perception that these are common myths in the UXdesign community. In particular, myths 1, 3, 5, and 6 ring especially true based on my own experience developing usability project valuation metrics for web sites. It seems to me that the usability community is seeking a general statement like: “investment of x dollars in usability will have a return of y dollars” – the thinking is that this will prove value to executives and will result in increased development resources. However, my own belief is that seeking a macro level assertion of this kind is a red herring (most likely instilled by years of reading reports by Jakob Nielson).

In real-world finance, no executive or analyst expects such a generalizable formula for any ROI calculation. Whether building a new factory or opening a new line of business, investment decisions are made by comparing and adjusting the multiple factors that make up a good ROI estimate. In this way, ROI is a project valuation tool and a means for executives to choose between multiple projects given their access to scarce resources (capital, people, etc.). It is not a means to say: “investing $1 billion in a new operation in China always yields a $5 billion return” – that would be a ridiculous assertion. Therefore, what the UXdesign community needs most is not a general panacea to demand more funding, but rather a standardized methodology for calculating a project-specific ROI estimate given the needs of their customers and the current and future goals of their business.

In this light, I found Mr. Rosenberg’s generalization that usability investments have no tangible return very troubling. However, his inclusion of myth 2 illuminates why nine years at Oracle may have given him this view. Mr. Rosenberg talked at length about why Oracle’s cost and return on usability investment is not “important” for enterprise software firms. Instead, total cost of ownership (TCO) for the customer is a much more meaningful metric in assessing competitive advantage.

Now why is that?

First, enterprise software usually requires a great deal of customization, and firms have been able to get away with pushing the cost of UI development on to the customers. As a result, customers have notoriously low expectations for usability in enterprise software. Mr. Rosenberg himself said that Oracle mostly views usability investments as a means to mitigate competitive risk. Therefore, their only financial incentive to make significant usability investments is to keep their TCO metric low compared to the competition – not to develop an optimal UX.

Second, let’s talk about competition in enterprise software. Basically, there is little or none. In fact, since customers typically don’t know how broken the UX is until after they are in the process of implementation, competition doesn’t really matter because all the providers have adopted a standard lazzes faire attitude toward usability. That attitude can be summed up in 3.5 words: “it’s your problem.” To their credit, PeopleSoft is already starting to realize the market opportunity in developing more usable software and has convened meetings of customers to proactively address common problems in future releases.

Finally, let’s talk about pricing. Enterprise software is enormously expensive, both in terms of nominal costs and costs of ownership. Once the decision has been made to go with a certain vendor, the customer is essentially making a commitment to stay with that software platform for 5 to 10 years before any benefits can be fully realized. Unfortunately, because the up-front cost is so high (both price and TCO), the cost of switching to another vendor is also high – firms have little incentive to compete with each other on the basis of UX, and it is the customer who ultimately becomes responsible for ensuring that the tools are usable.

So I’ve spent three paragraphs explaining why enterprise software isn’t a good example of where a usability ROI calculation would be meaningful for the field. To me, it seems there is a logical continuum where the ROI of UX can be expected and where natural competition will ensure that UX plays a vital role in customer satisfaction:

Enterprise software Consumer software Online software

Price / switching cost High Medium Low
Barriers to entry (by competition) High Medium Low
Need for customization High Medium Low

Expected UX ROI Low Medium Potentially High

The chart above may be biased toward my own research, but I think it is a good illustration of why the expected ROI of usability should first be analyzed for online software. Since barriers to entry, price / switching costs, and need for customization are lowest for online software, it is the space where good user experience has the potential for a strong competitive advantage, and likewise, where the most meaningful examination of its financial impact should occur. In addition, analysis of metrics affected by user experience interventions is much more instantaneous online than in packaged software (e.g., you don’t have to wait until the next product release to see impact), so you can better control for other market factors (myth 3).

Once a methodology has been identified and the relevant metrics and their cofactors analyzed, there is potential for realistic application to packaged software. If like Mr. Rosenberg, you believe that the ROI of usability for software is negligible, a decade-old example comes to mind that may change your mind:

As a 22 year old print designer, I was a die-hard WordPerfect user and only very grudgingly switched to MSWord after my responsibilities became more focused on writing. I liked WP because of the control it offered, but I changed to MSWord because of its usability (and because it became more ubiquitous – whether this was the result of monopolistic anti-competitive behavior or true market forces is another debate). As control became less important to me, I appreciated the UI of Word because I didn’t have to remember the key strokes and menus necessary to do very simple layouts. Of course, Word was (and still is) often wrong when it assumes how I want to format an outline, chart, or numbered list, but it is still much “easier” to work around Word’s assumptions than opening and remembering how to adjust WP layout under “reveal codes.”

To sum up the whole ROI debate, I think that Dan Rosenberg offered a very intuitive (and humorous) anecdote that captures the true nature of the beast (or the red fish) in calculating an ROI for usability. As an example of myth 6, he provided the following quote:

“There are 1 billion users on the internet and half of them could come to your site. If the average cost of an abandoned shopping cart is $20 you will lose $10 billion a year in sales of your designer pet food”
-- Rosenberg (2003) Parody of J. Nielsen

Like all of Rosenberg’s observant myths, the misguided belief that statements like these can be made (and more importantly believed!) is the great red herring of usability ROI research. Let’s rid ourselves of these top-down, macro-level assertions and get down to the real work of analyzing specific usability interventions at the project level. Only through rigorous and in-depth analysis can larger patterns emerge and applications be developed.

Friday, October 10, 2003
Web Channel Conflict

It seems to me that at the enterprise level, channel conflict is one of the biggest impediments to fully realizing the business value of a web-based line of business. This problem is most apparent when the success of the offline business is viewed as being compromised by the online business (and vice versa). Generally speaking, it is a problem of managerial accounting and organizational structure – afterall, it is in the firm’s best interest to meet the needs of customers, not to bolster an unproductive asset (whether it be website or storefront).

Over the next few posts, I will present some thumbnail case studies of firms where the dynamics of cross-channel conflict are well illustrated. But first, here’s an example of a company that has limited its trouble with cross-channel conflict

Design Within Reach
Channels: web, catalog, store

I think that the main reason DWR has been so successful in avoidingchannel conflict is because the company was founded as a purely catalog/web business – the design studios are a recent development. As a result, DWR owns the whole distribution system and more importantly, thinks of each of their sales channels (web, catalog, and store) as a fully integrated system. It’s like one big store with three doors: the catalog, the website, and the design studios.

Value of website – DWR values the website because it is their cheapest means of order fulfillment. The catalog and stores funnel orders to the website to the greatest extent possible. In fact, you can’t really buy anything in the store – rather, you place an order with a sales associate. Likewise, the catalog pushes people to order on the web because it is cheaper than having a customer call in an order or place one by mail. Despite the growth of their network of design studios, which doubled in the last year, the size and cost of their call center has remained constant because they continue to improve their ability to funnel orders to their web site.

Value of catalog – As with any catalog business, the biggest value in this channel is their ability to do targeted mailing and response analysis. Generally speaking, e-commerce can learn valuable lessons from the catalog business and its focus on sales metrics, including response rates, sales per page, sales per square inch, etc. Of course, web sites offer marketers even more control of the tools to influence these metrics. And, changes can be tracked and measured instantaneously. Web natives, like Amazon.com, understand the value of this analysis, but it has yet to be comprehensively applied at any of the non-native firms I have worked with.

Value of studio – DWR originally did not plan to open physical stores. I can hypothesize why: Its founders and management team come from specialty retail (in particular, Williams Sonoma / Pottery Barn), where asset management is a major challenge. It’s hard to stay nimble and responsive to a changing marketplace with a huge inventory to manage and storefronts to invest in. However, DWR realized that if they choose their storefronts wisely and don’t over-saturate the market, the return on investment is high and manageable. In addition, they made the very wise decision not to carry inventory. So for DWR, the value of the studio is purely to serve the function of an interactive catalog and 3-D website.

So here are my take-aways from this case:
1. Channel conflict is minimized when the various channels are viewed as equal parts fo one integrated system.
2. Although part of one system, channels make different contributions to the value chain, and should be invested in and evaluated accordingly.
3. Old school contribution: hey, let’s steal from two centuries of catalog history to figure out better ways to evaluate our website.
4. New school thinking: what do you mean retail success doesn’t have to be measured by numbers of stores, inventory turnover, or same-store sales?

Sunday, July 27, 2003
In the internet world of the future, everyone will be famous (but for very small things).
Net Now - User Experience Valuation
I've been trying to come up with a cute name for measuring the financial impacts of good user experience design. I like how "net" has three meanings:
1. in accounting, it refers to the bottom line sum (revenues after expenses);
2. here, it is short for internet; and
3. most commonly, it just refers a weblike structure to catch things.
All of these are apropos in some way or another -- I don't really like "Net Now," but it's a start.
What do you think?

Powered by Blogger