Usability: How Users Can Help Us Assess It

Rory Channer
David Jennings

Overview
Who is this for?
What does the framework do?
What are the benefits?
Where should the framework be used?

Introduction: The Business Context and Usability Measures
What is usability?
The business context
Usability measures

The Framework: How to Evaluate the Usability of a System
Step 1 - Collect information on the business context
Step 2 - Select and define usability measures
Step 3 - Run the evaluation
Step 4 - Analyse and use your results

Annex A - The Components of the Business Context
Users, Tasks, Equipment, Environment

Annex B - Where the Framework for Usability

Evaluation can be Applied

Annex C - References

Note: This document was initially published for the UK Employment Service, as Psychological Services Customer Report No. 456, in August 1994. The original version can be obtained by telephoning Occupational Psychology Branch on 0114 259 7792 (+44 114 259 7792 from outside UK). The document is (c) Crown Copyright 1994.

Rory Channer is now at: Business Systems Division, Thames Water Utilities Ltd, 8th Floor Nugent House, Vastern Road, Reading, UK, RG1 8DB - Tel: 01734 593495 - Fax: 01734 593100 - Email: channerr@twul.demon.co.uk

Who is this for?

The document provides a framework for people who want to specify or evaluate the usability of IT systems. The framework explains:

• the main aspects of usability
  
• how to assess these aspects in different business contexts.
  

What does the framework do?

The framework explains how usability can be measured in terms of user performance and satisfaction. It identifies the main aspects of usability and analyses the context of use.

It details the steps that need to be taken when specifying, designing or evaluating the usability of a system or product. The framework is based on a Draft International Standard (see Annex C for reference).

The Annexes give examples of how and when to use the framework - including a "fastpath" approach - as well as further pointers to enable readers to pursue usability in more depth.

What are the benefits?

Assessing and improving usability means:

• A better fit of systems to business processes and tasks, leading to higher productivity and efficiency
  
• Fewer "user errors"
  
• Less staff training required
  
• Users are more satisfied, leading to more system use and to secondary benefits such as lower staff turnover.
  

Where should the framework be used?

The framework can be used whenever a sample of users is available to use a real, simulation or prototype system. When users are not available, different human factors techniques can be used to improve usability.

1. What is Usability?

Usability can be defined as the extent to which a product can be used to achieve specified business goals with

• effectiveness,
  
• efficiency and
  
• satisfaction
  

in a specified business context.

Effectiveness and efficiency can be grouped together as measures of users' performance. Satisfaction can also affect business performance as dissatisfied users do not use a system to best effect.

2. The Business Context

Usability is affected by the context in which a system is used.

The characteristics of the business context may be as important in determining usability as the characteristics of the system itself.

Changing any relevant aspect of the business context may change the usability of the product. For example a product which is usable by trained users may be unusable by untrained users.

The business context includes:

• the users
  
• their tasks
  
• their equipment
  
• their environment
  

Users can be seen in terms of their knowledge, skill, experience, education, training, physical attributes and motor and sensory capabilities.

Tasks are the means by which business goals are achieved. The characteristics of these tasks that may influence usability include variables such as their frequency and their duration.

The equipment is the set of one or more products which may be the focus of a usability specification or evaluation. Or the equipment may be a set of attributes or performance characteristics of the hardware, software and other materials.

The environment includes parts of

• the wider technical environment e.g. local area network
  
• the physical environment e.g. furniture
  
• the ambient environment e.g. temperature, humidity and
  
• the social and cultural environment e.g. organisational structure and attitudes.
  

Further information about the components of the context of use can be found in Annex A.

3. Usability Measures

Usability measures consist of target or actual values of effectiveness, efficiency, and satisfaction for the required contexts.

Effectiveness

Definition: Effectiveness can be defined as the accuracy and completeness with which users achieve specified goals.

Example: If the desired goal is to copy-type a memo in a set format, then accuracy could be specified or measured by the number of spelling mistakes and the number of deviations from the format. Completeness would be the number of words of the memo typed divided by the number of words in the original document.

Efficiency

Definition: Efficiency is the level of effectiveness achieved as a proportion of the resources expended by users to obtain a specified goal. Relevant resources may include mental or physical effort, time, materials or financial cost.

Example: If the desired goal is to print copies of a report, then efficiency could be specified or measured by dividing the number of usable copies of the report printed by the resources spent on the task such as time or paper. Human efficiency could be measured as effectiveness divided by human effort, speed efficiency as effectiveness divided by time, and economic efficiency as effectiveness divided by cost.

Satisfaction

Definition: Satisfaction describes the comfort and acceptability of a system experienced by both the people who use it and the people affected by its use. Comfort refers to overall physiological or emotional responses to a system. Acceptability refers to the user's perception of specific aspects of the system (e.g. Does the user feel that the software supports the way they carry out there tasks?). Measures of satisfaction may relate to specific or more global, overall aspects of the system.

Example: A user does not like using an electronic mail system. As there are other forms of communication, this system will not be utilised and is hence unacceptable to the user. However, if a user has to type a document then a word processing package has to be used, even if it is not particularly usable. Hence satisfaction, and in particular comfort, will be low.

The Framework: How to Evaluate the Usability of a System

Step 1 - Collect Information on the Business Context

1. Identify what you want to get out of the evaluation.

Consider the context of the evaluation and assess, for example, whether you want "formative" assessments to inform the further development of a bespoke system, or "summative" measures to inform a choice between different package options. Being clear about what you want from the evaluation is critical to directing all the other steps. See Annex B for more details on the business contexts where this framework could be applied.

2. Analyse the characteristics of different types of user.

Concentrate on the characteristics that you would expect to affect their performance and satisfaction with the system.

For example, one characteristic would be the different work roles of users; another would be their previous experience of different systems or packages (e.g. Windows software).

3. Analyse the relevant tasks for which the users will be using the system.

Select a set of key tasks that are important to the business. Specify the steps that the user has to carry out to achieve the goals of the task.

Information about goals and tasks can be broken down into sub-goals. For example, a vacancy taking clerk may have the overall goal to maintain sufficient information on a computerised vacancy system to allow the greatest number of job seekers to be matched to a relevant vacancy. This task can be broken down into sub-goals such as "Take all the details from the company offering the vacancy" and "Input this information into the information system".

4. Analyse the characteristics of the equipment and environment.

Again concentrate on the characteristics that you would expect to affect users' performance and satisfaction with the system.

For example, how many interruptions do users have to deal with in the course of carrying out their tasks?

5.Write down specifications of the characteristics and tasks identified from the previous three activities.

Include working assumptions about how certain characteristics may or may not affect users' performance and satisfaction.

These specifications are a key resource for future steps.

For further examples of the business context, including characteristics that may be relevant, see Annex A.

1. Define measures of effectiveness based on business and user goals.

Based on the overall goal and sub-goals of the business tasks (step 1.3), identify and specify ways in which you can measure how completely and accurately these goals are achieved.
Decide what constitutes the required output of each goal. Identify how you could tell how completely each goal is achieved. Identify how you could tell how accurate the output is. Specify a scoring procedure for measuring how complete and accurate the output of each goal is compared to what is required. This should also take into account any output that was not asked for.

The completeness and accuracy of users' achievement of goals are your measures of effectiveness.

2. Define measures of efficiency.

Based on the business tasks (step 1.3), identify and specify ways in which you can measure how much resource users have to invest to carry out the tasks. This resource will usually be the time it takes them, but may also include any possible measure of effort they have to put in. The resources invested to carry out tasks are your measures of efficiency.

For example, a simple measure of efficiency is the number of standard tasks completed by users in a fixed time.

Consider whether it is possible to get any benchmark measures of efficiency for comparison purposes.

3. Define measures of satisfaction.

Ensure that these measures address users' perceptions of the system, including the degree of control they have over it, its helpfulness, its learnability, and their interest and enjoyment in using it. Consider measures such as attitude rating scales or the ratio of positive to negative comments. Longer term measures such as rate of absenteeism, health problem reports, or frequency of job transfer may also be relevant.

There are standardised questionnaire for measuring user satisfaction with IT systems available, such as the Software Usability Measurement Inventory (SUMI).

Make sure that you provide at least one measure of efficiency, effectiveness and satisfaction. It may be necessary to repeat measures in several different contexts.

Tips on the selection of measures:

• When selecting usability measures consider how they will affect the business benefits of the system. Where usage is discretionary (i.e. users do not have to use the system to achieve their goals), the importance given to measures of user satisfaction may be much greater than when usage is not discretionary. Dissatisfied users will look for ways to avoid using the system.
  
• Satisfaction may be assessed for specific features, or for use of the product as a whole.
  
• Measures of satisfaction may assess emotional response to use of the product, or assess the user's perception of aspects such as efficiency, helpfulness or learnability.
  
• Measuring over a short time period may miss infrequent occurrences such as intermittent system errors which affect usability. So care should be taken when selecting the time period. Furthermore the initial learning by users on a new system will effect the quality of the earlier data so care must be taken in ensuring that users are all at the same level.
  

Your choice of measures may depend, in the main, on what you are evaluating usability for (see Annex B for some examples of where to apply the framework).

1. Prepare task instructions.

Based on your analysis of important business tasks (Step 1.3), write some instructions for users to follow when doing the tasks on the system.

Make sure the instructions are described in business, not system terms. The instructions should not give direct guidance on how to use the system, but should specify what business goals the user should try and achieve.

Pilot your task instructions by trying them out with a few users. If you don't do this, you may end up evaluating the usability of your instructions, not of the system!

2. Prepare the evaluation environment and equipment.

Based on your analysis of the characteristics of these (Step 1.4), make sure the evaluation environment simulates the important aspects of users' the real working environment.

3. Recruit a representative sample of users.

Based on your analysis of the characteristics of users (Step 1.2), make sure the team of users is balanced on all the important characteristics, so that your results can be generalised to the wider user community.

4. Train the users in use of the system.

Give sufficient training to ensure that users are reasonably well up the learning curve when they move on to do the tasks (see next activity).

5. Ask users to follow the task instructions, using the system.

Make sure that they are well briefed in advance. Emphasise that you are not evaluating them; you are evaluating the system. Ensure that users do not get irretrievably stuck at any point. Ideally run the evaluation with both training and technical support on hand, in case they are needed.

6. Collect the evaluation data.

It is best to give users any satisfaction measures after they have completed the tasks. Collect any questionnaires they have completed. Also collect any outputs from their work on the system, either on paper or on disk.

1. Apply the measures of effectiveness, efficiency and satisfaction to your data.

Based on the activities in Step 2, this should be fairly straightforward.

2. Report the results and recommend what further action should be taken on the basis of the results.

Review what you wanted to get out of the evaluation (Step 1.1), and specify how the results provide this - that is, what formative or summative assessments you can make, and what should be done on the basis of these.

3. Consider the results and recommendations in the context of other information on the system.

The results from a usability evaluation will contribute to an overall assessment of a product and hence will be used in conjunction with information such as

• How well the system fits with related business strategy
  
• Other product evaluation results e.g. cost-effectiveness, practicalities of implementation, technical performance.
  
• Policy considerations e.g. Making sure of supplying companies financial position.
  

4. Beware of "overusing" the results.

Note that the results from any measurement of usability may not be generalisable to another context with different types of users, tasks or environments.

When evaluating a product for general use by different users in different contexts you may have to obtain several different representative contexts and users.

Annex A - The Components Of The Business Context

This Annex gives a framework for acquiring the necessary information for the context of use.

The context of use consists of the following:

 Users                 Equipment
     \                 /
      Business Context
     /                 \
 Tasks                 Environment - Organisational
                                   - Physical

The information that is needed for a usability evaluation can therefore be gathered using the following headings:

Users

The type of user

For example:

• A middle manager who occasionally uses a computer.
  
• A secretary would be a frequent user of a word processing system.
  
• A senior manager receiving minutes of a meeting would be a secondary or Audience user.
  

For example:

• Physical capabilities. If a user's hearing is impaired this will have an impact on the usability of a system which gives auditory feedback.
  
• Intellectual Ability. A local office may have a public access system, such as database holding information on current job vacancies. A member of the public may find this too difficult to understand and use.
  
• Attitude. If an employee believes that computers are too complex to learn and use (no matter how usable they are), then this impacts upon usability evaluation.
  
• Motivation. Evaluation using an unmotivated user may well give misleading results about the usability of the system.
  

For example:

• Product and general systems experience, keyboard/input skills
  
• Task experience
  
• Organisational experience
  
• Qualifications, training, language skills and general knowledge.
  
• Physical capabilities, limitations and disabilities.
  

Tasks

• The name and goal of the task. Are there any sub-goals within the task?
• Frequency, Duration and flexibility of the Task.
• Physical and Mental demands.
• What is the Output? What happens if errors occur during performance of the task? Who is dependent on the results of the task

Equipment

• A basic description should include:
  
  Product Identification
• Product Description
• Main Application Areas
• Major Functions
• A specification of the hardware, software and any other materials
  
  

Environment

The organisation and its environment

Attitudes and culture towards computers and their related products.

• Job design:
  • Job flexibility
  • Feedback and monitoring received on Performance
  • Autonomy and discretion
  • Deadlines and time pressures
  • Assistance received
    

Organisational structure and policy:

• Hours of work
  
• Group working
  
• Departmental and job functions
  
• Work practices
  
• Remuneration
  
• Communication Structure
  

• Workplace conditions:
  
  
  • Atmospheric conditions
    
  • Auditory and visual environment
    
  • Thermal Environment
    
  • Environmental Stability
    
    
• Workplace design:
  
  
  • Space and furniture
    
  • User Posture
    
  • Location of Office
    

This Annex gives a list of where this framework can be applied. It is not comprehensive, but aims to give a flavour of the possible areas of application. These are not necessarily discrete areas and hence it is likely that they overlap in places. Furthermore it should be noted that this framework is one method for evaluating usability. The references in Annex C will point you in the direction of other methods.

Demonstrations - "Fastpath" evaluation

When a software supplier is offering a demonstration of a system, you will not have much time for preparation, evaluation or analysis. However the framework can still be useful. Although you cannot do full analysis of the business context, it is valuable to write down your assumptions and estimates of what the important factors of the context will be (see Step 1.5). This should take about an hour and one or two sides of A4. The same goes for Step 2: it should be possible to devise and specify some on-the-spot rough and ready measures. These will require limited preparation and analysis in Steps 3 and 4.

Instead of letting the supplier set the agenda it will be more useful to ensure that a selection of users "have a go" with the system. After using the system it may be appropriate to test satisfaction with the Software Usability Measurement Inventory (SUMI).

Pilots

Any evaluation of a system in the pilot stage of implementation should include usability as a measure. By following the guidelines in this framework, in advance of the pilot going 'live', can ensure that usability is monitored.

Products - off the shelf comparative evaluations

Before a particular product is piloted or implemented it is often necessary to ensure that the chosen system is the most appropriate for the context and is generally out performing it's competitors. When exploring the different system options it is strongly advisable to assess their usability as well as other criteria such as cost, implementation procedures, availability etc.

Formative evaluation of bespoke developments

A system designed for particular users is unlikely to be created without an ongoing interchange between the designers and the those users. Both should be aware of the concepts of usability and usability evaluation.

Formal Usability Laboratories

This type of approach assesses usability in a laboratory type setting and tends to be more academic in nature. Video and other recording equipment are used to monitor users. It should be highlighted that there may well be large differences in the context of use between the laboratory and where the system is to be used. However, it may not be possible or appropriate to test users elsewhere due to the organisational or physical environment restraints.

The focus of usability is on a product which is part of the hardware and software. However the evaluation to improve quality of use could be on any part of the work system, such as environmental influences (e.g. lighting) or training requirements. Measures of user performance and satisfaction can be used to evaluate how any component of a work system contributes to the overall quality of use.

The aim of this annex is to list relevant references and briefly describe what can be found in them.

1. Benyon, D. and Rogers, Y. (1990) A Guide to Usability: Usability Now. The Open University in association with the Department of Trade and Industry.

This guide book begins with brief overview of the major issues in Human Computer Interaction (HCI). It then addresses some of the issues about the impact of introducing a computer system on people working in an organisation. It gives some clear guidelines on designing 'usable' systems and a chapter on methods and techniques for evaluation of products. The guide introduces good practice in usability and the important issues that need to be addressed. There is also a fairly extensive list of references and further reading.

2. Clegg, C., Warr, P., Green, T. et al (1988) People And Computers - how to evaluate your company's new technology, Ellis Horwood Books.

The book is intended for both people experiencing problems or considering investing further in their present equipment and can be used as a guide or as a source of information. It will help in an evaluation by highlighting the important questions which need to be addressed. These questions are derived from thirty different issues, however usability is covered in chapter 3. Chapter 6 is also relevant as it details ways of carrying out an evaluation but it should be noted that this is broader than a usability evaluation.

3. ISO 9241 Ergonomic requirements for office work with visual display terminals - Part 11: Guidance on Usability. .

This International Organisation for Standardisation document explains how usability can be measured in terms of user performance and satisfaction. This standard was the basis for the framework presented here.

4. Macleod, M. (1992) An Introduction to Usability Evaluation: Usability Now, The Department for Enterprise. National Physical Laboratory.

This booklet provides a bridge between introductory material and the numerous methods and tools that can be used in usability evaluation. It addresses a range of issues about the different methods, their benefits, where and when they can be applied and how the results can be used. The 'Evaluation Methods at A Glance' on page 12 gives a good general overview and guide. The booklet then details several of the specific methods for evaluating usability.

5. Ravden, S.J. and Johnson, G.I. (1989) Evaluating Usability of Human-Computer Interfaces: A Practical Method, Ellis Horwood books.

This book offers a practical method of evaluating the usability of computer-based application systems both during and after development. It is aimed at those who are directly involved in the design, development and evaluation of human-computer system interface. It is also aimed at those evaluating one or more systems with a view to purchase. The method outlined is a checklist based on a set of goals which a well designed user interface should aim to have. After explaining all the components of the checklist, this book then details how to conduct an evaluation and analyse the results obtained after using this particular tool.

6. Wright, P. and Monk, A. (1991) Cooperative Evaluation: The York Manual. The Department Of Psychology, The University of York, York.

Cooperative evaluation is a way of obtaining information about problems experienced with different types of software. It is collaborative as users and designers evaluate the system together. Here users are encouraged to ask questions about interacting with the system. The method can be used on existing products, prototypes and simulators. This manual explains the steps that need to be taken in the evaluation, how to overcome certain problems and how to use the results of the evaluation to formulate recommendations for redesign.

7. Jennings, D. and Kelly, C. (1991) Improving the Usability of Information Systems: A Briefing for System Developers. Psychological Service Customer Report 400. Telephone 0114 259 7792 (or extension 7792 within Sheffield) for a copy.

This report is structured around a number of Employment Department Group case studies of improving usability. These cover initiatives at different stages through the development lifecycle, so some include user involvement and others do not. Although some of the case studies may now seem slightly dated, the principles underlying them remain a good overview of the scope of designing for usability.

Copyright © David Jennings & Rory Channer