Research Methods in Human-Computer Interaction

About the Authors xvii

Acknowledgments xviii

Preface xix

1 INTRODUCTION 1

1.1 Changes in topics of HCI research over time 3

1.2 Shifts in measurement in HCI 5

1.3 Inherent conflicts in HCI 9

1.4 Interdisciplinary nature of HCI research 11

1.5 Communicating your ideas 13

1.6 Research and usability testing 14

2 EXPERIMENTAL RESEARCH 19

2.1 Types of behavioral research 20

2.2 Research hypotheses 22

2.2.1 Null hypothesis and alternative hypothesis 23

2.2.2 Dependent and independent variables 25

2.2.3 Typical independent variables in HCI research 25

2.2.4 Typical dependent variables in HCI research 26

2.3 Basics of experimental research 27

2.3.1 Components of an experiment 27

2.3.2 Randomization 28

2.4 Significance tests 30

2.4.1 Why do we need them? 30

2.4.2 Type I and Type II errors 32

2.4.3 Controlling the risks of Type I and Type II errors 34

2.5 Limitations of experimental research 34

3 EXPERIMENTAL DESIGN 41

3.1 What needs to be considered when designing experiments? 43

3.2 Determining the basic design structure 44

3.3 Investigating a single independent variable 45

3.3.1 Between-group design and within-group design 46

3.3.2 Choosing the appropriate design approach 49

3.4 Investigating more than one independent variable 53

3.4.1 Factorial design 53

3.4.2 Split-plot design 54

3.4.3 Interaction effects 55

3.5 Reliability of experimental results 57

3.5.1 Random errors 57

3.5.2 Systematic errors 57

3.6 Experimental procedures 63

4 STATISTICAL ANALYSIS 69

4.1 Preparing data for statistical analysis 70

4.1.1 Cleaning up data 70

4.1.2 Coding data 71

4.1.3 Organizing data 73

4.2 Descriptive statistics 73

4.2.1 Measures of central tendency 73

4.2.2 Measures of spread 74

4.3 Comparing means 74

4.4 T tests 76

4.4.1 Independent-samples t test 76

4.4.2 Paired-samples t test 76

4.4.3 Interpretation of t test results 77

4.4.4 Two-tailed t tests and one-tailed t tests 78

4.5 Analysis of variance 78

4.5.1 One-way ANOVA 79

4.5.2 Factorial ANOVA 80

4.5.3 Repeated measures ANOVA 82

4.5.4 ANOVA for split-plot design 83

4.6 Assumptions of t tests and F tests 86

4.7 Identifying relationships 86

4.8 Regression 89

4.9 Nonparametric statistical tests 91

4.9.1 Chi-square test 92

4.9.2 Other non-parametric tests 94

5 SURVEYS 99

5.1 Introduction 100

5.2 Benefits and drawbacks of surveys 101

5.3 Goals and targeted users for survey research 102

5.4 Probabilistic sampling 103

5.4.1 Stratification 105

5.4.2 Response size 106

5.4.3 Errors 106

5.5 Non-probabilistic sampling 107

5.5.1 Demographic data 107

5.5.2 Oversampling 108

5.5.3 Random sampling of usage, not users 109

5.5.4 Self-selected surveys 109

5.5.5 Uninvestigated populations 109

5.6 Developing survey questions 111

5.6.1 Open-ended questions 111

5.6.2 Closed-ended questions 112

5.6.3 Common problems with survey questions 113

5.7 Overall survey structure 113

5.8 Existing surveys 115

5.9 Paper or online surveys? 116

5.10 Testing the survey tool 118

5.11 Response rate 119

5.12 Data analysis 120

6 DIARIES 125

6.1 Introduction 126

6.2 Why do we use diaries in HCI research? 127

6.3 Participants for a diary study 130

6.4 What type of diary? 132

6.4.1 Feedback diary 132

6.4.2 Elicitation diary 133

6.4.3 Hybrid feedback and elicitation diary 134

6.5 Data collection for the diary study 134

6.6 Letting participants know when to record a diary entry 136

6.7 Analysis of diaries 137

7 CASE STUDIES 143

7.1 Introduction 144

7.2 Observing Sara: a case study of a case study 145

7.3 What is a case study? 147

7.3.1 In-depth investigation of a small number of cases 147

7.3.2 Examination in context 147

7.3.3 Multiple data sources 148

7.3.4 Emphasis on qualitative data and analysis 149

7.4 Goals of HCI case studies 150

7.4.1 Exploration 150

7.4.2 Explanation 151

7.4.3 Description 152

7.4.4 Demonstration 154

7.5 Types of case study 156

7.5.1 Intrinsic or instrumental 156

7.5.2 Single case or multiple cases 156

7.5.3 Embedded or holistic 160

7.6 Research questions and hypotheses 161

7.7 Choosing cases 163

7.8 Data collection 164

7.8.1 Data sources and questions 164

7.8.2 Collecting data 165

7.9 Analysis and interpretation 167

7.10 Writing up the study 168

7.11 Informal case studies 170

8 INTERVIEWS AND FOCUS GROUPS 177

8.1 Pros and cons of interviews 178

8.2 Applications of interviews in HCI research 180

8.2.1 Initial exploration 180

8.2.2 Requirements gathering 184

8.2.3 Evaluation and subjective reactions 186

8.3 Who to interview 187

8.4 Interview strategies 189

8.4.1 How much structure? 189

8.4.2 Focused and contextual interviews 191

8.5 Interviews vs focus groups 192

8.6 Types of question 194

8.7 Conducting an interview 197

8.7.1 Preparation 197

8.7.2 Recording the responses 198

8.7.3 During the interview 199

8.8 Electronically mediated interviews and focus groups 203

8.8.1 Telephone 204

8.8.2 Online 204

8.9 Analyzing interview data 206

8.9.1 What to analyze 207

8.9.2 How to analyze 208

8.9.3 Validity 212

8.9.4 Reporting Results 212

9 ETHNOGRAPHY 217

9.1 Introduction 218

9.2 What is ethnography? 219

9.3 Ethnography in HCI 221

9.4 Conducting ethnographic research 224

9.4.1 Selecting a site or group of interest 225

9.4.2 Participating: choosing a role 227

9.4.3 Building relationships 230

9.4.4 Making contact 231

9.4.5 Interviewing, observing, analyzing, repeating, and theorizing 232

9.4.6 Reporting results 236

9.5 Some examples 237

9.5.1 Home settings 237

9.5.2 Work settings 238

9.5.3 Educational settings 239

9.5.4 Ethnographies of mobile and ubiquitous systems 240

9.5.5 Virtual ethnography 241

10 USABILITY TESTING 251

10.1 What is usability testing? 252

10.2 How does usability testing relate to traditional research? 254

10.3 Types of usability testing or usability inspections 256

10.3.1 Expert-based testing 256

10.3.2 Automated usability testing 258

10.4 User-based testing 260

10.4.1 Types of usability testing 260

10.4.2 Stages of usability testing 262

10.4.3 How many users are sufficient? 263

10.4.4 Locations for usability testing 264

10.4.5 Task list 268

10.4.6 Measurement 270

10.4.7 The testing session 271

10.4.8 Making sense of the data 274

10.5 Other variations on usability testing 275

11 ANALYZING QUALITATIVE DATA 281

11.1 Introduction 282

11.2 Stages of qualitative analysis 282

11.3 Grounded theory 283

11.4 Content analysis 285

11.4.1 What is content? 286

11.4.2 Why do we need to collect text or multimedia information? 286

11.4.3 Questions to consider before content analysis 287

11.5 Analyzing text content 289

11.5.1 Procedure 289

11.5.2 Identifying coding categories 290

11.5.3 Coding the text 292

11.5.4 Ensuring high-quality analysis 294

11.6 Analyzing multimedia content 300

12 AUTOMATED DATA COLLECTION METHODS 307

12.1 Exploiting existing tools 308

12.1.1 Web logs 309

12.1.2 Stored application data 315

12.2 Using software to observe and record 317

12.2.1 Web proxies 317

12.2.2 Instrumented software 321

12.2.3 Custom-built software 324

12.2.4 Handling stored data 327

12.2.5 Keystroke and activity loggers 328

12.2.6 Analyzing log files 329

12.3 Hybrid data collection methods 330

12.4 Automated interface evaluation 333

12.5 Challenges of computerized data collection 333

13 MEASURING THE HUMAN 343

13.1 Eye tracking 344

13.2 Physiological tools 350

13.2.1 Physiological data 351

13.2.2 Challenges in data collection and interpretation 356

13.3 Examples of physiological research in HCI 359

14 WORKING WITH HUMAN SUBJECTS 367

14.1 Identifying potential participants 368

14.1.1 Which subjects? 369

14.1.2 How many subjects? 371

14.1.3 Recruiting participants 373

14.2 Care and handling of research participants 376

14.2.1 Protecting participants 376

14.2.2 Informed consent 381

14.2.3 Institutional review boards 384

14.2.4 Potentially deceptive research? 387

14.2.5 General concerns 388

14.3 Online research 389

14.3.1 Appropriate topics for online research 389

14.3.2 Recruiting 389

14.3.3 Study design 391

14.3.4 Ethical concerns 391

14.3.5 Data collection 392

15 WORKING WITH RESEARCH PARTICIPANTS WITH IMPAIRMENTS 399

15.1 Introduction 400

15.2 How many participants? 401

15.2.1 Small sample sizes 401

15.2.2 Distributed research 401

15.2.3 In-depth case studies 402

15.3 Proxy users 403

15.4 Multi-Population Studies 404

15.5 Recruiting users through community partners 405

15.6 Pilot studies 407

15.7 Scheduling users with impairments 408

15.8 Documentation for users with impairments 409

15.8.1 Human subjects forms 409

15.8.2 Research documentation 410

15.9 Differing levels of ability 412

15.10 Bringing extra computer parts 413

15.11 Payment 415

Index 419