BIB_08: Macefield, R. (2009). How to specify the participant group size for usability studies: A practitioner’s guide. Journal of Usability Studies, 5(1), 34-45.

In this article, Macefield offers a guide to usability practitioners to determine the optimal participants group size in usability studies in commercial environment. The author maintains that usability studies literature should be applied with the consideration of the context of the study. With this basis, Macefield discusses the optimal participants group size in three types of usability studies, studies related to problem discovery, comparative studies, and punctuated studies. In discussion about studies related to problem discovery, Macefield considers (1) the problem discovery level and context criticality that is related to the severity of the problems, and (2) the complexity of the study. In addition, he discusses the case of prototype testing, which usually requires fewer participants than non-prototype testing. He summarizes that the participants group size must be determined considering these contexts, and 5-10 participants are "a sensible baseline range" (p. 39). In the his discussion on comparative studies, Macefield emphasizes the statistic significance of the differences between groups, which is determined by the effect size and the sample size. He concludes that although the determination of the group size can be arbitrary, it is again beneficial to consider the context of the study, and 8-25 participants per group is a sensible range, while 10-12 is a good range. Macefield describes studies that studying the participants in chunks as "punctuated studies." This type of studies, according to Macefield, promote the spirit of the iterative design process, and can lead to cost and time savings. Finally, as in almost all the article published in Usability Studies, Macefield provides a "practitioners take-away" list that summarizes his discussion for application.

This is a very good article to turn to when one designs her/his usability study. However, one has to bear in mind that the audience of this article is practitioners. That means that it is inevitably and rightly so pragmatic. When used in academic context, these guidelines need to be supported with literature and statistics. That said, Macefield's emphasis on contextualizing usability studies offers great insight to both academic researchers and practitioners. (And in spite of myself, I realized that Nielsen is a sort of the God Father of usability studies.)

BIB_08: Faulkner, L. (2003). Beyond the five-user assumption: Benefits of increased sample sizes in usability testing. Behavior Research Methods, Instrument, & Computers, 35(3), 379-383.

Faulkner's study challenges the "5-user assumption" (testing five users can identify 80% of all the problems) widely accepted and applied in usability studies. Faulkner conducted a usability study with 60 users, and analyzed data from sampled groups of different sizes. He found that testing five users can identify 55%-100% of problems, and increasing the sample size to ten significantly increased the minimum problems identified to 95%. Therefore, he argues that 5-user assumption should not be readily accepted, and testing 10 users can increase the confidence of the data and thus the validity of the testing.

This piece is a good example of an empirical study in usability. It raises the question of the sample size in usability testing and validity. The results can be used as a ground for future research design. It also shows how taking a recommendation or rule out of context (the 5-user assumption) and accepting it readily as applicable across the board can hurt the validity of research.