Introduction

The study of human factors and its relevance for living labs

The study of “human factors” involves a scientific focus on the interaction between human individuals and systems with the goal of improving safety, performance, and user acceptability (Bergman, 2012; Weir et al., 2020). The term “system” can refer to specific tools, technologies, or tasks, a general working environment, or in some cases even a social, political, and/or economic environment (Weir et al., 2020). This broad scope and interest in wider systems distinguishes the study of human factors from related fields, such as ergonomics, usability, and user-centred design, although the terms are often used interchangeably (Norris, 2009). Human factors can be situated on the crossroads between engineering and psychology, since they involve both the design of tools and environments, as well as the cognitive and social functioning of users (Parker, 2015). While human factors were first studied in safety critical industries, such as defence and aviation, the approach has gained entry to a broader field of design and safety management in the past decade (Norris, 2009). In the meantime, a Systems Engineering Initiative for Patient Safety (SEIPS & & SEIPS 2.0) was developed with a human factors framework specifically tailored to healthcare; Holden et al., 2013). While research generally concerns itself with outcomes, human factors research has a strong complementary focus on processes. For example, SEIPS 2.0 focusses on the work system, processes (physical, cognitive, and social/behavioural), and outcomes (Holden et al., 2013). Instead of merely assessing whether a system improves efficiency or user outcomes, it is important that research also focuses on safety, ease of use, contextual fit, and implementation processes.

Human factors are of great interest to living labs since these innovation ecosystems aim to facilitate the development and optimization of innovative solutions and hold an intermediary position between the relevant stakeholders (for example, citizens, regulatory agencies, professional organisations, and developers). Although many definitions exist, the living lab approach can be seen as a methodology centred around the co-creation of innovations through end-user involvement and experimentation in real-life contexts (Dell’Era & Landoni, 2014; Ballon, et al., 2018). Living lab research generally follows an iterative cycle, including exploration, co-creation, testing and evaluation, along with implementation and upscaling (Ballon et al., 2018; Van Den Kieboom et al., 2019). While actual human factors studies are conducted in the testing phase, all four stages contribute to providing safer and user-friendlier products. Figure 1 provides examples of information relevant to human factors that can be collected in the different phases of living lab research.

Figure 1. Overview of the different stages of living lab research, along with relevant exemplary focal points for data collection in relation to human factors.

To be able to design for safety, performance, and acceptability, it is paramount to collect ample information about the environment in which an innovation is set to be implemented. The exploration phase allows for the collection of information on physical, practical & organizational circumstances, as well as current potential safety risks. Circumstances can refer to the actual working environment (for example, amount of space, internet access), or also the subjective experience of a given context, such as cognitive demands (for example, working in a stimulus-rich or noisy environment that influences performance) (Norris, 2009). When co-designing an innovation in collaboration with stakeholders, perceived risks, elements of high cognitive demand, and an innovation’s usability should be considered. In addition, the fit with existing processes, workflows, and workplace habits should be documented, since this is key to maximizing appropriate and long-term usage. At this point in the cycle, it could be useful to include a hierarchical task analysis, which is widely used as a human factors technique that describes an investigated activity through a hierarchy of goals, sub-goals, operations, and plans (Stanton, 2017). Such a detailed analysis of an innovation can guide further design and the development of test protocols.

The testing phase requires field tests to gain insights into prolonged usage, usage in real contexts with varying demands and circumstances, and latent conditions that are harder to identify in previous stages (Norris, 2009). However, Georges, Schuurman, Baccarne, and Coorevits (2015) have also proposed pre-field or usability trials, depending on the functional maturity of the innovation. A lab-based human factors study may not only account for technical difficulties related to lower functional maturity, but may also provide additional opportunities to document interactions and preferences. Finally, when an innovation is implemented in the field, monitoring and documentation should continue, since societal needs, challenges, and contexts may change quickly, which requires innovation adaptiveness.

Human factors research suggests that multiple stakeholders should be involved in all stages since the design of innovations is a dynamic process that involves continuous improvement and adaptation. The process is therefore not usually linear in nature, but rather more often allows flexible mobility across the stages through multiple iterations. The goal of living lab research including human factors is ultimately to provide innovations that are relevant, safe, reliable, and easy to use. Nevertheless, insights that are being collected can additionally be used to optimize procedures, shape support materials and training, or validate other process and implementation factors.

Human factors studies

Human factors studies and live tests allow researchers and practitioners to move away from basic assumptions through exploration and co-creation that provide insight into stakeholder perceptions and beliefs. Weir et al. (2020) observe a strong contrast between positive perceptions of technological innovation regarding safety versus data collected on errors and other usability problems in actual implementation. Several testing paradigms can be used to gain insights into human factors. In a “human factors study”, sometimes also referred to as a “usability study”, users are asked to interact with an innovation in simulated real-life circumstances (Bergman, 2012). Table 1 provides an overview of some prototypical characteristics of a human factors study. The design of such studies should always be tailored to the research questions and innovation of interest.

Table 1. Prototypical characteristics of a human factors Study.

A human factors study aims to provide insights into actual interactions with innovations, and accordingly, usage problems or errors, in challenging yet controlled situations that simulate real life. Having a diverse sample from the target population, including potentially vulnerable targets, allows organizations to design their innovations for their most vulnerable users (for example, those with low digital literacy), which will promote safety and usability. According to the condition or target population, a sample size of around 8 individuals is common in human factors studies, and appears sufficient to detect the vast majority of usability problems (Bolle et al., 2016). However, the required sample size can differ depending on the richness of the dataset, and on data collection methods used. Using a lab-based simulated context allows the observation of behaviours that occur widespread over time, or are difficult or unethical to evaluate in real life. For example, we can simulate that a patient has forgotten to take their medication and observe the resultant behaviour, while retaining an ethical basis for conducting the research. Human factors studies can be designed to be very challenging since providing a stringent test of an innovation allows to put rigorous safety precautions in place before it is implemented in practice. By providing a very stringent test of an innovation, rigorous safety precautions can be put in place before it is implemented in practice.

Reviews show that implementation methodologies vary greatly, and can include observations, interviews, focus groups, and questionnaires (Valdez et al., 2017; Weir et al., 2020). Most human factors studies implement multiple data collection methods. While the think aloud protocol is a hallmark in the human factors methodology toolkit, the number of studies specifically implementing the “think aloud” paradigm or a “task analysis” remains limited (Valdez et al., 2017; Weir et al., 2020). In the think aloud protocol, also known as “verbal protocol analysis”, participants are asked to perform a task and simultaneously verbally report everything that goes through their mind, unedited, and without evaluation. This protocol provides insight into the cognitions and processes that underlie behaviour. Research generally shows that merely reporting thoughts does not influence a person’s cognitive process, however, being asked about motivations (that is, why individuals are performing actions) could interfere with their natural processes, since it requires self-interpretation (Güss, 2018). The think aloud data is recorded and subsequently qualitatively analysed and coded to extract themes relevant to the study’s particular research questions. An inductive qualitative analysis is typically preferred since it may be difficult to capture the variability of thought processes relating to task interactions in a-priori models and codebooks. Triangulation, or combining several methods or sources of information, can improve trustworthiness in the findings. Thus, we found that an approach combining thinking aloud data with observation checklists or survey and interview data may be preferred (Aitken et al., 2011; Güss, 2018).

The results of human factors studies can help organizations formulate concrete suggestions to improve the design of innovations and community services. However, the impact of human factors studies on the innovations themselves under investigation has often been insufficiently demonstrated or documented (Carayon, 2019; Weir et al., 2020). We suggest that maintaining a good report structure for design and end-user iterations following human factors studies will allow researchers and organizations to better document the effects of their considerable efforts, and also monitor whether further optimizations are warranted.

Healthcare as an exemplary context

Human factors and user-centred design can have a particularly large impact in the field of healthcare, where medical and pharmaceutical dispensing errors, for example, can cause serious, yet preventable, harm (Carayon & Hoonakker, 2019; Weir et al., 2020). Healthcare is a complex and dynamic field with many stakeholders (hospitals, pharmacies, patients, companies, families), whose needs and goals can be very dissimilar. Designing healthcare products, such as medication packaging, can therefore be challenging, and potentially benefit from several iterations of end-user involvement and research that optimizes the design and implementation. In line with this, the UK National Health Services (NHS; Department of Health Human Factors Reference Group, 2012) and U.S. Food and Drug Administration (FDA) have supported and encouraged the exploration of human factors (U.S. Department of Health and Human Services, Food and Drug Administration, Center for Devices and Radiological Health, & Office of Device Evaluation U.S. Department of Health and Human Services, 2016). The following section will describe four exemplary human factors studies from the field of healthcare.

Research Design

While human factors studies have a common goal, implemented research designs can differ depending on the type of innovation, implementation context, or sample. Table 2 provides concrete examples of what a human factors study can look like, based on four healthcare innovation cases executed by Living & Care Lab (LiCalab). LiCalab is a living lab situated in Flanders, Belgium, which primarily focuses on supporting companies and organisations in the health and heath care sector. LiCalab therefore co-creates, evaluates, and tests innovative solutions with end-users.

The authors of this paper were actively involved in each of the listed cases. For the first case, the team designed the study, performed it in Belgium, and analysed the data. For the other international cases, LiCalab collaborated with other living labs abroad. The study’s design and data collection methodology was discussed in detail upfront with partner organisations. Only for case 3, the LiCalab team additionally supported the other living labs on-site, while the other studies were all set up from a distance. Depending on the goal of the study and the target group, various study components were selected to simulate challenging situations that occur in everyday life. The human factors study components could also be combined with other living lab services (for example, co-creation), but these are not included in the table.

Table 2. Four case design examples in which human factors studies were implemented in living lab context

In the first case, participants were asked to go 3 days using the smart medication package multiple times per day. Two conditions were designed based on how the product could be implemented in practice, which only differed in their instructions. With the first condition participants only received a folder explaining how to use the smart medication package, while the second condition received additional verbal instruction and a demonstration of tablet removal. Participants were subsequently asked to interact with the smart medication package while performing normal daily activities, such as reading a newspaper article, watching a video clip (simulating watching television), and talking with someone. Their behaviour was observed and documented with the help of a codebook. In addition to normal usage, participants were also explicitly asked to make certain errors so they could experience and comment on the resulting sequence of events on the smart medication package and app. For instance, participants were asked to mimic forgetting to open a medication slot or opening an incorrect slot, so that they could experience and evaluate the Resulting reminders and notifications from the medication package and accompanying app, which were designed to support correct medication intake. Observational data was supplemented with a self-report questionnaire on usability and user preferences.

The second case concerns a multi-country design in which two alternative packaging designs were compared. Like case 1, participants were asked to mimic multiple days of removing tablets from the package, while their interactions with the package were observed. In a second task, they were presented with a used medication wallet and were asked to think aloud about whether any errors were made with it, and what the user should do about it. Finally, a questionnaire of participants provided further input regarding their experiences and preferences.

The third and largest international study we did, concerned the visual design of medication packaging. It consisted of a medication retrieval task with 2 conditions that varied with medication stacking, in which behaviours were observed using a codebook (a subsequent questionnaire also assessed their experiences in more depth). We performed a colour sorting task to assess possible cross-cultural differences in how colours are perceived and interpreted. In the questionnaire, participants were also presented with opposing word pairs (for example, beautiful vs. ugly, strong vs. weak) based on Osgood’s semantic differential (Osgood et al., 1957) to explore the connotative meaning of the package design. The design was further discussed and evaluated in patient focus groups and expert panels.

In the fourth exemplary case, neurology patients interacted with a web-based platform while thinking aloud. After receiving a folder with instructions and their login details to access the secure platform, they were presented with situations and questions that they could encounter in real life, and for which they could use the platform. They performed the task together with an informal caregiver, as previous results from co-creation sessions showed that these older or disabled patients would often rely on their support network to help them use such a platform. Data collection consisted of observations as well as self-reported data from a questionnaire.

For all of the four cases above, the project team decided on using human factor studies at an initial kick-off meeting. The safety of participants was considered paramount, and as the health products were still in a minimal viable technical phase, the human factor studies helped them to gain insights into both potential opportunities and pitfalls. To be more specific, observing which aspects of product use led to usage errors in these cases, allowed the respective companies to optimize design. The results and reports of these four cases all had an impact on the design or implementation circumstances of these innovations. Documented changes following the impact of having conducted human factors studies consisted of making a choice between two competing designs, changing terminologies, selecting more appropriate colours, and adapting usage instructions. Two of the cases above included data collection that was performed in multiple countries, which can provide added value for the organizations as customs, perceptions, and opinions can vary across cultures (De Witte et al., 2021). Organizations often aim to launch their product internationally, yet first need to make sure that designs are suitable for a wide range of end users.

Conclusion

Organizations are developing and upscaling new innovations and technologies at an unprecedented pace. However, it is important that these innovations be adapted to the intended user and meet quality standards in terms of safety, performance, ease of use, and contextual fit. Human factors research in healthcare and other fields is still in its infancy, yet it allows organizations to properly assess these aspects of innovations and, if need be, improve their quality. Carayon and Hoonakker (2019) state that, “If we want human factors to be taken seriously into account, we should not be shouting from the sideline, but get actively involved in the design and implementation of health IT, and evaluate the impact of our human factors methods and principles on the technology in practice”. Living labs can play a key role in making sure innovations are safe, efficient, and designed with users in mind.

The current paper aimed to inform the field on how human factors methodologies can be designed and what role they can play in an iterative development cycle. While certain hallmark human factors techniques, tasks, and data collection methods exist, the design of a human factors study will nevertheless always remain a very individual and tailored process given that innovations, circumstances, and targeted end users vary. The study of protocols using a multi-method approach to mimic stringent real-life circumstances and gain insights into error-prone processes can provide important insights for organizations and governments, thereby improving the potential for more responsible, better and safer services for the end user.

---

References

Aitken, L.M., Marshall, A., Elliott, R., & McKinley, S. 2011. Comparison of “Think Aloud” and Observation as Data Collection Methods in the Study of Decision Making Regarding Sedation in Intensive Care Patients. International Journal of Nursing Studies, 48(3): 318-325. DOI: https://doi.org/10.1016/j.ijnurstu.2010.07.014

Ballon, P., Van Hoed, M., & Schuurman, D. 2018. The Effectiveness of Involving Users in Digital Innovation: Measuring the mpact of Living Labs. Telematics and Informatics, 35: 1201-1214. DOI: https://doi.org/10.1016/j.tele.2018.02.003

Bergman, E. 2012. Introduction to Human Factors. Journal of Diabetes Science and Technology, 6(2): 229-230. DOI: https://doi.org/10.1177/193229681200600202

Bolle, S., Romijn, G., Smets, E.M.A., Loos, E.F., Kunneman, M., & Van Weert, J.C.M. 2016. Older Cancer Patients’user Experiences with Web-Based Health Information Tools: A think-aloud study. Journal of Medical Internet Research, 18(7). DOI: https://doi.org/10.2196/jmir.5618

Brooke, J. 1986. SUS: A “Quick and Dirty” Usability Scale. In P. W. Jordan; B. Thomas; B. A. Weerdmeester; A. L. McClelland (eds.). Usability Evaluation in Industry. Taylor and Francis.

Carayon, P. 2019. Human Factors in Health(care) Informatics: Toward Continuous Sociotechnical System Design. Studies in Health Technology and Informatics, 265: 22-27. DOI: https://doi.org/10.3233/SHTI190131

Carayon, P., & Hoonakker, P. 2019. Human Factors and Usability for Health Information Technology: Old and New Challenges. Yearbook of Medical Informatics, 28(1): 71-77. DOI: https://doi.org/10.1055/s-0039-1677907

Dell’Era, C., & Landoni, P. 2014. Living Lab: A Methodology between User-Centred Design and participatory design. Creativity and Innovation Management, 23(2): 137-154. DOI: https://doi.org/10.1111/caim.12061

Department of Health Human Factors Reference Group. 2012. Department of Health Human Factors Reference Group. Interim Report. https://www.england.nhs.uk/wp-content/uploads/2013/11/DH-rep.pdf

De Witte, N.A.J., Adriaensen, I., Broeckx, L., Van Der Auwera, V., & Van Daele T. 2021. Cross-cultural Differences in User-Centred Research: an international living lab study. Health Informatics Journal. DOI: https://doi.org/10.1177/14604582211038268

Georges, A., Schuurman, D., Baccarne, B., & Coorevits, L. 2015. User Engagement in Living Lab Field Trials. Info, 17(4): 26-39. DOI: https://doi.org/10.1108/info-01-2015-0011

Güss, C.D. 2018. What is Going Through your Mind? Thinking aloud as a method in cross-cultural psychology. Frontiers in Psychology, 9: 1-11. DOI: https://doi.org/10.3389/fpsyg.2018.01292

Holden, R.J., Carayon, P., Gurses, A.P., Hoonakker, P., Hundt, A.S., Ozok, A.A., & Rivera-Rodriguez, A.J. 2013. SEIPS 2.0: a human factors framework for studying and improving the work of healthcare professionals and patients. Ergonomics, 56(11): 1669-1686. DOI: https://doi.org/10.1080/00140139.2013.838643

Nemeth, C.P. 2004. Human factors methods for design: Making systems human-centered. CRC press. Preface XiV

Norris, B. 2009. Human Factors and Safe Patient Care. Journal of Nursing Management, 17(2): 203-211. DOI: https://doi.org/10.1111/j.1365-2834.2009.00975.x

Osgood, C., Suci, G., & Tannenbaum, P. 1957. The Measurement of Meaning. University of Illinois.

Parker, S.H. 2015. Human Factors Science: Brief History and Applications to Healthcare. Current Problems in Pediatric and Adolescent Health Care, 45(12): 390-394. DOI: https://doi.org/10.1016/j.cppeds.2015.10.002

Stanton, N.A. 2017. Human Factors Methods: A Practical Guide for Engineering and Design. CRC Press.

U.S. Department of Health and Human Services, Food and Drug Administration, Center for Devices and Radiological Health, & Office of Device Evaluation U.S. Department of Health and Human Services. 2016. Applying Human Factors and Usability Engineering to Medical Devices: Guidance for Industry and Food and Drug Administration Staff. https://www.fda.gov/media/80481/download.

Valdez, R.S., McGuire, K.M., & Rivera, A.J. 2017. Qualitative Ergonomics/Human Factors Research in Health Care: Current state and future directions. Applied Ergonomics, 62: 43-71. DOI: https://doi.org/10.1016/j.apergo.2017.01.016

Van Den Kieboom, R.C.P., Bongers, I.M.B., Mark, R.E., & Snaphaan, L.J.A.E. 2019. User-driven Living Lab for Assistive Technology to Support People with Dementia Living at Home: Protocol for developing co-creation-based innovations. JMIR Research Protocols, 8(1): 1-9. DOI: https://doi.org/10.2196/10952

Weir, N.M., Newham, R., & Bennie, M. 2020. A Literature Review Of Human Factors and Ergonomics within the Pharmacy Dispensing Process. Research in Social and Administrative Pharmacy, 16(5): 637-645. DOI: https://doi.org/10.1016/j.sapharm.2019.08.029