The short-term usability of the RELab tenoexo, an RHO to assist hand function in people with SCI, was evaluated with a mixed methods approach. With a total of 15 users, the RHO was evaluated in a study design employing both qualitative and quantitative methods to holistically assess usability. We discuss our main findings in dedicated sections focusing on specific usability attributes to provide a comprehensive review of the RELab tenoexo and generate generalizable insights relevant for the comparison with similar RHO.

Immediate functional benefit for people with severe hand impairment

A mean improvement in hand function above the clinically important difference on the ARAT [33] was found in group B, with three participants achieving at least 9 additional points and one being slightly below clinically important difference. This underlines a significant immediate functional benefit from the active assistance of the RELab tenoexo. An improvement in the “Grasp” and the “Grip” subscale of the ARAT could be observed for all participants of group B, while most participants’ scores in the “Pinch” stayed similar or even decreased with the RHO. This can be explained by the functional principle of the RELab tenoexo, as it supports mainly gross grasping motion and provides additional grip strength rather than assisting dexterity. Similar observations were made by Yun et al. [35] and Radder et al. [36] in their evaluations of RHO for target users with hand impairments. In both studies, participants also faced difficulties in fine motor tasks while achieving higher function in tasks requiring a power grasp when using the respective RHO. The decreased ARAT score observed for one of our study participants (P13, − 10 points), compared to the unassisted condition, further highlights this limitation of the RELab tenoexo design. Without the RHO, the participant was able to pinch small objects, as well as lift and manipulate larger objects using a tenodesis grasp and compensatory movements (indicated also by the highest GRASSP score in group B). With the current design of the RHO and without a specific personalization in terms of device size and fit, these strategies were restricted such that the ARAT score was lower with the device.

Four participants of group A discontinued the study due to too weak proximal upper limb function. These insights indicate that the RELab tenoexo is most beneficial for people with more severe distal (hand) impairment, low tone, and residual proximal upper limb function. It is yet to be evaluated if combining the RHO with passive gravity compensation mechanisms or wearable robotic devices for the proximal joints could widen the target population. Similarly sleek and lightweight solutions such as by Georgarakis et al. [37] or O’Neill et al. [38] could combine well with the portable design of the RELab tenoexo.

Furthermore, the RELab tenoexo works best with persons that experience a flaccid paralysis of the hand. For individuals with a strongly increased muscle tone or spasticity in the hand muscles, the extension force provided by the RELab tenoexo is not sufficient to open the fingers to a fully extended position. These findings have been further demonstrated and supported in a device variation tailored to the pediatric population [39].

The subjective data from the SUS and CUQ correlates with the ARAT scores, as participants rated the usability of the RHO higher if there was a clearer functional benefit and vice-versa. More specifically, the participants of group B agreed that they would like to use the RHO frequently, mainly because the device enabled them to perform the tasks more easily, quickly, and with more voluntary control of their hand function.

The RELab tenoexo is perceived as comfortable, ergonomic, and safe

Among other aspects, we specifically studied the human-robot interaction in terms of comfort, ergonomics, and safety with a range of outcome measures in this work. First and foremost, no major discomfort required a premature termination of any testing session during a total usage time (i.e., sum over all participants) of more than seven hours (RHO powered, donned). In terms of ergonomics, as rated by dedicated CUQ items, the RHO weight was considered adequate (Q18), the closing and opening speed were good (Q17), and the guided motion was comfortable (Q20). The dimensions and weight of the RELab tenoexo also fulfill the RHO design criteria defined by Boser et al. [23], who established RHO requirements from the perspectives of target users with hand impairments and clinicians. Further, we can infer from the gross movement subscale of the ARAT that the weight and dimensions of the RHO are nonrestrictive, as all participants were able to achieve the same freedom of motion with and without the RHO. Still, a few participant statements were in favor of a further reduction in weight (“Less weight would improve handiness, especially when wearing but not using the RHO.”)

The absence of major adverse events arguably indicated the general physical RHO safety. The individual adverse effect of overstretching for one participant derives from the physiological shortening/contraction in hand muscles and tendons after years of reduced use. In these cases, a RHO to force hand flexion and extension is not a viable solution to assist hand function due to the risk of injuries and the restriction of trained compensatory movements (e.g., tenodesis grasp) that could better support ADL. The current fixed-wrist design of the RELab tenoexo restricts trained compensation movements such as the tenodesis grasp and increases the finger rigidity in a flexed position, requiring a higher force from the RHO to extend the fingers. A more adjustable or flexible wrist design as proposed by Dittli et al. [40] could allow for more versatile support from the RHO.

Donning is key for ease of use and device adoption

While ergonomics and safety are absolute must-have criteria for any wearable robotic device, the ease of use is arguably one of the most influencing factors determining actual daily use and most frequently related to overall usability [4, 41,42,43].

The ease of use was assessed subjectively by the SUS (Q3, Q4, Q8) and objectively by the donning time. In combination with our qualitative results, we could assess the ease of use of the RHO in two dedicated usage phases: (i) preparation and donning and (ii) active operation.

While the overall ease of use was rated positively, lower ratings of SUS items Q4 and Q8, CUQ item Q14, and user feedback from the semi-structured interviews indicated that the donning process (i), which required assistance, was considered cumbersome and a limitation to frequent and independent daily use (“It is a disadvantage that I can not don the RHO independently.”). The observation that the donning process and physical attachment system are critical to device adoption has also been made in other studies [13, 36, 39, 44, 45]. However, the donning interface and process of RHO is rarely reported and analyzed in detail. Based on the results of this study, the overall complexity of RHO (i.e., number of components and donning steps) and the donning time should be reduced to increase the ease of use and intuitiveness, indicated by the high rating of SUS item Q2 and subjective feedback, respectively. An average donning time of 4 min 55 s was recorded across all testing sessions when performed at an eased pace. Similar donning times were recorded in a previous clinical utility study with adapted version of the RELab tenoexo for pediatric target users [39] and a dedicated usability study investigating the manual-guided setup of the RELab tenoexo on a mock-user (neurologically intact hand) by untrained persons in a caregiver role [41]. Accordingly, a donning time of approximately 300 s is the current benchmark for the RELab tenoexo. Yurkewich et al. [44] found that an average donning time of 180 s for their RHO was perceived as cumbersome, indicated by a low rating of ease of use in the QUEST usability questionnaire. Accordingly, target users seem to expect this process to be done more efficiently, especially when performed by an assisting helper. Most participants stated that optimally, they would be able to don the device by themselves, imposing a major challenge, particularly for glove-based systems intended for users with bilateral hand impairment. Radder et al. reported further usability limitations that need to be considered in the design of the physical attachment system for RHO, such as heat building and sweating [36]. In the semi-structured interviews, many participants provided valuable suggestions to improve the physical attachment system of the RHO to fit their individual needs better (“A donning aid would be helpful.”).

In terms of active device operation (ii), participants were provided push buttons as the only option to trigger RHO motion. While other input modalities such as electromyography (EMG) [35] or force myography [46] may provide more natural, direct control for RHO users [45], buttons appear to offer the most robust and simple solution as of today [4]. All participants quickly learned how to operate the RHO with the button, as reflected by the high ratings in SUS items Q7 and Q10. In the semi-structured interview that followed the second session, several participants agreed that a selection of additional intention detection systems (e.g., voice control or a smartphone application) could further increase the RHO usability.

Other phases of the usage cycle such as doffing, storage, or maintenance (e.g., cleaning and disinfecting, recharging of batteries) were not specifically investigated in this study, although being an integral part of potential daily use. For future studies, these aspects should be considered in the evaluation of ease of use.

Individual needs call for tailored solutions

One of the main observations and learnings of our study sessions was the astonishing diversity of needs (and wishes) of our participant sample. Within our group of users with very similar impairment types (e.g., GRASSP score, injury level, time-post-injury), not all would be willing and able to use the RELab tenoexo due to their strongly differing personal and environmental context (e.g., expectations, living situation, previous experiences). As for any assistive device, the match between person and technology has to be carefully evaluated and optimized in order to achieve technology acceptance [47]. In the case of RHO, tailoring could solve usability issues with functional benefit and comfort that were identified in our analysis, and improve the match to the individual target user needs. Our results indicate that critical aspects which should be tailored to the user are device size (“I would need a more flexible adaptation to hand size”), aesthetics (“The RHO looks sleek.”; “I don’t like the looks of the RHO”), device functionality (“It really feels like a firm grasp.”; “Fingertips should close more.”), and user interfaces. In particular, the limited size availability and adjustability of the RELab tenoexo led to the discontinuation of three participants and may limit the ergonomics, user comfort, and safety (e.g., due to misalignment in the force transmission to the hand). We thus suggest that, if possible, RHO should be specifically tailored for individual users instead of aiming for a one-fits-all concept. Tailoring of a RHO could be achieved with modular systems, providing a range of solutions from which the user can choose from (e.g., physical attachment systems, different combinations of intention detection systems).

Gained value from the mixed methods approach

The most important contribution of the mixed methods approach is likely the large number and variety of qualitative user statements collected from two sessions in this work (e.g., improvement suggestions, critical feedback) and the rich data to understand if the targeted problem has been appropriately addressed. The quantitative outcome measures largely covered the predefined core attributes. However, the thematic analysis showed that these attributes were not sufficiently broad to cover all aspects of usability. Thus, the analysis uncovered five additional attributes relevant to the target users. These allowed us to gain important insights into the user’s perception of the usability of the RHO (e.g., in terms of aesthetics, complexity, or learnability), which would have been missed in an evaluation using purely quantitative measures.

The value of mixed method evaluations and thematic analyses has been highlighted before for other rehabilitation devices than RHO. Bhattacharjya et al. [19] and Warland et al. [18] used a mixed methods approach (e.g., quantitative surveys, semi-structured interviews) to evaluate devices for upper limb therapy after stroke. They claim that the approach allowed capturing a holistic picture of the system usability [18] and that the qualitative findings strengthened the validity of quantitative findings or, when different, provided possible explanations for the results found [19]. In both studies, the qualitative data were analyzed regarding specific device attributes rather than general usability attributes. We believe that our analysis based on generalizable usability attributes extracted from literature and identified during post-processing enables more directly transferable results to other RHO.

Design limitations of the RELab tenoexo and improvement suggestions

Besides the general insights relevant for other RHO designs, residual usability limitations specific to the evaluated RELab tenoexo were observed that need to be addressed in the next design iteration.

Even though the majority of our study participants appreciated the grasp strength provided by the RELab tenoexo, a further increased grasp force would be desirable, as indicated by CUQ items Q15 and Q16, as well as by the discontinuation of two participants due to insufficient opening/closing force for spastic hands. Increasing the provided assistive force in both flexion and extension could enlarge the target user group and further increase the functional benefit from the RHO. However, an excessively increased force on the users hands may negatively impact the ergonomics and comfort of the RHO, in which case alternative solutions such as reduction of spasticity via medication could be investigated. Other design features of the RHO, such as the thumb opposition mechanism and the fixed wrist position, might need to be refined and optimized for individual needs and capabilities. Adjusting the thumb opposition should require less effort (e.g., by increasing the handle size or adding a loop to it). The wrist should be adjustable in position or flexible not to restrain residual function (e.g., wrist extension for a tenodesis grasp).

Several robustness issues resulting in technical failures of the actuation system, control, and mechanics of the hand module were recorded. The RELab tenoexo hardware was also used for other studies in parallel, which might have increased the wear of the device and thus prevented drawing a conclusion as to how the use recorded in this study led to the technical issues. However, to enable long-term use of the RHO in potentially unsupervised settings, these failures have to be further investigated and addressed.

Methodological considerations and limitations

In this study, we evaluated the short-term usability as only one step of an iterative design process. Therefore, it cannot be concluded whether the selected approach generates a concrete benefit and whether the identified usability issues can be successfully addressed in the next design iteration. Further, a learning effect on the RHO might influence some aspects of usability such as the functional benefit and easy of use. A similar protocol should be optimally conducted repeatedly after completed design iterations to optimally follow a user-centered design approach. Also, further testing in the intended usage environment (i.e., outside of the controlled laboratory environment) will be required to investigate the device uptake by the target users as well as other technology stakeholders. Yet, the comprehensive mixed methods evaluation applied in this study allowed us to capture a broad picture of the short-term usability of the RELab tenoexo and identify directions for future developments. The presented approach to usability evaluation could potentially be applied for other RHO with minor adaptations of the protocol to specific evaluation focuses and target user groups. Such studies could help to challenge insights of this study and identify global trends and gaps in RHO development.

Even though the assessment with the ARAT generated valuable insights, it is important to state that the clinical test has not been validated for individuals with an SCI. Further, the ARAT does not evaluate bimanual tasks, while most daily activities require the use of both hands simultaneously. Alternatively, more SCI-specific hand function tests such as the JTHFT or the TRI-HFT or assessments involving bimanual tasks such as the BeBiTT [48] would allow for more reliable characterization of hand function. Furthermore, results from interviews and think-aloud methods collecting semi- or unstructured user statements have to be interpreted with care, as they are prone to primarily capture usability issues due to the human’s natural negativity bias [49]. Users are more likely to point to things they think can be improved rather than things they find good and satisfying. Lastly, our analysis might be biased due to the small number of participants of group B or by statements of individual participants if others did not comment on specific attributes. Particularly qualitative evaluations are often biased by several personal factors such as expectations or learning effects, which may have influenced, e.g., the decrease in SUS scores from session 1 to session 2 in some participants in this study. The best way to mitigate these effects would be to have multiple evaluation sessions and more users to reduce potential bias in overall results due to individual users’ opinions from single sessions.