JCPSLP Vol 20 No 2 July 2018

referrals received in both acute and subacute inpatient settings. Using the allocated method, the clinicians were asked to prioritise the nine patients for each of the 5 days. Given that the prioritisation tools had varying numbers of prioritisation categories, responses were standardised by asking participants to allocate the patients on each day’s list into three priority groups: 1. Must see today (the 3 highest priority patients); 2. Plan to see later in the day (the next 3 priority patients); 3. Deferred for consideration in the context of the following day’s patient list (last 3 patients) Statistical analysis The interrater reliability within the three groups was measured using the kappa statistic for multiple raters, using an online calculator (Geertzen, 2017), selected according to theoretical principles described by Fleiss (1981). Kappa measures (K) the agreement between raters beyond what would be expected by chance alone. As a guide to interpretation, kappa scores of 0.2 to 0.4 were considered fair, 0.41 to 0.6 considered moderate, 0.61 to 0.8 were considered substantial and above 0.8 was considered almost perfect (Koch & Landis, 1977). It was hypothesised that disagreement in priority decisions for patients “deferred until tomorrow” were likely to be higher than for disagreements in priority decision for patients “must see today”. A sensitivity analysis was conducted to determine agreement between clinicians when the patients were grouped dichotomously into “must see today” (priority groups 1 and 2) and those to be “deferred until tomorrow” (priority group 3). To determine whether the redesigned tool changed SLPs’ behaviour in rating the priority of patients with dysphagia, Chi square statistics were used to compare the proportion of cases with a primary diagnosis of dysphagia selected for deferral to the following day for clinicians using the original and redesigned tools. Results Interrater reliability Approximately one-third of the clinical scenarios received the same priority rating from all clinicians in the group. One-third of patient cases had agreement from all but one clinician, and the remaining third had a maximum of three of five clinicians in agreement, or two of four clinicians in agreement in the case of the original tool (Table 3). Agreement between clinicians using redesigned tool was moderate ( K = 0.50), and was similar to the group of clinicians who used clinical reasoning alone ( K = 0.48). Observed agreement between clinicians who used the original tool was fair ( K = 0.39).

Table 3. Agreement between raters within each of the groups

Group 3: No tool n = 5 raters

Group 2: Redesigned tool n = 5 raters

Group 1: Original tool n = 4 raters

0.39

0.50

0.48

Kappa (K) Highest proportion of raters in agreement 100% agreement (n cases) 75–80% agreement* (n cases) 50–60% agreement** (n cases) <50% agreement*** (n cases)

15 (33%)

16 (36%)

16 (36%)

16 (36%)

15 (33%)

13 (29%)

14 (31%)

12 (27%)

13 (29%)

NA

2 (4%)

3 (7%)

When priority groups were recoded dichotomously into those “must see today” and “deferred until tomorrow”, agreement was higher for those using the redesigned tool ( K = 0.61) and those using clinical reasoning alone ( K = 0.55), but remained fair for those using the original tool ( K = 0.32). Relative priority of dysphagia patients The redesigned tool was designed so patients with dysphagia were not routinely prioritised over patients with other conditions. A slightly higher proportion of dysphagia patients were deferred to the next day by clinicians using the redesigned tool (32% of observations) compared to those using the original tool (27%). Similarly, a smaller observed proportion of communication/cognition referrals for those using the redesigned tool were deferred (43%) compared to the original tool (48%). These differences, however, were not statistically significant at p = 0.24 and p = 0.34 respectively (Table 4). Discussion This study assessed the interrater reliability of experienced SLPs prioritising the same hypothetical scenarios using two different priority tools and implicit prioritisation (without the support of a tool). Agreement was measured using the kappa statistic and indicated that agreement ranged from * 4/5 raters in Groups 2 and 3, 3/4 raters in Group 1. ** 3/5 raters in Groups 2 and 3, 2/4 raters in Group 1. *** 2/5 raters in Groups 2 and 3 (e.g., 2 ratings in Priority 1, 2 in priority 2, and 1 in priority 3). NA to Group 1.

Table 4. Proportion of referrals with a primary diagnosis of dysphagia (n = 24 cases) and communication/ cognition disorders (n = 14 cases) allocated to same day or next day service

Original tool (n = 4 clinicians)

Redesigned tool (n = 5 clinicians)

Significance (Chi square)

Dysphagia referrals (n = 24) Same day (priority groups 1 and 2) (number, % of observations) Next day (priority groups 3) (number, % of observations) Total observations (cases x clinicians) Communication/cognition referrals (n = 14) Same day (priority groups 1 and 2) (number, % of observations) Next day (priority groups 3) (number, % of observations) Total observations (cases x clinicians)

Not statistically significant p = 0.24

81 (68%) 39 (32%) 120

70 (73%) 26 (27%) 96

Not statistically significant p = 0.34

40 (57%) 30 (43%) 70

29 (52%) 27 (48%) 56

59

JCPSLP Volume 20, Number 2 2018

www.speechpathologyaustralia.org.au