JCPSLP Vol 20 No 2 July 2018

fair to moderate for the three approaches. These results are consistent with previous research indicating that high rates of interrater agreement in prioritisation decisions are rarely achieved (Harding & Taylor, 2013) meaning that patients with similar needs can be given different priority ratings depending on which clinician prioritises the referral. The redesigned tool tested in this study showed improved interrater agreement when compared to the original tool (K = 0.5 vs 0.39). This may have been due to the use of decision rules based on explicit criteria in the redesigned tool (for example, the amount of service provided over previous days). The redesigned tool also had fewer prioritisation categories, which may have been another factor that contributed to higher agreement (Kreindler, 2008). However, agreement between clinicians using the redesigned tool was similar to that observed when clinicians used implicit prioritisation (K = 0.48). This finding challenges the assumption that prioritisation tools increase consistency of decision-making (MacCormick, et al., 2003). Health managers should not underestimate the ability of experienced clinicians to effectively prioritise their resources. It is unknown whether the clinical reasoning and judgement of less experienced clinicians, particularly new graduates, would reveal similar levels of interrater agreement and this is an area for further research. When the ratings were collapsed into the dichotomous categories of patients “seen that day” and those “deferred to tomorrow”, agreement improved both for clinicians using the redesigned tool (K = 0.61) and those using implicit prioritisation (K = 0.55). These results are consistent with previous findings that prioritisation tools may be most useful for supporting clinicians to decide which patients need immediate attention, and less useful for further sorting of less urgent patients (Harding, Taylor, Leggat, & Stafford, 2012). The findings also support previous suggestions that simpler prioritisation tools with fewer categories (in this case “must see today” or “deferred until tomorrow”) may be more effective than more complex protocols (Kreindler, 2008). Another argument for the use of prioritisation tools is to help translate organisational priorities to frontline care. A secondary aim of this study was therefore to determine whether changing prioritisation criteria influenced clinician behaviour. The original tool prioritised patients with a diagnosis of dysphagia. Recent literature, however, highlights risks associated with communication disorders for hospital inpatients, including avoidable adverse events and increased length of stay (Bartlett, et al., 2008). Changes in the tool intended to acknowledge these competing risks resulted in a small change in the observed distribution of priority ratings in alignment with intent, but this was not statistically significant. Data collected from prioritisation tools is sometimes used to assist health managers to allocate resources in times of high demand. This study builds on previous evidence that attaining a high level of reliability in prioritisation tools is challenging, and therefore variations in the reporting of priority ratings are likely to be influenced by clinicians’ application of the prioritisation criteria as well as by differences in the characteristics of the patients. Caution must therefore be exercised in regards to investment of time and resources into prioritisation tools, and the use of data derived from them. The results of this study may have been influenced by several factors. All clinicians participating in the research had been previously exposed to emerging evidence relating to risks associated with communication and cognitive

disorders, and several of the clinicians participating in the study also contributed to the design of the redesigned tool. Given the random allocation of clinicians to groups it is likely that these clinicians were evenly distributed within the groups, but there is a possibility that a preference for the new tool among some clinicians may have been a source of bias in the data. In addition, the scenarios were typical of “real-life” referrals, but were presented in the absence of other cues that could influence prioritisation decisions and make priority decisions clearer (for example, a corridor conversation with a nurse that provides additional information), or more complex (an assertive relative). The absence of such cues is therefore unlikely to lead to a bias in one direction more than another, but results may differ in a “real world” setting. This SLP department has continued to use the redesigned prioritisation tool as a guide to prioritisation decisions. The study raises questions on the benefits of using a prioritisation tool over relying on clinical judgement of experienced staff, but it may have a role in improving consistency of prioritisation for less experienced clinicians, particularly new graduates. There also continues to be strong support from senior management for the use of prioritisation tool to allocate resources. Over time the use of the tool has reduced for explicit prioritisation decisions, but its development has contributed to a common understanding of the terminology of “prioritised patients” and “standard patients” to discuss clinical caseloads, workflow, and to assist with allocation of staffing. Conclusion Interrater agreement among SLPs prioritising a hypothetical caseload was improved with a redesigned tool, but was still no more reliable than SLPs using implicit methods. These findings challenge the assumption that formal prioritisation tools lead to increased fairness and consistency. This study supports previous evidence that suggests that simple systems that differentiate patients with the most immediate needs from those who require routine care are likely to make a more useful contribution to the management of patient caseloads. Health managers should be aware of the limitations of prioritisation tools when devoting time and resources to their development and implementation. References Bartlett, G., Blais, R., Tamblyn, R., Clermont, R. J., & MacGibbon, B. (2008). Impact of patient communication problems on the risk of preventable adverse events in acute care settings. CMAJ : Canadian Medical Association Journal , 178 (12), 1555–1562. doi:10.1503/cmaj.070690 Brown, A. M., & Pirotta, M. (2011). Determining priority of access to physiotherapy at Victorian community health services. A ustralian Health Review , 35 (2), 178–184. Fleiss, J. (1981). Statistical methods for rates and proportions (2nd ed.). New York: John Wiley and Sons Ltd. Gauthier, R., Straathof, T., & Wright, S. (2006). The Ottawa Hospital Occupational Therapy Prioritization Guidelines. Occupational Therapy Now , 8 (6), 10–12. Geertzen, G. (2017). Inter-Rater Agreement with multiple raters and variables. Retrieved from https://nlp-ml.io/jg/ software/ira/ Harding, K. E., & Taylor, N. F. (2013). Triage in non- emergency services. In R. Hall (Ed.), Patient flow: Reducing delay in healthcare delivery (2nd ed., pp. 229–250). New York: Springer. Harding, K. E., Taylor, N. F., Leggat, S., & Shaw-Stuart, L. (2009). Triaging patients for allied health services: A

60

JCPSLP Volume 20, Number 2 2018

Journal of Clinical Practice in Speech-Language Pathology