JCPSLP Vol 18 no 2 July 2016

• focus of site content (population, age); • credibility indicators (including the presence of citations, date of posting / update, and domain type). These metadata were only collected for the page directly linked to the Google search result. The full coding schema has been included in the Appendix. Findings underwent descriptive statistical analysis in Excel. Inter-rater coding was performed by the second author on 20% of the sample. Cohen’s Kappa statistic was used to calculate inter-rater reliability (Hallgren, 2012), with an average Kappa score of 0.42 (moderate strength). Kappa values for each criteron are presented in Table 2.

Table 1. AAC-related search terms containing the words communication or communication device , in order of search frequency

Search term

Estimated monthly

Estimated daily (Google)

Containing the term “communication”

augmentative communication

1560

43

facilitated communication

1560

43

picture exchange communication system

1560

43

Table 2. Inter-rater reliability for each criteria, calculated using Cohen’s Kappa

communication board

1200

33

Criteria

Kappa value

Strength

communication devices

1200

33

Relevance

.42

Moderate

Containing the term “communication device”

Purpose

.23

Fair

augmentative communication device

168

5

Citations

.73

Substantial

tech talk communication device

60

2

Recency

.47

Moderate

go talk communication device

60

2

Domain

.96

Near perfect

dynavox communication device

60

2

Location

.45

Moderate

springboard communication device

48

1

Population

.41

Moderate

Diagnosis-specific terms

Age

.48

Moderate

aphasia communication

24

1

Average

.50

Moderate

autism communication

252

7

Note: Cohen’s Kappa compares the observed agreement against an agreement level that could be expected by chance. A score of 1 would indicate perfect agreement between raters, while a score of –1 would indicate perfect disagreement. Any scores between 0 and 1 indicate agreement at better-than-chance levels (Hallgren, 2012). Results Search results were analysed as a group, and then in separate groupings according to their keyword, purpose, and domain types. The analyses are presented below. Viability, relevance and format The total search set was analysed according to link viability and relevance of the page to AAC (see table 3). Nine of the resulting links (3%) led to expired or password-protected pages, and were excluded from analysis. Of the remaining 291 sites, 204 (70%) were judged “mostly relevant”, 32 (11%) were judged “somewhat relevant” (for example, listing AAC strategies among other communication strategies or interventions), and the remaining 55 were found to have no relevance to AAC (19%). For four search terms (augmentative communication, augmentative communication device, and communication board ) all viable top-20 results were judged “mostly relevant”; product-/technique-specific search terms also yielded high numbers of mostly relevant results. By comparison, the search terms communication app, AAC, communication device resulted in high numbers of non-relevant pages (12, 11 and 9, respectively), typically concerning mainstream computing. Diagnosis-specific searches tended to result in somewhat-relevant sites that mentioned AAC among a range of other diagnosis-specific intervention techniques and strategies, for example, the use of gestures in combination with other language strategies for adults with aphasia.

cerebral palsy communication

24

1

Additional terms

AAC

39 720

1 103

communication app

No data available

No data available

Search strategy One factor that features heavily in a search engine’s algorithms is the individual searcher’s prior search behaviours and browsing history. As a consequence, a parent of a child with newly diagnosed autism and an experienced AAC clinician may in fact receive different results from the same keyword search. To ensure results were not influenced by the researcher’s own search history, searches were conducted using an anonymous browser setting, and new sessions were launched for each search. Given Spink and Jansen’s 2004 findings that most searchers do not look beyond the second page of results, we restricted the number of harvested results to 20 (2–3 standard Google results pages), which were then transferred as URLs to an Excel spreadsheet for later analysis. Google-generated definitions, sponsored websites, advertisements, and other suggestions (e.g., “images” and “scholarly articles”) were not included on this list. Search result analysis The first 20 results harvested for each term were rated in Excel, according to the following criteria: • purpose of the website; • relevance to AAC and Australian location;

69

JCPSLP Volume 18, Number 2 2016

www.speechpathologyaustralia.org.au