The criteria used by key decision makers in Australia to judge the academic quality of NTROs

Thirty experts in the assessment of the quality of Non-Traditional Research Outputs (NTROs) as academic research outputs were asked to rate the importance of 19 criteria that might be used in making these judgements. Analysis of responses identified four criteria where there is substantial agreement among the community of experts: (a) demonstrated familiarity in the research statement with the current state of knowledge in the relevant academic disciplines (very important); (b) demonstrated familiarity in the research statement with the current state of knowledge in the relevant industry (important); (c) evidence that the work has been engaged with by other academic researchers (relevant); (d) whether the NTRO creator is a substantive university staff member or an adjunct/honorary (unimportant). Fifteen other criteria either reached a less than ‘fair’ level of agreement, or larger numbers of respondents nominated ‘It depends’. Qualitative analysis of comments also revealed noteworthy disagreements in the expert community about how the criteria should be applied.