Multi-method longitudinal assessment of transferrable intellectual learning outcomes

Multi-method longitudinal assessment of transferrable intellectual learning outcomes. Queen's University is part of a six-­‐institution consortium in Canada committed to developing assessment techniques for generic learning outcomes and cognitive skills. These skills, including critical thinking, problem solving, communication and lifelong learning, are the subject of discussion in higher education generally (e.g. the American Associate of Colleges and Universities (AAC&U) Essential Learning Outcomes) and engineering in the form of accreditation requirements. The three-­‐year project at [University X, (blinded)] is using multiple methods to assess the longitudinal development of these skills in engineering, humanities, physical science, and social science sectors. Methods are aimed at sustainable assessment achieved within standard course contexts and developing internal processes for the implementation, management, and assessment of university-­‐wide learning outcomes that recognize and enhance disciplinary expectations. We have taken a three-­‐pronged approach to the longitudinal assessment of general learning outcomes across targeted programs, including engineering: A. Assessment using standardized quantitative instruments and qualitative processes B. Working with course instructors to align teaching, learning and assessment of complex cognitive skills, embedding course-­‐based “authentic problem tasks” for course grading and assessment C. Assessment of course artefacts using meta-­‐rubrics scored independently of course grading The project is using multiple instruments to measure identified learning outcomes, including two standardized instruments: Collegiate Learning Assessment Plus (CLA+) Test, Critical Thinking Assessment (CAT) Test, think-­‐aloud sessions, AAC&U VALUE rubrics, and a new triangulated qualitative/quantitative measure of Transferable Learning Orientations (TLO) based on the VALUE rubric for lifelong learning and the Motivated Strategies for Learning Questionnaire (MSLQ). In the first year of the project a double cross-­‐sectional study of first and fourth year students was conducted to pilot the instruments and identify key themes. Over 2000 first and fourth year students from the Faculty of Arts and Science (Psychology, Drama and Physics), and from the Faculty of Engineering and Applied Science (Chemical Engineering, Civil Engineering, Geological Engineering, and Mechanical Engineering) have consented to participate in the project. Preliminary analysis indicates significant correlations in student scores on specific CLA+ and VALUE rubrics dimensions, and significant changes in learning outcomes from first year to fourth year in engineering. The TLO was piloted among approximately 1000 students in Engineering and Applied Science over two years. The instrument was refined through quantitative analysis and focus groups, and found significant improvement between first to fourth year learning orientations. This paper will present results from the first year of the study including descriptive statistics and factor analysis for the first year and fourth year student samples, correlations between the CLA+ and assessment course artefacts using VALUE rubrics, approach to calibration and scoring, and demographic analysis. It will discuss observations about instructor buy-­‐in and feedback, and broader impact. It will also present the development and dimensions of the TLO, alignment with the VALUE rubrics, and results of the pilot study.