Competence-based assessment and digital badging as guidance in vocational teacher education

Digital pedagogy means applying new technologies to teaching and learning in online, hybrid and face-to-face learning environments. Digital open badges, a set of micro-credentials, support equal and egalitarian competence-based assessment models. Criterion-based digital badging combined with gamification promise learning solutions that have the potential to improve learning outcomes substantially. The aim of this study is to investigate how a competence-based assessment process in an open badge management system enhances learning and guides students to improved learning outcomes. The theoretical framework is focused on concepts of gamification and instructional badging. Data were collected in 2016 from group interviews (n=6) of trained Finnish professional teachers (n=17) along with students in vocational teacher education (n=12) who earned 645 badges over one year in the Professional Development (PD) program, Learning Online. Inductive thematic analysis revealed several significant features of competence-based assessment and badge management, which reflected the students' individual experiences of the optimal form and frequency of assessments, feedback, guidance and advice. The preliminary results of this study emphasise the importance of open study groups and the option of joining and leaving the learning network freely. Shared expertise and shared learning experiences increase cohesion within freely formed study groups. The results of this study show the challenges and opportunities involved in badge management from the perspective of digital guidance and gamification, providing additional insight into the design and development of badge-driven learning in the future. This paper suggests that researchers should consider using a badge management application as an environment to guide badge-driven learning.


Introduction
Evaluation is often seen as a final (or repeating) stage of the learning process.Competence-based assessment has previously been simplistic; the evidence is evaluated to determine whether the relevant knowledge is possessed, or not (Gonczi, Hager & Athanasou, 1993).Institutioncentred assessment management platforms support formative and summative assessment, storing qualitative and quantitative data concerning students' performance (Barrett, 2004).Today, evaluation has increasingly shifted to open online environments; instead of final evaluation, competence-based assessment represents a rather complex learning process.Assessments can include a student's self-assessment, peer assessments, peer group assessments and teachers' assessments of the path towards competencies, in both face-to-face and online learning.
Teachers need skills in digital pedagogy along with discipline-specific digital competencies that enhance innovative teaching and the use of technology (European Commission, 2017).A teacher's role shifts from teaching to planning, guiding, orchestrating and supporting the learning process of students.Evaluation criteria help teachers to specify the knowledge and skills needed for specific grades (Sadler, 2005).Comprehensible criteria and standards of assessment help students to understand their existing competencies and how to deepen them.Digital open badges, as an emerging concept, refer the learner's completion of a certificate, participation in educational process or achievement of a specific competence (Abramovich, Schunn & Higashi, 2013).Digital badges (e.g.Mozilla Open Badges) allow the recognition of excellence in small fractions (Davies, Randall, & West, 2015) and motivate students to continue learning new things (Brauer, Siklander & Ruhalahti, 2017).Digital open badge management platforms, such as Open Badge Factory (OBF), provide the infrastructure required to create and issue badges.Additionally, badges may be granted based on an application and students will be assessed in relation to the badge criteria and a demonstration or evidence of the competence in question.OBF was not designed to provide a learning environment; however, the integration of badges into an active learning process allows a comprehensive system of assessment supporting learning.In the near future, digital badges may offer criterion-based learning solutions that combine different learning communities and empower alternative ways to acquire knowledge and skills (Knight & Casilli, 2012).Therefore, the aim of this study is to examine and describe how to structure competencebased assessment processes in an open badge management system to guide students to successful learning outcomes.

Theoretical Framework
This paper follows a digital open badge-driven learning process along with an implementation of competencebased assessment that the authors have experienced and observed.The theoretical framework is focused on the concepts of gamification (Deterding, 2015;2012) and instructional badging (Gamrat, Bixler & Raish, 2016;Reid, Paster & Abramovich, 2015;Ahn, Pellicone & Butler, 2014).

Gamification
Digital pedagogy combines theory with practice, and making with thinking, aiming to foster creativity, play and problem solving among learners (Spiro, 2013).The essential goal of this approach is to encourage participation, collaboration and public engagement, while increasing critical understanding of digital environments.
Gamification as a term originates from the digital media industry (Deterding, Khaled, Nacke & Dixon, 2011).The idea of gamification is to use game elements and techniques in a new context, to motivate users towards desired behaviours, and arousing enthusiasm about online learning similar to the excitement and enjoyment experienced while playing games (Deterding, 2012;2015).Reid et al. (2015) found that badges are often used to recognise learning and to motivate the learner, as a 'game-like encouragement' in non-game and educational contexts.Gamification is based on simple game design elements instead of ludic qualities -the 'gamefulness' of gameful design (Deterding, 2015).The reduced complexity of a gamified learning application retains only the simplest components of gamification, e.g.badges, levels, points, and a leaderboard (Deterding, 2012).Developing technologies promote novel possibilities, raising the question of how to combine gamification to digital badging in non-game platforms and contexts.

Instructional Badging
Instructional badges are designed to prompt a learner to demonstrate required competencies; the design of badges, and of families of connected badges, relates to the behaviours instructional designers want to reward and encourage (Reid et al., 2015;Gamrat et al., 2016).However, appropriate pedagogical models and sound instructional design are required to create quality badges.Further, the design processes should be complex and multifaceted to engage the full potential of badges that provide promising solutions in pursuit of a variety of goals.It is useful for learners to understand the constellation of instructional badges and metabadges as a personalised digital pathway of learning to structure their studies (Gamrat et al., 2016;Davies et al., 2015;Ahn et al., 2014).Clear and consistent design of meta-badges supports the visualisation of learning and summarises accomplished learner's achievements.Gamrat et al. (2016) suggest that badge designers should consider whether learners could personalise their learning pathways using badges from different badge families.The concept of a "choose-your-own-adventure online course" (McDaniel, Lindgren & Friskics, 2012) describes the scale of customisation required for such a learning process and evokes the role of badges in the connected learning ecology, "acting as a bridge between contexts, making these alternative learning channels and types of learning more viable, portable, and impactful" (Knight & Casilli, 2012).
Badge-driven learning on a customised study path consists of instructional modules, badge application process and assessment, which requires a demonstration of competence or other evidence provided by the student (Reid et al., 2015;Brauer et al., 2017).The aim of scalable badges and badge families is similar to gamified constellations, allowing students to reflect on their accomplishments and strengthen their sense of competence and progress (Deterding, 2012).Hierarchical badges provide students with progressively deeper and more complex challenges, similar to progressive obstacles in games.Gamrat, Zimmerman, Dudek and Peck ( 2014) describe a dual model, with badges and stamps equalling respectively more or less effort.Gamrat et al. (2016) call for a badge design that would offer both granularity and flexibility, to expand the evaluation of the degree of mastery or levels of credentials beyond the most basic level.
The techniques of peer review and automated response have been pursued to solve the large workload of teachers and tutors in badge evaluation (Gamrat et al., 2016).However, experienced peer-reviewers and automatic solutions are both elusive, especially in cases where the desired process for badge applications should include unique claims and evidence (Hickey, Willis & Quick, 2015).It is essential that students receive prompt and precise feedback; meanwhile, automated responses are valued differently than peer-review or professional evaluations (Gamrat et al., 2016).As an answer for rejected badge applications Gamrat et al. (2016)

Research question
The aim of this study is to investigate how to structure a competence-based assessment process in an open badge management system to guide students to enhanced learning outcomes.The research question is, how assessment management on an open badge platform supports pedagogical guidance through gamification?The context of the study is the competence-development continuum of vocational teachers, in particular the identification and recognition process of digital pedagogical competencies.

Context and participants
The context of the study is a competence-based vocational teacher education.Participants were Finnish professional teachers (n=17) and students (n=12) of vocational teacher education, both men and women.They were asked to form groups for the interviews (n=6) based on their achievements in the Learning Online PD program.The investigated Learning Online program offers in-service and pre-service ICT-training for teachers, based on national guidelines and the UNESCO ICT competence framework for teachers.Participants were known to be highly functional online, representing badge earners on every level of the requisite skill set.Learning Online badges visualise the digital pedagogical expertise achieved and help participants to plan and customise their personal development to meet the individual requirements and the needs of working life.Instructional guidance is always related to rejection of badge application to direct the guidance to those who need it the most.

Data
Data were collected in the spring of 2016.Online group interviews (n=6) with in-service teachers (n=17) and student teachers (n=12) provided interview transcripts 439 minutes or 141 pages in length.The interview groups consisted of 3-8 people.A guided group interview gave participants the opportunity to share their own thoughts and reflect on their experiences.Meanwhile, an interviewee chooses the point of view of the story itself, what and in what way he or she tells it.The role of the interviewer is to sustain the debate and encourage the story to be told by presenting additional questions.The interviewer was prepared to ask questions about criterion-and competence-based assessment, learning motivation, and digital open badge-driven learning experience.During each interview, it was verified that all these topics had been discussed in each group.The researcher did not raise questions where the group had already discussed the subject on its own initiative.

Analysis
Research was conducted via data-driven content analysis (Schreier, 2012) using NVivo 11.3.2software.The unit of analysis was a short expression of words that captured the meaning of an aspect related to learning phenomena.Hierarchically inclusive relationships were analysed in an ongoing comparison, to examine the structure and components of competence-based assessment process in an open badge management system.This visual description related to the research question, was revised in a triangulation process to increase the validity of findings.

Preliminary findings
Inductive thematic analysis revealed significant aspects of competence-based assessment and badge management, suggesting how to structure a competence-based assessment process in a digital open badge management system to guide students towards successful learning outcomes.These findings reflect students' personal understandings of the optimal form and frequency of assessment, feedback, guidance and advice.Clear badge criteria are crucial for independent self-evaluation of competencies; guiding learners in how to proceed with respect to demonstrating achieved competencies.In the PD program, Learning Online, trainers provide brief feedback to students who have succeeded in the assignment; guidance resources are allocated mostly to students who do not meet the badge criteria on the first attempt.Nevertheless, students felt that feedback about success was very important.The short personal message was significant, and reminded in-service teachers how good it feels to receive positive feedback.
How nice it is to get feedback and sense that someone really reads and looks at (one's badge application).You get so much, if there is a comment or a funny line; if there's more than just an automatic response.I do not expect written novels, it would be really horrible, really time consuming, but a precise comment, it's so nice to get it.

Pre-service teacher on skills set Novice-level I
I decided to start giving more personalised feedback for students, as I remembered how good it feels to get feedback.

In-service teacher on skills set Developer-level III
Students who failed in the assessment receive more extensive feedback, so they can learn more and further develop the evidence needed to meet the requirements defined in the badge criteria.Students were promised that they would receive the assessment no later than two weeks after their badge application.Students found a maximum of two weeks to be a reasonable time to wait for the evaluation.However, the faster the assessment is completed, the more it supports and inspires learning.
If it takes two weeks, then it's probably too long.However, the assessment is a sign that the badge has been issued and the competence has been approved.Then you can move forward, since you know that the previous ones have been approved.Like -what's next?
In-service teacher on skills set Developer-level III The feedback received inspires additional study; students intensify their studying following the waves of assessment.Badge earners appreciate this expert guidance and find it important that the evaluators are professional teacher trainers and experts on the subject instead of peers.
I do not support it (peer review) yet.Yes, I feel that the feedback from the teacher or the tutor was good.I do not rule out peer feedback as an opportunity.But how is it then?You'll have to try.I was grateful that the feedback came from the tutor.
I noticed that the auditor was really accurate that any blog did not do; that was supposed to consist of the things required.And that's good.Reliance on this system increased greatly.

Pre-service teachers on skills set Novice-level I
Badge-driven learning enhances progress on customised study paths; guidance is most needed for students who fail the task for the first time.Based on the rejected badge application and the feedback and guidance received, the student continues to learn and continues to develop evidence of their mastery.The feedback provided with the rejected badge application shows the direction of necessary studies, but students must search for the needed information themselves, either in professional development materials or within the study group on Facebook.The results of this study emphasise the importance of an open study group, with the option to join and leave the network freely.Shared expertise and shared learning experiences increase cohesion within freely formed groups of students.The study group provided students with significant new networks beyond institutional boundaries.

Discussion
The aim of this study was to investigate how to structure a competence-based assessment process in an open badge management system to guide students towards improved learning outcomes.The main result is that structured competence-based assessment and badgedriven learning seem to support student guidance and gamification.On the basis of these preliminary findings we now raise the following issues to discuss further.
First, the authors conclude the competence-based assessment and digital badging in an open badge management system as a multifaceted process consisting of the badge-criteria, the badge application and pedagogical guidance.The badge application and assessment process require a demonstration of competence or other evidence provided by the student.The badge-criterion is aimed for the student to provide required information to identify competencies, to self-evaluate the mastery and to support procedures of badge application.
Second, the results of this study have identified that the best opportunity to give appropriate feedback relating to badge rejection, confirming the suggestion of Gamrat et al. (2016) to provide feedback or remediation as guidance for a second submission.This feedback encourages waves of enthusiasm towards learning.It is necessary to look further, to investigate how new, affordable solutions for individual, professional assessment will work.Automated answers are adequate for situations where the student has been successful, but do not provide enough for those needing to resubmit after a rejected application.In light of these preliminary results, peer review seems not to be an option, because students desire experienced professional reviewers.
According to Gamrat et al. (2016), recommended learning pathways "require collaboration between various badge stakeholders".Self-education and learning by doing should be considered the predominant ways to acquire expertise in the digital age; however, students also appreciate the option of collaborating with their peers in problem solving and learning in general (Lewis, Spiro, Wang & Cawthorne, 2015).We will continue to complement these preliminary results, deepening the theoretical framework of inspiring gamification, because badges seem to work better when the learning is social and networked (Hickey et al., 2015).This also calls for an in-depth review of such concepts as co-regulation, selfregulation and socially shared regulation of learning (Järvelä, Kirschner & Hadwin, 2016); and the game models including achievement goals intended to encourage collaborative rather than individual work (Deterding, 2012).
This paper suggests that future researchers should consider a badge management platform as a guidance environment of badge-driven learning.However, additional research is needed to optimise the assessment process on the badge management platform for student guidance and improvement of learning outcomes.
the data within the coding process reveals what students consider important in the badgedriven learning process from the point of view of guidance.Students' experiences describe how guidance and assessment are related to badge-driven learning, and what kind of online solutions may enhance learning, in addition to guidance of the studies.As the final outcome the results allow to draft the guidance process related to competence-based assessment within digital open badge-driven learning.

Figure 1 :
Figure 1: Structure and components of a digital open badgedriven learning process: competence-based assessment and badge management related to guidance suggest providing feedback or remediation to guide learners towards a second submission.The guidance process in relation to digital open badge-driven learning is a new interest for practitioners and researchers.