The PCA includes six small programming tasks (referred to as Challenges) spaced out over the semester. Each challenge must be completed within three days. On the first day (Wednesday, 4:00 p.m.), the students download instructions from the university’s eLearning platform. They tackle the programming task and submit their output to the CAFÉ correction program (Liénardy et al., 2020). The artifact corrects the students’ Challenges and, nearly instantaneously, quantifies their performance with a mark (from 0 to 20 points) that is explained in a message containing individual feedback (the result obtained by running their program, the expected result, and the locations of their mistakes) and feedforward (suggestions for improving their work). Based on this, the students can correct their mistakes and resubmit a new version of their code (twice at most). At the end of the Challenge (Friday, 6:00 p.m.), the latest submission determines the final mark. Each Challenge accounts for 2% of the final mark for the course.

The timeline for the PCA is provided in Figure 1. The first Challenge (called “Challenge 0” – not shown in Figure 1) helps students understand how CAFÉ works; it does not count toward the final mark. Challenges 1-5 are cumulative in terms of the course subjects covered (loops, plus Invariant, plus arrays, plus functions…). Challenges 2 to 4 build on each other to foster the internalization of the “Graphical Loop Invariant” programming methodology. The fifth Challenge is designed as an integrative task revolving around dynamic memory allocation and pointers.

The PCA also incorporates “Trump Cards,” which allow students to skip one of the Challenges. In effect, this Challenge will not count toward the student’s mark. Failure to submit an answer to a Challenge is equated with playing one’s single Trump Card.

Each individual Challenge and the progression from one to the next are moulded on “Assessment for Learning” (AfL) tenets in the field of higher education (Liénardy et al., 2021; Sambell et al., 2013): offering tasks of increasing difficulty, balancing summative and formative assessment, creating opportunities for practice and rehearsal” (Sambell et al., 2013) by allowing the students to submit up to three times and improve their output.

Following Karavirta et al.’s (2006) recommendation, CAFÉ limits the number of possible submissions to three in order to prevent “trial and error” strategies that would be contrary to the Graphic Loop Invariant approach that emphasizes reflection.

The feedback and feedforward produced by CAFÉ were carefully based on the literature promoting self-regulated learning. Among the established quality criteria for feedback (Keuning et al., 2019), CAFÉ instantiates the following procedural items: (i) individualized feedback (Brookhart, 2008), (ii) feedback focused on the task, not the learner (Narciss & Huth, 2004), and, (iii) feedback made directly available to the student to prevent them from becoming bogged down or frustrated (Knoblauch & Brannon, 1981). Contentwise, the feedback in CAFÉ can be informed by Keuning et al.’s threefold classification (2019): knowledge of mistakes (CAFÉ compiles and tests a student’s code, runs it, and warns the student if there are errors), knowledge about how to proceed (through feedforward, CAFÉ provides references to the theory course or hints about things that could improve the solution, as well as hints about improvements to the submitted Challenge), and knowledge of metacognition (CAFÉ checks that the student’s code aligns with the programming methodology introduced by our course).

Lastly, there are two pedagogical reasons for the “Trump Card” option: first, to relieve the instructors from dealing with the excuses of students who have not submitted their Challenge (Brauer, 2011), and second, to possibly increase the students’ perception of control – a factor in determining their motivation – (Viau, 2009) by allowing, on the learning path, a choice extending into the realm of self-regulated learning.

The submission platform on which CAFÉ is run allows us to gather various data: submission time, number of submissions, and the student’s mark given by CAFÉ. The various submission times for a particular Challenge and a particular student can be used to determine the amount of time between two submissions.

In our CS1 course, each student is evaluated in two ways: continuous assessment and periodic assessment. Continuous assessment refers to the PCA (the subject of this report) and five multiple choice tests (MCQs) assessing the understanding and knowledge of theoretical concepts and C programming language specificities. Periodic assessment refers to two exams: a mid-term exam in late October or early November, and the final exam in January. MCQs and the PCA each account for 10% of the final grade, while the mid-term and final exams account for 15% and 65%, respectively. As this report is focused on the PCA, we limit the performance analysis to the grades automatically computed by CAFÉ after each submission during all the PCA.

A survey was sent to students after the final exam and about 22 of them responded, i.e., half of the students who took the final exam (see Table 1). The survey was anonymous so that the students could express themselves freely.

Figure 2 shows the students’ participation in the PCA over the semester. Participation decreased during the semester in absolute numbers from 86% to 29% (see blue in Figure 2). Logically, we can see a parallel rise in absences from 16% to 55% (see black in Figure 2). As far as the “Trump Cards” are concerned, they were played quite evenly among the Challenges at around 17% (see red in Figure 2), except for Challenge 3, which shows a peak of 27%. Overall, 7% of the students did all of the Challenges. The downtrend in participation did not affect triple submissions, which stood at 50% of participants for each Challenge (see dark blue in Figure 2).

Figure 6 shows a box plot of the students’ marks obtained for their first submission in each Challenge. Considering the four quartiles of the mark distribution, the box plots reveal that first-submission results tend to decrease as the semester goes along.

Since CAFÉ allows students to resubmit enhanced solutions to a Challenge, it is worth making sure that multiple submissions do indeed enable students to improve their results. Table 2 shows that, except for Challenge 5, the majority of the students who submitted twice ended up with higher marks (See the “+” line). The same observation can be made for students who submitted three times. For Challenges 0 to 5, the total percentages of students who submitted three times and improved their grades were 79%, 50%, 82%, 58%, 82%, and 73%, respectively. Reduced grades (lines “= –” + “– =” + “– –”) were uncommon and mainly involved 10% of the students dealing with Challenge 3.

Respondents claim to benefit from the PCA, since 86% agree with the statement “Submitting 5 Challenges consisting of writing code to solve a problem was a good way to make me work regularly” (see Figure 7, Q1). Most respondents also agreed with the statement “Submitting 5 Challenges consisting of writing code to solve a problem made me feel confident about my programming skills” (see Figure 7, Q2).

When asked whether the feedback prompted them to resubmit, the response was positive: 82% (18/22) of the students stated this for three Challenges or more (see Figure 8, 1st line). Responses to other questions bolster this positive impression. When asked if the feedback and feedforward helped them to better understand the course content, 59% of the respondents reported that this was the case for three Challenges or more (see Figure 8, 2nd line). When asked if the feedback made them realize that they had a learning gap, 46% of the respondents answered that this was so for three Challenges or more (see Figure 8, 3rd line). One question gives insight into how the students used the feedback/feedforward between consecutive submissions. In fact, 59% of the respondents (13/22) reread the theory course for three Challenges or more (Fig 8, 4th line), 9% of them (2/22) retried some additional exercises for three Challenges or more (Fig 8, 5th line), 27% of them (6/22) looked for information on the course website for three Challenges or more (Fig 8, 6th line) and 18% of them (4/22) asked the teaching team questions during three Challenges or more (Fig 8, 7th line).

The students were asked why they played their Trump Card when they did. As shown in Table 3, 23% (5/22) of the respondents never played their Trump Card. A deeper analysis of the students who never played their Trump Card reveals that all of them responded to the survey. The most common reason to play one’s Trump Card (8/22, 36%) was an organizational problem (e.g., lack of time to complete the Challenge) and was quite well distributed throughout the semester (the Trump Cards played by the respondents that gave this reason are spread over all of the PCAs – see Table 3). The second most common reason was always mentioned in relation to Challenge 5: 27% of the students (6/22) were afraid that they would lower their grade if they took the Challenge or were already satisfied with their grade. They adopted a strategy that gave more weight to the score already “earned” than to the experience they could gain by completing Challenge 5. Apart from these main reasons, one student mentioned the difficulty of Challenge 4 and two of the respondents confessed to some “laziness.”

For all of the Challenges, a majority of participants actually took advantage of resubmitting to improve their performance (see Figure 2). This is exactly what the PCA was designed for, in line with the creation of “opportunities for practice and rehearsal” espoused by the AfL (Sambell et al., 2013). Similarly, participants took the opportunity to play their Trump Card regularly throughout the semester. In this respect, two very different behaviours could be distinguished: some students used it as it was intended and others simply dropped the course, in which case their first absence was labeled as a Trump Card. These latter students did not participate in subsequent Challenges. The number of drop-outs increased regularly over the semester. A steep increase can be seen for Challenge 3, which came right after the mid-term exam (see Figure 1). This suggests that the mid-term exam results made some students decide to drop the course (and perhaps change their program)[3]. By the end of the semester, more than 40% of the students seem to have dropped the course. This is confirmed by the final exam participation rate (61%; see Table 1).

The Results section presents data regarding the submission times during the course of a Challenge and the time between consecutive submissions. The following discusses the PCA parameters related to this data, i.e., the length of the Challenge and the number of submissions allowed.

The decreases observed throughout the semester (see Figure 6) could be explained by the increase in the tasks covered by the Challenges, which are cumulative: students might experience a snowball effect if they progressively accumulated learning gaps. But it should be kept in mind that Figure 6 shows students’ marks for the first submission, and CAFÉ allows several submissions. For Challenges 0 and 1, nearly 20% of students made a single submission: students getting a good grade on the first attempt do not need to resubmit. For Challenges 2 to 5, the lower results make resubmission essential in order to make progress and accumulate knowledge. This is not surprising. In fact, it is exactly why multiple submissions were allowed in the first place, followed by feedback that the students could use to progressively improve their performance.

Students who only submit twice may be satisfied with their score, which is evidenced by the high number of increases. On the other hand, students who were not satisfied could submit again, since CAFÉ allows this, establishing itself as a tool likely to have a positive impact on the goals students set for themselves and their ability to feel highly committed, a condition notably required by Tinto (1999).

The students’ perceptions tend to confirm that CAFÉ achieved one of its primary goals, i.e., to make them work on a regular basis. Students also state that they gained confidence in their programming skills (see Figure 7). This result is consistent with the purpose of “creating opportunities for practice and rehearsal” from the AfL (Sambell et al., 2013). Moreover, they acknowledged that the feedback itself encouraged them to take advantage of an additional submission (see Figure 8, first line).

As for the students’ reactions to the feedback and feedforward (see Figure 8, lines 4 to 7), it is not surprising that a majority of the respondents reread the theory course for more than three Challenges because CAFÉ directs them specifically to the course (e.g., gives the exact location of the relevant subsection). The other actions are less often explicitly suggested. Contact with the teaching team is the last resort if the student has a question about the feedback or a problem with the CAFÉ system itself. However, CAFÉ has been designed to limit that kind of interaction.

With regard to the feedback portion of the information transmitted by CAFÉ about the students’ performance (see Figure 8, lines 2 and 3), the data tends to show that it is useful for the students to know where they went wrong, if we follow the classification by Keuning et al. (Keuning et al., 2019). This result is also aligned with the AfL principle of providing students with “formal feedback to improve learning” (Sambell et al., 2013).

Introducing the Trump Card mechanism is a double-edged sword. On one hand, it was thought that the Trump Card would make students take responsibility for their learning, avoid making excuses for not submitting, and increase their perception of control over the course (and thus their engagement and motivation). Data shows that some students indeed used their Trump Card to cope with organizational issues (see Table 3). On the other hand, it allows students to develop short-term strategies to maximise their PCA grade (e.g., by avoiding a bad mark on a Challenge perceived as too difficult, see Table 3) that can lead them to avoid practising the task featured in the Challenge that they discarded using the Trump Card. This is illustrated by Challenge 5. According to the survey, at the open question “The Challenge 5 submission rate was low this year. However, it had been announced that one or more questions in the final exam would focus on the subject tackled by this Challenge. In your opinion, what is the cause of this?”, some students (4/22) “regretted using [their] Trump Card” and recognized that “taking the Challenge would have helped them for the final exam”. If we deepen this analysis by looking at the final exam results for the questions addressing the same subject as Challenge 5 (pointers and dynamic memory allocation), we observe that every student who succeeded in Challenge 5 also got these questions right. It should be noted that they first improved their score in Challenge 5 through multiple submissions. However, those who did not submit code for Challenge 5 failed at the same kind of questions on the final exam.

We still believe that the Trump Card system must be maintained. However, this means, first, that the students must be made more aware of the potential consequences of their choices (i.e., applying a poor/short-term strategy) and, second, that there must be debunking of any rumors about the difficulty of a given Challenge that would discourage students from even trying it.