Peer Evaluation

The peer evaluation is a synchronous, oral assessment of your Gazprea compiler. Your team demonstrates and defends your compiler to another team of students, who assess each of you individually and your team as a whole against the rubric on this page.

The evaluation assesses understanding; the number of tests you pass is graded separately. What is graded here is whether you can navigate your own code, explain why it is built the way it is, and reason about a compiler as a whole system.

Most of your result is your own: three quarters of it comes from how you personally answered, and the remaining quarter from how your team handled the questions put to it collectively.

Schedule

Three evaluations are held: one for Part 1 and two for Part 2. All three follow the format described below.

In each evaluation, your team plays two roles:

You are evaluated by one team.
You evaluate a different team.

No team evaluates the team that evaluates them. The pairings are redrawn for every evaluation, so the team you are evaluated by and the team you evaluate are both different each time.

Evaluations run in person. Several rooms in the same building are booked for each session and teams rotate between them. The schedule for each evaluation is posted on Canvas ahead of the session: it names the team you are evaluated by, the team you evaluate, and the room for each of those two roles. Check it before the session starts.

Format

Each evaluation is allotted 80 minutes. About 10 minutes of that is buffer for changing rooms and setting up, leaving roughly 70 minutes across two phases.

Presentation (5-10 minutes)

Your team gives a brief account of the compiler: the top-level architecture and who implemented what. Each member individually names the parts of the compiler they worked on.

This phase is not graded. It is how the evaluators learn who to direct which questions to, and a vague account of who did what leads to questions that do not match your work.

Q&A (~60 minutes)

The evaluators ask questions guided by the rubric and take notes as they go.

Evaluators do not see your source code before the session. You are expected to navigate your codebase live on your own machine, pointing at specific code to support your answers. Have your development environment open and ready — a build of the compiler, your tests, and an editor you can search in quickly. Evaluators may ask follow-up questions based on what you show them.

Four evaluators reading code over your shoulder can be cramped. Consider joining a muted Discord (or similar) call for the session instead, with every member sharing their screen for its whole duration and one member’s laptop connected to the room’s TV. That member switches the TV to whichever share belongs to whoever is answering. Any arrangement that lets the evaluators see the code you are pointing at will do.

Evaluators are instructed to follow up with clarifying questions when a first explanation comes out hesitant or garbled, so a rough first answer by itself does not cost you marks.

Multiple members may contribute to a single answer. Questions put to a specific member still count towards that member’s own mark, though, and that mark suffers if you consistently need a teammate to answer for you.

What you will be asked

Three kinds of questions appear in the Q&A.

1. Questions to a specific member. Chosen or improvised by the evaluators, aimed at the parts of the compiler you implemented. Examples of the sort of thing to expect:

How does your compiler deal with type aliases that give a type the same name as a variable?
How does your compiler distinguish l-values from r-values at each stage?
Open the code that creates the basic blocks for control flow (if, loop, and so on), and walk through what it emits.
How are types represented in the MLIR backend?
Do you have separate AST nodes for a global versus a local variable declaration?
How do you handle implicit type promotion?
How are array types handled? What type does an empty array have?

2. Questions every member must answer. The evaluators do not choose these; the same three are asked at every session, so you can prepare them in advance:

Give an example of a test you wrote, and the part of the compiler it was intended to test.
Showcase what you are most proud of in your work.
What did you struggle most to implement, why, and how did you solve it?

Each member confirms which of the three they were asked, on the form described under Rating your evaluators.

3. Questions to the group. Answered collaboratively by whichever members hold the relevant knowledge. These cover cross-cutting design. The evaluators choose their own; these are examples of the sort of thing to expect:

How was Part 1 designed to accommodate Part 2?
How does the AST design support the language’s features?
How is the distinction between functions and procedures enforced end-to-end?
Take an error your compiler can report, and show us every place it passes through, from where it is detected to what the user sees.
Where does the type system meet the AST representation, and what does each one assume about the other?

Coverage

Before the Q&A ends, the evaluators must have asked at least one question touching each of:

grammar and parse tree
AST design and node structure
symbol tables and scoping
type system: checking, inference, and promotion
functions versus procedures — the semantic difference and how it is enforced
MLIR code generation
error detection and reporting

The team you evaluate is asked which of these its session reached, on the form described under Rating your evaluators.

Ownership expectations

Each member must be able to speak in depth about a real share of the compiler.

Gazprea builds on the VCalc pipeline, so the work divides most naturally by language feature: one member takes arrays through the grammar, the type checker and code generation; another takes arithmetic on reals through the backend. Split the work this way. Splitting by compiler stage instead, with one member on the type checker and another on code generation, makes every feature wait on three or four people finishing in the right order, and teams that try it tend to stall.

When an evaluator asks about something you implemented, you can explain it at every stage it touches and point to the code that does it.

Your responsibilities as an evaluator

Evaluating is part of the exercise, and doing it badly denies the other team the chance to show what they know.

Submit a question list to the instructor before the lab. Submission is required. The list is not graded on coverage; it exists so that you arrive prepared and so there is a record of it.
You are not bound to your list. Ask what the session calls for, and improvise follow-ups based on what the team shows you.
Spread the questions. If one person has fielded several answers already, move to a question aimed at a member who has not been tested yet. By the end, every evaluated member should have answered enough for their demonstrated understanding to be clear.
Draw the knowledge out. Follow up on a weak or garbled first answer instead of recording it as a failure. A student may explain something poorly and still understand it well; your job is to find out which.
Track coverage as you go. You are responsible for the coverage list above being satisfied before time runs out.

After the evaluation, each evaluator individually fills out a rubric, assigns a mark and writes a justification for each of the four evaluated students and for the team as a whole, and distributes ten contribution points across the evaluated team. All of this is described under the grading matrix.

Submit it to receive any peer evaluation marks of your own. A member who does not submit their assessment of the team they evaluated scores zero for that evaluation.

Rating your evaluators

Every member of the evaluated team individually fills out a short form on Canvas about how the session was run, due shortly after the session ends. It asks for one mark out of 100 for the evaluating team, with a written justification.

Submit it to see your own results. Your marks for an evaluation are released once you have submitted the form for it.

Anything serious — evaluators who did not arrive, hostility, or being stopped from showing your own code — goes in the second question on the form, which reaches the instructor directly.

Grading Matrix

Each evaluator produces five assessments per session: one for each of the four evaluated students, and one for the team as a whole. Each assessment has three parts.

A filled-out rubric. For a student, place them at one of the four levels on each of the four individual objectives. For the team, place the team at one of the four levels on each of the two group objectives.

A mark out of 100. For a student, this reflects their four individual placements; for the team, its two group placements. Assign it by judgement, keeping it near the anchors below.

A written justification. This covers both the placements and the mark, and says what they rest on — which answers, which code, which moment in the session. If the mark sits away from where the placements alone would put it, the justification is where that gap is explained.

Each evaluator also distributes ten contribution points across the evaluated team, individually, alongside their five assessments. The ten whole points are split among the four members according to how their contributions compared to one another, based on what the session showed. Points cannot be split in half, so ten points across four members can never come out even; every evaluator is required to rank the team. This is a relative signal only, and is separate from the marks out of 100.

Objectives and weights

Four objectives are assessed for every student individually and make up the individual mark, a quarter each. Two are assessed once per group and make up the group mark, half each. The weight column below is each objective’s resulting share of a student’s peer result.

Objective	Scope	Weight
Navigate and explain their work in the codebase	Individual	18.75%
Criticise and defend the design of their work	Individual	18.75%
Diagnose failures and validate features through the use of tests	Individual	18.75%
Outline information flow across compiler passes and identify where language features are handled	Individual	18.75%
Assess the implementation complexity of an unfamiliar language feature and argue a position on its design	Group	12.5%
Evaluate and criticise whole-compiler design decisions	Group	12.5%

Each objective is described at four levels of performance. An evaluator places you at one of the four on each individual objective, and your team at one of the four on each group objective; the descriptors say what a level looks like.

Level	Meaning
Excellent	Understanding is demonstrated fluently and extends beyond what was directly asked.
Good	Understanding is solid within familiar territory, with gaps at the edges.
Satisfactory	Surface-level understanding; the what is present but not the why.
Needs improvement	The objective is not demonstrated.

The whole rubric is also laid out as a single chart, sized for printing and for use during a session: see Peer Evaluation Rubric.

How the marks combine

A student’s peer result is 75% their own individual mark and 25% their team’s group mark. Every member therefore carries how the team performed on the cross-cutting questions, whoever answered them.

Anchors

The mark and the rubric placements must be consistent with each other. These are the reference points:

Rubric placement	Mark
Every objective at Excellent	95
Consistently Good	80
Consistently Satisfactory	65
Consistently Needs improvement	35

Mixed placements land between the anchors. Excellent on two objectives and Good on the other two sits in the mid to high eighties. Needs improvement on one objective and Good on the rest sits near 70, and the justification should say which objective pulled the mark down.

The weight of the peer evaluation within the overall Gazprea grade is announced separately; see the course grading matrix.

Individual objectives

Navigate and explain their work in the codebase. You can navigate the compiler live, point to code you worked on, and explain what it does without relying on teammates.

Level	Descriptor
Excellent	Navigates confidently and explains clearly what the code does and how it fits into the surrounding pipeline.
Good	Navigates their area well and can explain what the code does, with minor hesitation or gaps.
Satisfactory	Locates relevant code with some hesitation but gives surface-level explanations, or relies on teammates for context on parts of their own area.
Needs improvement	Cannot navigate to or explain their contributions without significant help from teammates.

Criticise and defend the design of their work. You can articulate why your section is designed the way it is, identify tradeoffs or limitations in your choices, and engage with alternatives or critiques.

Level	Descriptor
Excellent	Gives specific, reasoned justifications for their design decisions, acknowledges tradeoffs or things they would do differently, and engages with a hypothetical alternative or critique without becoming defensive or dismissive.
Good	Explains their design choices and identifies at least one tradeoff or limitation, but struggles to engage meaningfully with alternatives or critiques beyond restating what they did.
Satisfactory	Describes their design at a surface level but cannot explain why choices were made, or deflects critique without engaging with it.
Needs improvement	Cannot articulate design choices or engage with any critique of their work.

Diagnose failures and validate features through the use of tests. You can write a test targeting a specific compiler behaviour, explain what it is designed to catch, and use test output to reason about whether the compiler is behaving correctly.

Level	Descriptor
Excellent	Gives a concrete example of a test they wrote, explains what compiler behaviour it targets and why, and can reason about what a different failure output would imply about correct or incorrect compiler behaviour.
Good	Explains a test they wrote and interprets test output, but struggles to reason about edge cases or unfamiliar failure modes.
Satisfactory	Describes a test at a surface level but cannot reason about what a different failure output would imply, or needs significant prompting to connect test output to compiler behaviour.
Needs improvement	Cannot explain what compiler behaviour a test is designed to verify, or cannot interpret test output to draw any conclusion.

Outline information flow across compiler passes and identify where language features are handled. You can describe the role of each pass in the pipeline, explain what information is available or produced at each stage, and locate where a specific language feature is represented, checked, or emitted.

Level	Descriptor
Excellent	Gives a clear, accurate account of the pipeline as a whole and can trace an unfamiliar feature through the relevant passes without prompting.
Good	Outlines the pipeline and traces familiar features accurately, but struggles with unfamiliar features or passes they did not write.
Satisfactory	Describes individual passes but cannot connect them into a coherent account of information flow, or can only trace features they personally implemented.
Needs improvement	Cannot describe the pipeline structure or locate where a feature would be handled.

Group objectives

Assess the implementation complexity of an unfamiliar language feature and argue a position on its design. Given a feature that is not in the compiler, your team can reason about where it would live, what it would interact with, and what its implementation would cost — and commit to a defensible view on whether and how it should be designed.

Level	Descriptor
Excellent	Gives a specific, reasoned account of where a hypothetical feature would live and what it would cost, and argues a defensible position on its design — including tradeoffs or ways they would do it differently.
Good	Reasons about implementation complexity at a reasonable level and offers a position, but the argument is underdeveloped or not well grounded in their implementation experience.
Satisfactory	Identifies roughly where a feature would be handled but cannot reason about interactions or costs, or offers a position without any supporting argument.
Needs improvement	Cannot reason about where a new feature would be handled, or offers no position on its design.

Evaluate and criticise whole-compiler design decisions. Example topics: how Part 1 was designed to accommodate Part 2; how the AST design reflects and supports the language’s features; how the compiler enforces the distinction between functions and procedures end-to-end; how the type system interacts with the AST representation; how error reporting is threaded through multiple passes; how scoping and the symbol table interact across nested constructs; how type inference and promotion are handled consistently across contexts.

Level	Descriptor
Excellent	Gives clear, reasoned answers and evaluates specific choices — including acknowledging tradeoffs or things they would do differently.
Good	Addresses questions well but struggles to justify choices beyond “it worked.”
Satisfactory	Addresses questions at a surface level but cannot connect design decisions across passes, or answers are inconsistent across members.
Needs improvement	Cannot engage with cross-cutting design questions, or answers are contradictory across members.

Preparing

The evaluation rewards work done throughout the project. In practice:

Take features end to end. A feature you carried through the grammar, the type checker, and code generation gives you something to say at every stage of the pipeline. This is what the ownership expectations above ask of you, and it is the single largest thing you can do to prepare.
Work outside what you built. The individual objectives ask you to trace features through code you did not write. Fixing a bug in a teammate’s feature is the cheapest way to get there.
Know the reasoning behind the design. Every objective above distinguishes describing the design from justifying it. Keep track of the decisions your team made and the alternatives you rejected.
Write tests you can talk about. One objective is entirely about your tests. Be able to name a test, say what behaviour it targets, and reason about what a different failure would have told you.
Be able to find your code. Live navigation is graded. Know your way around the repository without searching blindly.

Note