How to evaluate a university guidance tool: a counselor's pilot checklist

A guidance-tool trial can leave you with plenty of activity and still no clear answer to “Was it worth the time?” To evaluate a university guidance tool, compare what it adds to your school's existing process: the preparation students bring, how counselors use it, and the work required around it. A small, focused pilot can help you decide whether to continue, adapt or stop.

Agree what the pilot needs to help you decide

Imagine reaching the end of a trial with a tidy usage chart and no clear answer to leadership's question: “Should we keep using this?” The missing piece may be an agreed definition of useful preparation.

Before the trial, complete this sentence with your team:

“We want to find out whether this helps [these students] do [this task], so that [this part of guidance] becomes more useful.”

For example: “We want to find out whether students can explain why two course directions interest them, so that our meetings can explore the reasons behind their choices.” This gives the review something observable to look for.

Write down what happens now. Perhaps students complete a short form before a meeting, bring course research, or arrive without preparation. Note what already works, what you still have to uncover in conversation, and the staff effort involved. Those observations give the pilot a baseline.

Keep the task and meeting conditions reasonably comparable. If you also introduce a clearer prompt, extra tutor time or provider-led support, record it. Otherwise, you may credit the tool for a change in how the activity was introduced.

Keep the question close to the workflow being trialed. If the purpose is meeting preparation, admission offers are too distant to judge it. If the purpose is course research, a student reflection report alone will not answer whether course information is accurate.

The Education Endowment Foundation's implementation guidance emphasizes how an approach works in everyday school practice, including staff behavior and context. The checklist below is our suggested way to structure a guidance-pilot discussion, rather than an EEF evaluation instrument.

Use a checklist that separates participation from usefulness

Agree who will gather each piece of evidence and when you will review it. Use the records your school already keeps where possible.

Keep completion counts alongside this checklist. They can show where participation drops off, but they cannot tell you what a completed response contributed to a meeting.

Choose a sample that includes more than the provider's strongest examples. Include different experiences of the task, and ask why some students never began or did not finish. Report whose experience the review covers; do not present a convenient sample as representative of the whole school.

Work through one example before judging the whole trial

Consider this hypothetical pilot: students complete a reflection task before a course-choice meeting. Staff are pleased with participation, but the responses repeatedly say things like “I want a good career” or “I enjoy science.”

Can students explain what they mean, and did the task help them get there?

Ask one student to talk through their response. Perhaps they misunderstood the prompt and thought they were supposed to sound certain. Perhaps they gave a concrete example that the summary omitted. Or perhaps the activity never asked them to connect their interest with an experience.

Each explanation suggests a different change:

This is where a pilot earns its value. It helps the school distinguish an instruction problem, a product problem and a poor fit for the intended use.

Ask students and counselors about specific moments

With students, try: “Was there anything in this activity you wanted to bring into our conversation?” Follow their answer before offering examples. Ask what was confusing or uncomfortable, and let “nothing” be a legitimate response.

With counselors, ask: “What did this help you understand before the meeting, and what did you still need to ask?” Invite an example of material they corrected or disagreed with. When AI-generated interpretations are involved, check whether the student recognizes the interpretation and can explain or challenge it.

Keep the student's account and the counselor's interpretation distinguishable in your notes. If someone calls the tool useful, ask which moment made it useful.

Use your school's normal permissions and records for reviewing student work. A leadership brief usually needs an anonymized example or synthesis, rather than access to private student reflections.

Count the work that sits around the tool

Include the time spent explaining the activity, fixing access, prompting students, reading preparation and correcting misunderstandings. Record estimates as estimates, with their basis. Ask who would do that work during an ordinary week.

This matters especially when guidance sits alongside teaching or DP coordination. A trial may feel manageable because a colleague temporarily helped with reminders or the provider led every session. Write that support into the conclusion so the proposed next step reflects the conditions the school can actually maintain.

Before claiming time savings, compare similar tasks and include the surrounding work. A short informal trial may show a promising change without establishing a reliable percentage saving. The Counselor's Desk piece on thinking time offers a useful perspective on the work around student meetings.

Bring leadership a decision they can examine

Use this short brief to keep the recommendation connected to the evidence. It can sit in your existing meeting notes.

Decision we are asking the school to make:
Students, task and trial period covered:
Existing process and what changed during the pilot:
What useful preparation was meant to look like:
What we observed, with an example:
What did not work, and whose experience is missing:
Staff effort and support needed:
What the trial cannot yet tell us:
Recommendation: continue / adapt / stop / investigate a named question
Condition to meet before expanding:
Owner and next review point:

Hypothetical completed pilot: A school trials a reflection tool with one tutor group before course-choice meetings, replacing its short preparation form. Counselors find more specific examples in some students' preparation, and those students can explain their interests more clearly. Other responses remain broad; several students need help starting, and report review adds work. An extra tutor session also supported the trial, so the school cannot attribute the improvement to the tool alone.

Decision: continue with a limited second round, simplify the introduction and repeat the review within normal tutor provision. Expand only if useful preparation persists and staff can accommodate the work; otherwise adapt or stop. The first round identifies a promising use and the conditions still to resolve.

Apply the same questions to Lyra

Lyra uses structured, conversational reflection to organize student interests and concerns into a report for counselor discussion. In a Lyra pilot, examine how that reflection and report contribute to the same kind of meeting you already hold:

• Recognizable detail: can the student connect an interest or concern in the report to something they have experienced, and explain or correct the interpretation in your conversation?
• A better starting question: does the report help you ask something specific, such as “What did you enjoy about comparing those options?”, rather than reconstructing all the background?
• A useful next action: can you and the student turn one point from the discussion into a course investigation, a question for a teacher or another appropriate step?
• Workable preparation: does the value of that starting material justify the time spent introducing the reflection, helping students and reading their reports, compared with your current process?

Record one example where the report helped and one where it needed clarification or added little. Ask the student what the reflection itself contributed. This lets you judge the value of both the student experience and the counselor preparation without assuming that a well-written report necessarily improved the meeting.

For the student preparation behind that process, see how self-knowledge supports choices and starting with student clarity. To discuss a pilot around your school's question, talk to the Lyra team.

Bring the review back to your school's original question: what did this add, for whom, and under what conditions would it be worth continuing?