How to Evaluate Engineers on AI-Assisted Coding (Not Just the Code)

Most technical interview loops were built for a world where a candidate sits alone with a blank editor and no help. That world doesn't exist anymore. Engineers use AI assistants constantly — to scaffold functions, debug errors, explore APIs, and sanity-check edge cases. Banning AI in an interview doesn't test a more 'pure' version of the candidate's skill; it tests a skill they'll rarely use on the job.

The harder problem is that once you allow AI, the thing you actually want to measure changes. It's no longer just 'can this person write correct code.' It's 'can this person direct a powerful tool toward a correct, well-reasoned solution, and know when to push back on it.' That's a different skill, and most hiring processes have no way to see it.

What good AI collaboration actually looks like

A handful of behaviors separate engineers who use AI well from engineers who just accept whatever it outputs:

• Prompt specificity — providing real context (error messages, constraints, prior attempts) instead of vague one-line requests.
• Decomposition before delegation — breaking a problem into pieces and forming a hypothesis before asking the AI to solve it, rather than pasting the whole problem in and hoping.
• Critical verification — testing AI-suggested code against edge cases instead of accepting it because it compiles.
• Iteration quality — adjusting the approach quickly when a suggestion is wrong, rather than repeating the same prompt or giving up.
• Judgment about when not to use it — recognizing the handful of moments where reasoning it out themselves is faster or safer than a round trip to the assistant.

The red flags are just as telling

The inverse pattern shows up just as clearly, and it's arguably more useful signal: a candidate who pastes the task description into a chat window, copies the output verbatim, and submits without running it. Or one who can't explain why a suggested fix works when asked. Or who never disagrees with the AI, even when it's clearly wrong. None of this is visible in a final code submission — it only shows up in the process.

Structuring a task that surfaces this

The task itself doesn't need to change much from a normal take-home or live coding exercise. What changes is what you capture and what you allow:

• Tell candidates explicitly that AI is allowed and expected — the goal is to remove the ambiguity that makes people hide their AI usage.
• Give them a real editor with an assistant built in, rather than trusting them to screen-record their own external tool usage.
• Capture the workflow, not just the outcome: prompts sent, edits made, code run, and how those relate to each other over time.
• Ask for a short written reflection at the end — it surfaces reasoning that raw telemetry alone can miss.

The final diff tells you almost nothing about how someone actually works. The 45 minutes before it does.

This is the gap DevTrace is built to close: a real coding or written task, a built-in AI assistant, and a captured event stream (edits, prompts, runs, pastes, focus shifts) that turns into a short prose summary of how the candidate actually thinks and works — not a score, a narrative your team can read in two minutes before a follow-up interview.