The performance harness is becoming an observability tool

The first version of the PeakTrace performance harness was mostly about getting repeatable numbers for DSP and UI scenarios. That was already useful, but recent work changed its role.

Now it is less like a benchmark button and more like an observability workflow. The real questions are: what did we test, under which conditions, where did the time go, was the data actually fresh, and can another agent make the same comparison later?

For an audio analysis application, “performance” is not one number. A visual can feel late because of audio delivery, analysis scheduling, a worker hand-off, UI cadence, geometry submission, the compositor, or the GPU. Optimising the wrong layer is easy if all we have is an overall FPS number.

From a question to a stability decision

flowchart TD
    H[Question and hypothesis] --> S[Choose the smallest valid scope]
    S --> I[Fixed workload: fixture or replayed session]
    I --> R[Measured run under matched conditions]

    R --> C[Code and DSP timing]
    R --> D[Runtime and data-flow health]
    R --> V[Render submission data]
    R --> O[OS, compositor and GPU evidence when needed]

    C --> B[Verified evidence bundle]
    D --> B
    V --> B
    O --> B

    B --> M[Compare with a matching baseline]
    M --> A[Stability audit: keep, investigate or change]

Not every run collects every branch in this diagram. The point is to ask for the smallest evidence set that can answer the question, and then keep it in a form that makes later comparison possible.

A smaller question gets a better measurement

One important improvement was separating the kinds of performance questions.

For normal health checks, agents use a light, component-scoped capture. For a real GPU, power or compositor question, they can deliberately choose a heavier diagnostic run. Those two modes are not mixed casually, because the act of measuring can affect cadence itself.

The performance skill encodes that choice, so an agent starts by classifying the question instead of launching the most invasive capture by default. There is a whole-app profile when interactions matter, and narrower profiles when the question is spectrum, spectrogram, vectorscope, or the pitch/oscilloscope path.

Every meaningful comparison also keeps its conditions aligned: machine and OS, build and feature set, provider, fixture or replay, warm-up, display state, profile and diagnostic intensity. Without that, a neat chart can still be a false comparison.

What the harness can see

At the code level, the harness records both end-to-end scenario time and the time spent in named stages. That means we can see throughput, totals and percentile behaviour, then distinguish a local slow stage from a problem that actually changed the whole scenario.

At the application level, it records the health of the runtime: frame cadence and hitches, callback timing and delivery, buffer pressure and loss signals. Render evidence adds the amount of work the UI submits, such as shapes, geometry and texture activity.

For a system-level question, optional collectors add process and compositor CPU, window/surface information, GPU power and frequency, and deeper Core Animation or Metal traces. These are deliberately separate from code-stage timing. A collector can tell us what the operating system observed during the run; it cannot truthfully attribute that time to one source line in our application.

The newest layer connects this evidence to the data path itself. We can distinguish module compute time from source age, display delay, fresh results and deliberate held or dropped data. So when a visual updates late, the question is no longer only “was the GPU slow?” It can be followed through the provider, analysis, data hand-off and presentation cadence.

This is important because not every repeated frame is a failure. Sometimes there is genuinely no new audio; sometimes an analyzer is intentionally gated; sometimes a callback arrived late; sometimes the UI exceeded its budget. These are different causes and should not lead to the same fix.

Evidence first, raw detail only when needed

Each completed run produces a compact front door: a summary, a readable report, exact comparison metrics and run metadata. Detailed traces and raw system-collector output remain available for a specific unresolved question, rather than becoming the default wall of logs an agent has to digest.

The run is verified before it can influence rolling state. An interrupted or incomplete capture is preserved for forensic work, but it cannot silently become a baseline or a best result. This matters more than it sounds: a performance history is only useful if its records are comparable and trustworthy.

Baselines are for stability audits, not victory laps

A baseline is not a trophy number that every future run has to beat. It is a named, comparable record of expected behaviour under a defined workload. The harness keeps the run identity with its metrics, so it can reject comparisons that differ in the conditions which matter.

That gives agents a useful stability-audit loop. A change can be checked against the relevant baseline for end-to-end cost, local percentile regressions, runtime health, data freshness and any required diagnostic evidence. If the evidence is incomplete or the identities do not match, the correct outcome is uncertainty, not a claim of success.

Over time this gives us both a current best reference and a recent history. It makes regressions easier to spot, but also protects us from declaring an optimisation when it only moved cost, changed the workload, or produced a nicer-looking but incomparable run.

How it fits the agentic flow

The practical change is in the discipline. Before running anything, an agent states a hypothesis and chooses the smallest suitable profile. It fixes a replayed session when comparing a known audio problem, reads the result in a defined order, and reports remaining uncertainty instead of turning partial telemetry into a confident claim.

The harness cannot replace deterministic tests, code review or a real visual check. It makes those activities better connected: a replay can reproduce the input, the performance run can quantify the cost, a native check can show the visible result, and the stored baseline can tell us whether stability held.

That is the direction I want: agents that measure deliberately, preserve the result, and learn how the code behaves instead of producing more ungrounded optimisation ideas.