ONE Voice Lab evaluation methodology

1. 1. One fixture is bounded evidence

   One fixture cannot prove universal superiority, and no single result should be generalized beyond its stated input and environment.

2. 2. Align what can be compared

   Provider comparisons require the same input and disclosed acceptance criteria, while exact model, voice, transport, codec, region, and provider-native settings stay visible because they are not necessarily equivalent.

3. 3. Results are point-in-time observations

   Models, APIs, routing, infrastructure, and configuration change; results need dates and provenance.

4. 4. Latency depends on the system

   Network, region, client, buffering, audio, orchestration, and provider behavior can all change observed latency.

5. 5. Keep measurement points separate

   Provider-reported timing, ONE server-observed timing, browser playback readiness, derived metrics, and human ratings use different provenance and must not be merged into one latency or quality claim.

6. 6. No universal winner score

   Measured evidence and human-rated dimensions remain separate. A result can support a use-case decision without claiming that one voice or provider is universally best.

7. 7. Transcripts are not business outcomes

   Transcript quality and task or business outcome quality are related but not equivalent.

8. 8. Synthetic is not production

   Synthetic results test deterministic logic; they do not prove production provider behavior.

9. 9. Production needs representative testing

   Production decisions require representative customer audio, conditions, policies, consent, and human review.

Repository verified

Implementation evidence exists in this repository; this alone does not prove live provider behavior or production readiness.

repository_verified

Provider documentation verified

The claim is grounded in a dated official provider documentation reference, not a live lab measurement.

provider_documentation_verified

Manual verification required

A human must validate the current provider, account, environment, or subjective behavior.

manual_verification_required

No implementation evidence

The lab does not currently contain evidence that the capability is implemented.

no_implementation_evidence

Simulated

The result comes from a deterministic local fixture and makes no live-provider claim.

simulated

Measured

A dated result was measured under a disclosed environment and configuration.

measured

Assumption

A proposition to validate, not an established fact.

assumption

Experimental

A bounded exploration that is not a production-readiness claim.

experimental

• • Evidence schema one-voice-evidence/1.0.0; TTS method one-tts-compare/1.0.0; metric method one-tts-metrics/1.0.0.
• • Server first-audio and completion intervals use a monotonic clock; browser time-to-playable is recorded separately.
• • Region provenance is explicit: ONE's server region is not presented as a provider processing region.
• • Audio duration and real-time factor are derived only from validated normalized audio.
• • Human ratings remain separate from measured evidence; Phase 1 does not display model-judged evidence or a composite winner score.
• • Cost stays unavailable unless an exact, dated, versioned pricing formula supports it.
• • Phase 1 protected and local live comparisons are Standardized-only until native formats have a validated portable playback and provenance boundary.
• • One scenario is evidence for that scenario, configuration, environment, and date—not a universal provider conclusion.

• • Benchmark result schema one-benchmark/1.0.0; methodology schema one-benchmark-methodology/1.0.0; leaderboard schema one-benchmark-leaderboard/1.0.0.
• • Objective measurements, human judgments, automated judgments, and provider-documented claims remain separate evidence classes.
• • A repeated-run series must match its suite, case/input, methodology, exact provider/model/configuration, deployment, region, transport, and media boundary.
• • A scoring profile fixes the measurement source, point, clock, observation class, method, and source-schema version; provider-reported and ONE-observed latency are never pooled.
• • Partial runs retain exact provider-lane source identity, so one successful lane can remain inspectable while a failed lane receives its own exclusion.
• • Median is withheld below three samples; p95 is withheld below twenty. Sample count is always displayed.
• • Public ranking requires explicit eligibility, consent, comparable evidence, freshness, publication review, and verified integrity. Private and synthetic results are excluded.
• • SHA-256 proves canonical payload integrity. Optional Ed25519 signing is a separate server-only proof and is not configured by default.

The current on-screen leaderboard is a tied, nonbillable synthetic fixture used to verify the generic path. It is not public evidence and makes no provider quality, latency, availability, entitlement, or pricing claim.

• • Anonymous public API and MCP operations cannot invoke paid provider APIs.
• • Permanent provider credentials remain server-only and are never serialized into public evidence.
• • Live or billable testing retains an explicit human authorization boundary.
• • Synthetic and subjective outcomes require human interpretation before production decisions.