spec-driven · microservices · bench
Case study · v2 · snapshot as of July 21, 2026

archspec vs superpowers: How Spec-Driven Development Changes a Microservices Solution

This case study is for microservices developers and architects. Six models solved the same event-driven task in Go within one repository, using two approaches: a one-shot design document and the archspec process. Each model-and-process combination was run once (n=1). The results apply only to this repository and scenario.

Task: smart freelancer reassignment System: 12 microservices, Go Design: 6 models × 2 processes Replicates: 1 run per condition Method B: archspec
The Task

Reassign a Freelancer without Breaking Distributed-System Invariants

After the client rejects an offer, the system selects the next worker from the existing candidate pool. It can reassign a worker up to three times. After the fourth rejection, the task moves to failed status and the client receives a notification. The solution touches eight distributed-system invariants, including idempotency, atomic state-and-event persistence, service boundaries, and delivery order. This is where the models diverge.

8 traps in the task
01Access facade: private services are available only through the facade
02Data denormalization: the city ID is stored on the task; the system does not resolve a city name to an ID at request time
03Idempotency: each attempt needs a unique key, or duplicate events suppress one another
04Atomicity: state and event are written in one transaction through an outbox; events are never published directly
05N+1 problem: data is fetched in a batch instead of through individual calls in a loop
06Cyclic dependencies: services do not call one another in a cycle
07Event-driven entry: the flow enters through the event bus; the subscriber does not make a synchronous call
08Order and ownership: the event is published after persistence; only the state owner changes the status
Two Methods
Method A · baseline run

superpowers

The model writes one design document and then starts coding. The plan does not reference live service contracts, and the process includes no plan review. Initial assumptions therefore reach the implementation unchecked.

one plan 0% evolution no contract check
Method B · archspec

spec-driven

The Contract Slice gathers data from the live SERVICE_MAP. Open Questions records the decisions, the Fan-Out Trace lists subscribers, and the State-Ownership Map identifies state owners. The model then performs self-review and plan-review. Before implementation, it checks every invariant against the existing code.

Contract Slice Open Questions Fan-Out Trace self / plan-review

The values come from plan-review and code-review artifacts that follow the same format. The artifacts do not identify the exact reviewer model. The plan score, code score, 21-item implementation checklist, and end-to-end status measure different properties of the solution. A high checklist score therefore does not guarantee that the code builds or works.

The paired comparison covers 12 runs.

Run Conditions

Models, Access Environments, and Effort

Model names match the interfaces available when the runs were performed. Effort settings are comparable only when the client uses the same scale.

Model Environment and access Effort Parameters
Claude Sonnet 4.6 Claude Code medium · from the user's log not disclosed
GPT-5.4 in Codex Codex medium · from the user's log not disclosed
Gemini 3.5 Flash OpenCode / OpenCode Zen medium · provider setting at run time not disclosed
GLM-5 OpenCode / OpenCode Zen; the model is now marked as legacy/deprecated effort not recorded; thinking can be toggled on or off 744 billion total / 40 billion active (MoE)
MiniMax-M2.7 OpenCode / OpenCode Zen no separate effort setting recorded 229 billion total; the primary source does not publish the active-parameter count for M2.7
Qwen3.6 Plus OpenCode / OpenCode Zen no public low / medium / high scale; setting not recorded the provider does not disclose them for the hosted Plus version

On a narrow screen, scroll the table horizontally.

Parameters ≠ Tokens

Parameters describe model size. Tokens measure context and generation volume and determine billing. For MoE models, total and active parameters are reported separately. A single figure therefore cannot support a direct comparison of models' compute costs.

OpenCode Zen charges requests against a prepaid balance on a pay-as-you-go basis. Rates and catalog contents change over time. During these runs, Zen provided access to four models in the table; GLM-5 is now considered a legacy model.

Summary

Average Results across Six Models

Plan / 10
6.3→7.9
+1.7 · improvement for all 6 models
design quality from plan-review
Code / 10
4.8→6.0
+1.3 · improvement in 5 of 6 pairs
GPT-5.4 fell from 7 to 5.5
Implementation checklist · 21
66%→89%
+23 pp · improvement for all 6 models
from unrounded values: 65.9% → 88.5% (+22.6 pp)
End-to-end works at least partially
1/6→3/6
end-to-end works at least partially
Qwen: happy path; Sonnet and GLM: partial
Full Benchmark

Model × Method

Model Without archspec With archspec Δ
plancodecheckliste2e plancodecheckliste2e plancode
Claude Sonnet 4.6 * 6.0 6.0 64% no 9.0 8.0 93% partial +3.0 +2.0
GPT-5.4 in Codex 7.0 7.0 81% yes 8.0 5.5 88% no +1.0 −1.5
Gemini 3.5 Flash 5.5 3.0 48% no 7.0 4.0 86% no +1.5 +1.0
GLM-5 6.5 5.5 79% no 8.5 7.0 95% partial +2.0 +1.5
MiniMax-M2.7 6.5 3.5 69% no 7.5 5.0 83% no +1.0 +1.5
Qwen3.6 Plus 6.0 3.5 55% no 7.5 6.5 86% happy-path +1.5 +3.0

On a narrow screen, scroll the table horizontally.

high medium / low failure * Claude Sonnet 4.6: reference pair anchoring the scale at 6/10 and 9/10 checklist: share of the 21 implementation items completed row percentages are rounded to whole numbers; the average uses unrounded values happy path: the primary end-to-end scenario is confirmed; “partial”: the flow runs, but one user requirement is violated
Before → After

How the Paired-Run Results Changed

superpowers archspec improvement
Fixed and Remaining

Architectural Problems Fixed and Left Unresolved

The left column lists defects from the run without archspec. The right column shows the traps addressed by the spec-driven plan (+) and the problems that remained in the design or code (−). “Fixed” means changed relative to the corresponding baseline run without archspec.

Claude Sonnet 4.6

partially works
plan 6.0 → 9.0 code 6.0 → 8.0 checklist 64% → 93%
Without archspec · defects 8
  • match_id collision (idempotency)
  • city_id replaced with the city name
  • offer.declined written outside the outbox
  • Synchronous call inside the subscriber
  • N+1 rating queries
  • No client event (task.reassigned)
  • Incorrect deduplication key
  • Extraneous field in the rejection
With archspec · fixed 6
  • match_id idempotency
  • Denormalized city_id
  • Events sent through the outbox
  • Client notification
  • Sender taken from the token
  • Off-by-one limit error
Remaining 3
  • Inverted tie-breaker
  • city_id values disagree across services
  • Tests codify the bug

GPT-5.4 in Codex

end-to-end does not work
plan 7.0 → 8.0 code 7.0 → 5.5 checklist 81% → 88%
Without archspec · defects 6
  • Repeated analysis during reassignment
  • No offer.rejected deduplication
  • “No candidates” does not produce failed
  • Spurious task.reassigned event
  • city_id is not denormalized
  • match_id / attempt are unset
With archspec · fixed 5
  • Explicit deduplication key
  • Batch fetch instead of N+1 queries
  • Event-driven flow with an outbox
  • Atomic state-and-event persistence
  • Correct entry point
Remaining 4
  • match.found is not published
  • match_id collision
  • Errors are suppressed and written to the log
  • Diagrams were not resynchronized

Gemini 3.5 Flash

end-to-end does not work
plan 5.5 → 7.0 code 3.0 → 4.0 checklist 48% → 86%
Without archspec · defects 8
  • offer.rejected uses the wrong outbox key
  • The match is looked up by a guessed ID
  • match.found cannot be resent
  • Incorrect rejection entry point
  • Non-atomic counter increment
  • No attempt counter in the event
  • city_id is not denormalized
  • N+1 distance queries
With archspec · fixed 4
  • Entry point and event topology
  • Reuse of the candidate snapshot
  • Distance tie-breaker
  • task.failed is set by the state owner
Remaining in the implementation 4
  • Proto generation was not run: task-service and the API gateway do not compile
  • The client receives no notification after a successful reassignment
  • The offer.declined subscriber is not idempotent
  • The Evidence artifact claims that tests passed for a package that does not compile

GLM-5

partially works
plan 6.5 → 8.5 code 5.5 → 7.0 checklist 79% → 95%
Without archspec · defects 8
  • match_id collision
  • Deduplicated before sending: event lost
  • Incorrect entry point
  • Synchronous call inside the subscriber
  • city_id used as a name through an invented call
  • offer.rejected written outside the outbox
  • Off-by-one limit error
  • Race condition without locking
With archspec · fixed 4
  • match_id idempotency
  • Denormalized city_id
  • Events sent through the outbox
  • Off-by-one limit error
Remaining 4
  • Shared match_id (new collision)
  • Incorrect deduplication key
  • Non-atomic counter increment
  • No canonical city_id value

MiniMax-M2.7

end-to-end does not work
plan 6.5 → 7.5 code 3.5 → 5.0 checklist 69% → 83%
Without archspec · defects 8
  • Rejection requires an unreachable status
  • match_id collision
  • city_id is not denormalized
  • State and event are not atomic
  • No task.rejected deduplication
  • Incorrect entry point creates an event loop
  • Duplicate client notification
  • Event is silently lost in the outbox
With archspec · fixed 4
  • match_id idempotency
  • Off-by-one limit error
  • Events sent through the outbox
  • “No candidates” → failed
Remaining 4
  • City ID is resolved from the name at request time
  • Failure event is not wired up
  • No client notification
  • Subscriber is not idempotent

Qwen3.6 Plus

happy path confirmed
plan 6.0 → 7.5 code 3.5 → 6.5 checklist 55% → 86%
Without archspec · defects 8
  • offer.sent is not published
  • offer.rejected publishing is not implemented
  • The new worker never receives an offer
  • match_id collision
  • No transactional outbox
  • Synchronous call inside the subscriber
  • city_id is not denormalized
  • Off-by-one limit error (2 instead of 3)
With archspec · fixed 4
  • Idempotency for the new event
  • Off-by-one limit error
  • Atomic persistence
  • Owner of the assignment field
Remaining 4
  • Synchronous call instead of an event
  • Non-atomic event pair
  • city_id is resolved at request time
  • No notification-service tests
Recurring Errors

Two Traps Remained Even with archspec

almost universal

city_id as a City Name

Across all 12 runs, the models interpret city_id differently. The archspec process includes the field in the contracts but does not define one canonical value set for task, worker-profile, and geo. If city-msk ≠ moscow, the distance tie-breaker stops working.

most runs

match_id Collision

Using one match_id for every attempt causes failures in the baseline runs of Claude Sonnet 4.6, GLM-5, MiniMax-M2.7, and Qwen3.6 Plus. Most models recognize the risk in their archspec plans. The collision nevertheless returns in the deduplication key of the GPT-5.4 and GLM-5 implementations.

Limitations

How to Interpret These Results

  • The experiment covers one task in one repository, with one run for each condition (n=1).
  • The clients and providers differ, and some effort settings cannot be mapped to a common scale.
  • The plan and code scores come from review artifacts produced by a human and an LLM. No separate automated test validates these scores.
  • The sample cannot establish statistical significance and is not intended as a general model ranking.

The results show mechanisms and recurring failures only in the selected scenario. They do not predict outcomes in another codebase.

Takeaways

What the 12 Runs Showed

Plan and checklist scores improved in every pair. The average plan score rose by 1.7 points, and the implementation checklist rose by 22.6 percentage points. This result held for all six models on the selected task.

Code scores improved in five of the six pairs. GPT-5.4 fell from 7 to 5.5: the structured plan did not prevent a match_id collision or a break in the primary event flow.

Plan quality and readiness to run diverged. Gemini 3.5 Flash scored 7/10 for its plan and completed 86% of the implementation checklist, but its code does not compile. Qwen3.6 Plus passes the happy path; Claude Sonnet 4.6 and GLM-5 complete it only partially.

Planning needs a separate verification stage. That stage should run code generation and the build, check for one canonical city_id, verify event idempotency and conformance between the code and plan, and execute a real integration scenario.