Periodic OpenCode Zen Model & Effort-Level Benchmarking · Issue #47232 · anomalyco/opencode

mikemackay00 · 2026-09-04T06:36:17.000Z

### Feature hasn't been suggested before. - [x] I have verified this feature I'm about to request hasn't been suggested before. ### Describe the enhancement you want to request I’m a user of OpenCode and the OpenCode Zen API, and I have a suggestion that I think would be extremely valuable to the community. Would OpenCode consider periodically benchmarking all model/effort-level combinations available through the Zen API and publishing the results on the OpenCode website? Zen is particularly well positioned to provide this because you control the model/provider configurations and already test and benchmark the models you make available. At the moment, however, it’s difficult for users to answer a very practical question: Which Zen model and reasoning-effort level gives me the best coding performance for the money? Artificial Analysis provides excellent coding-agent benchmarks, but its coverage of OpenCode/Zen model-effort combinations is relatively limited. OpenCode could potentially fill this gap with an authoritative benchmark specifically representing the Zen experience. Suggested benchmark For every supported Zen model, test each applicable effort/reasoning level using the same standardized set of coding-agent tasks , for example: Repository understanding Feature implementation Bug fixing Refactoring Test creation/fixing Multi-file architectural changes For each model/effort combination, publish: Overall benchmark score Task success rate Tests passed Average/median completion time Input/output/reasoning tokens Cost per task Failure rate Human intervention, if applicable The resulting table could look something like: Model | Effort | Score | Success | Time | Tokens | Cost/task -- | -- | -- | -- | -- | -- | -- GPT-5.6 Luna | low | … | … | … | … | … GPT-5.6 Luna | medium | … | … | … | … | … GPT-5.6 Luna | high | … | … | … | … | … GPT-5.6 Luna | max | … | … | … | … | … Gemini 3.8 Flash | high | … | … | … | … | … Muse Spark | xhigh | … | … | … | … | … I would also strongly recommend publishing a Pareto frontier of quality vs. cost , since that would make the data considerably more useful for selecting models. Periodic re-testing Ideally, this would be repeated whenever the Zen catalog changes materially—perhaps monthly or quarterly —with historical results retained. That would allow users to see not only which models perform best, but also how the economics and performance of the Zen ecosystem are changing over time. Why I think this would be particularly useful One of the most interesting things about Zen is that it gives users access to multiple models through a relatively consistent coding-agent environment. That makes it possible to compare the actual model + OpenCode harness + effort configuration , rather than relying on generic model benchmarks. It would also help answer questions such as: Is GPT-5.6 Luna high actually worth the additional cost over medium? Does increasing reasoning effort materially improve task completion? Which inexpensive models are genuinely competitive? Where is the quality/cost sweet spot? Which models are best for repository-scale engineering versus simple coding tasks? Does a model’s performance change significantly when operated through OpenCode compared with another agent harness? A public benchmark like this could become one of the strongest differentiators of OpenCode Zen. Even a relatively small standardized benchmark would be useful. The computational/API cost should be quite manageable if the benchmark consists of a modest number of representative tasks, and the results could be published alongside the existing Zen model catalog. I’d be very interested in seeing something like an “OpenCode Zen Coding Agent Benchmark” become a regularly updated public resource. Thanks for building OpenCode and Zen. It’s a very compelling approach, and I think transparent model/effort benchmarking would make it even more useful for serious users trying to choose the right configuration. Best regards, Michael

Skip to content

View all features
- BY COMPANY SIZE
  Enterprises
  Small and medium teams
  Startups
  Nonprofits
- BY USE CASE
  App Modernization
  DevSecOps
  DevOps
  CI/CD
  View all use cases
- BY INDUSTRY
  Healthcare
  Financial services
  Manufacturing
  Government
  View all industries
View all solutions
- EXPLORE BY TOPIC
  AI
  Software Development
  DevOps