Skip to content

fix(workflows): keep non-ASCII text readable in written overlay files - #4148

Merged
mnriem merged 1 commit into
github:mainfrom
jawwad-ali:fix/overlay-allow-unicode
Sep 1, 2026
Merged

fix(workflows): keep non-ASCII text readable in written overlay files#4148
mnriem merged 1 commit into
github:mainfrom
jawwad-ali:fix/overlay-allow-unicode

Conversation

@jawwad-ali

Copy link
Copy Markdown
Contributor

Problem

Both overlay writers in overlays/_commands.py (lines 217 and 270 — the only two yaml.safe_dump calls in the module) omit allow_unicode=True:

staged.write_bytes(yaml.safe_dump(data, sort_keys=False).encode("utf-8"))

So every non-ASCII character in a user's overlay is rewritten as a \uXXXX / \xNN escape inside a double-quoted scalar.

Every other YAML writer in the repo already passes allow_unicode=True:

src/specify_cli/agents.py:163
src/specify_cli/bundler/lib/yamlio.py:91
src/specify_cli/extensions/_commands.py:530, 568
src/specify_cli/extensions/__init__.py:4800
src/specify_cli/integrations/base.py:69, 1429, 1440

Why it matters

Overlay files are explicitly hand-authored and hand-editeddocs/reference/workflows.md documents the file format and tells users to write these files. overlay add, enable, disable and set-priority all round-trip the file through safe_dump, so merely toggling an overlay mangles a UTF-8 file the user wrote by hand.

Reproduction on current main

Source overlay written in UTF-8 with message: "Revisar el plan — ¿aprobar? 日本語", then specify workflow overlay add overlay.yml:

id: revisar
extends: wf
priority: 10
edits:
- replace: a
  step:
    id: a
    type: gate
    message: "Revisar el plan — \xBFaprobar? 日本語"
    options:
    - approve

The value still parses back identically, so this is not corruption — it is the loss of a documented, hand-edited file's legibility, the same thing allow_unicode=True is already there to prevent everywhere else.

Verification

  • Fail-before / pass-after: 2 new-vs-baseline failures with the source reverted → 25 passed with the fix.
  • One test per writer: overlay add (the install path) and overlay set-priority (the _update_overlay_field round-trip path).
  • Each asserts both that the text survives readable and that it still round-trips to the identical value through yaml.safe_load.
  • Scoped regression over tests/workflows: no new failures vs a clean-main baseline (10 pre-existing, Windows symlink-privilege).
  • uvx ruff@0.15.0 check src tests → clean

No breaking change. allow_unicode only affects how characters are encoded in the output; the parsed value is byte-identical either way (both new tests assert this explicitly), and files are already written as UTF-8 via .encode("utf-8"). Pure-ASCII overlays — the overwhelming majority — produce identical bytes.


Written with assistance from Claude Code. Bug found, reproduced, and verified by me on current main.

Both overlay writers in `overlays/_commands.py` called
`yaml.safe_dump(data, sort_keys=False)` without `allow_unicode=True`, so
every non-ASCII character was rewritten as a `\uXXXX` / `\xNN` escape inside
a double-quoted scalar. Every other YAML writer in the repo already passes
`allow_unicode=True` (agents.py, bundler/lib/yamlio.py, extensions,
integrations/base.py, ...).

Overlay files are explicitly hand-authored and hand-edited -- the format is
documented in docs/reference/workflows.md and users are told to write these
files. `overlay add`, `enable`, `disable` and `set-priority` all round-trip
the file through `safe_dump`, so merely toggling an overlay mangled a UTF-8
file the user wrote by hand:

    message: "Revisar el plan — \xBFaprobar? 日本語"

The value still parses back identically, so this is not corruption -- it is
the loss of a documented, hand-edited file's legibility.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@jawwad-ali
jawwad-ali requested a review from mnriem as a code owner August 15, 2026 16:22
@mnriem
mnriem requested a balanced review from