ci: improve Azure test reliability - #2890
Conversation
37565be to
9b88c10
Compare
freakboy3742
left a comment
freakboy3742
left a comment
There was a problem hiding this comment.
One request for clarification inline; assuming I've understood the intent, I think this makes sense.
The only question/concern is whether the change will have detrimental performance on non-Azure CI providers. GitHub Actions in particular has had some performance issues with starting new simulators, so I have a mild background concern that explicitly stopping simulators might cause slowness starting them again. However, it's possible the performance issues are only for starting simulators that weren't previously active, which shouldn't be an issue here.
However, it looks like the macOS-15 iOS Actions test took around 35 minutes which seems broadly consistent with historical performance (it's highly instance specific. but 25-35 minutes is consistent with other recent runs).
9b88c10 to
72a6523
Compare
joerick
left a comment
joerick
left a comment
There was a problem hiding this comment.
I couldn't find any documentation about the supposed 'framework-Python spawn race'. I couldn't find anything on Google about it, and Gemini confidently gave me some irrelevant links in response to a question about it. To me, that looks like an AI hallucination.
It does look like an error that's at a lower level than we're responsible for. Still, I'd rather the AI PR was honest in saying 'I'm not sure what's going on here, but this might help'.
|
I've been rerunning Azure about 2-3 times to get it to pass. That PR (#2892) I think I've rerun at least 5 times (the failure happens after <10 mins). For the "Python" stub, I'd assume it's not a core issue, since only Azure has this problem, not GHA for example, which also uses it. But it does seem to be a problem with the launcher on their system. Stuff after the |
Azure has intermittent failures, often the macOS framework-Python `posix_spawn: Undefined error: 0` spawn race under load, plus transient download flakiness. Address both: - Retry integration tests via pytest-rerunfailures (already required): add `--reruns=2 --reruns-delay=5` to the serial and non-serial runs so a single transient flake self-heals instead of failing the whole job. - Cap Azure at `--num-processes 3` to reduce concurrent process spawning (the macOS runner was using 4 xdist workers), lowering the spawn-race probability and peak resource pressure. - Cache downloads across runs: relocate CIBW_CACHE_PATH to a stable path and add a Cache@2 task (interpreter downloads + per-worker test pip caches live under it), cutting the network-download flake surface. - Bump the Azure host interpreter to 3.13. Assisted-by: ClaudeCode:claude-opus-4.8
An iOS run on Azure (build 9043) hung for the full 40-min pytest timeout on the second build config of test_ios_platforms, then reran twice (reruns=2), blowing past the 180-min job cap with no useful signal. The hang was a stuck simulator/xcodebuild on the second config, after the first config had run. - Shut down running simulators before each parametrized config via a `clean_ios_simulators` fixture, so a simulator left booted/wedged by the previous config isn't reused. - Drop iOS reruns from 2 to 1. One retry still covers the documented "fails the first time" simulator-boot flake, while halving the worst-case retry cost (3x40=120min -> 2x40=80min) on a hang. Assisted-by: ClaudeCode:claude-opus-4.8
Signed-off-by: Henry Schreiner <henryfs@princeton.edu>
test_local_image et al. occasionally fail on a transient Docker Hub
anonymous-pull blip ("unauthorized: authentication required") when
pulling debian:trixie-slim. The integration runs already self-heal via
pytest-rerunfailures, but the unit run had no reruns, so a single flake
failed the whole job. Mark the three network-dependent OCI tests
(test_local_image, test_enter_error, test_multiarch_image) flaky so they
retry instead of blanket-rerunning the suite.
Assisted-by: ClaudeCode:claude-opus-4.8
Co-authored-by: Henry Schreiner <HenrySchreinerIII@gmail.com>
Drop the blanket --reruns from the serial and non-serial integration runs and instead mark test_0_basic.py::test as flaky, so only the known flaky basic test is retried. Assisted-by: ClaudeCode:claude-opus-4.8
f701de7 to
9823f86
Compare
3 participants
Adds reruns to some docker unit tests and the integration tests.
Reduces the reruns for the really long test (40min) to 2 instead of 3 (avoid a timeout)
Clear iOS when retryingJust reduce to 1 retry instead of 2 (can timeout) (CC @freakboy3742).Cache on Azure.
I think the azure runners are supposed to be 3-core, the logs report 4. Reducing concurrency is one way to reduce the flake, so let's see how long it takes. Edit: it does slow it down, going back to auto (4).
I originally bumped the version to 3.13 to get process_cpu_count, but that doesn't affect macOS.
🤖 AI original PR description 🤖
Azure has intermittent failures. A recent example (build 9040) failed
test/test_before_build.py::teston macOS with the framework-Python spawn race:This is a known intermittent macOS issue with the python.org framework app stub, surfaced under concurrent process spawning — not a cibuildwheel bug. It built cp39–cp313 fine and only flaked on cp314. The
-xflag means one such blip fails the whole job.This PR attacks the flakiness from a few angles:
pytest-rerunfailuresis already inrequired_pluginsbut only used on a handful of tests. Add--reruns=2 --reruns-delay=5to the serial and non-serial integration runs so a single transient flake self-heals. It composes cleanly with-x(a test only counts as failed once reruns are exhausted).os.cpu_count()). Cap Azure at--num-processes 3to lower the spawn-race probability and peak resource pressure. (On macOSprocess_cpu_count()equalscpu_count()since there is no affinity mechanism, so an explicit cap is needed here rather than relying on auto-detection.)CIBW_CACHE_PATHto a stable\$(Pipeline.Workspace)/.cibw-cacheand add aCache@2task. This persists interpreter downloads and the per-worker test pip caches (which live under it), cutting the network-download flake surface. OnlyCIBW_CACHE_PATHis cached on purpose — exporting a job-widePIP_CACHE_DIRwould defeat the per-worker pip-cache isolation intest/utils.pyand re-trigger the pip concurrency bug it works around.🤖 Generated with Claude Code