Copilot proxy mishandles fine-grained PATs and legacy provider placeholders on GHEC data-residency endpoints

[jaroslawgajewski]

Summary

AWF v0.28.10 has three independent Copilot proxy regressions when a GitHub fine-grained PAT is used with a GHEC data-residency copilot-api.<tenant>.ghe.com endpoint:

1. Authorization scheme selection is based only on the target host, so these targets receive Authorization: token ... even when the credential is a fine-grained PAT that requires Bearer.
2. GitHub-hosted model discovery does not consistently preserve the required Copilot request shape, including X-GitHub-Api-Version: 2026-07-01 and the Copilot integration identity.
3. A historical offline/custom-provider credential-isolation placeholder can be interpreted as a real COPILOT_PROVIDER_API_KEY. Because BYOK credentials take precedence for inference, /models uses the real GitHub PAT and succeeds while inference silently uses the placeholder and returns HTTP 400.

These layers fail independently and require separate fixes.

The target-only Authorization bug also applies to Copilot Business and Enterprise endpoints. That applicability is covered by direct-path evidence and focused regression tests; the final end-to-end live canary described below used a GHEC data-residency endpoint.

Sanitized reproduction and version matrix

The controlled runs used this non-secret configuration shape:

COPILOT_GITHUB_TOKEN=<GitHub fine-grained PAT>
COPILOT_PROVIDER_API_KEY=placeholder-token-for-credential-isolation
COPILOT_API_TARGET=copilot-api.<tenant>.ghe.com
COPILOT_INTEGRATION_ID=copilot-developer-cli

The Copilot CLI version was 1.0.80 throughout.

Runtime	Expected/observed transition
AWF v0.25.55	Working baseline: /models succeeds, inference succeeds with positive token usage, and safe outputs are produced.
Stock AWF v0.28.10	/models returns HTTP 400; inference returns HTTP 400 with zero token usage and no safe output.
Credential-aware Authorization patch	Safe diagnostic becomes fine_grained_pat/Bearer; /models and inference still return HTTP 400.
GitHub-hosted model-discovery request-shape patch	/models returns HTTP 200 with 26 models; inference still returns HTTP 400 with zero token usage.
Historical provider-placeholder compatibility patch	/models remains HTTP 200; inference returns HTTP 200 with positive token usage and a validated safe output.

Expected behavior

• A github_pat_... credential uses Bearer for startup model discovery, proxied GET /models, and inference on Business, Enterprise, and GHEC data-residency Copilot targets.
• GitHub OAuth and classic PAT credentials retain the existing token scheme on targets that require it.
• GitHub-hosted /models requests include the required API version and Copilot integration identity.
• Credential-isolation placeholders never become upstream Authorization credentials.
• Genuine COPILOT_PROVIDER_API_KEY values retain BYOK precedence.

Proposed fixes

1. Select the GitHub Authorization scheme from the resolved credential

Normalize the GitHub credential first, classify github_pat_ as a fine-grained PAT, and use Bearer for that credential on every target. Preserve the existing target-based token behavior for OAuth and classic PAT credentials.

Pass the credential actually selected by the adapter into scheme selection instead of re-reading a potentially missing or placeholder environment value. Use the resulting GitHub-token scheme consistently for startup /models, proxied /models, and GitHub-token inference.

2. Preserve the GitHub-hosted Copilot request shape

Treat canonical copilot-api.<tenant>.ghe.com targets as GitHub-hosted Copilot endpoints rather than generic BYOK routers.

Add X-GitHub-Api-Version: 2026-07-01 to both startup and proxied /models requests, and resolve the integration identity in this order:

1. COPILOT_INTEGRATION_ID
2. GITHUB_COPILOT_INTEGRATION_ID
3. The existing default

Preserve Copilot-Integration-Id for canonical GHEC inference without synthesizing unrelated enterprise interaction headers.

This request-shape hardening is covered separately; it was not sufficient to fix the remaining inference HTTP 400 by itself.

3. Reject historical provider-key placeholders before BYOK precedence

Extend resolveApiKey() to treat the historical placeholder-token-for-credential-isolation value as absent, alongside the current AWF placeholder and dummy-byok-key-for-offline-mode.

When a GitHub PAT is also configured, inference then correctly falls back to that PAT.

Expose only enum-valued diagnostics for the credential actually selected for inference, for example:

{
  "credential_kind": "fine_grained_pat",
  "selected_scheme": "Bearer",
  "inference_credential_source": "github_token",
  "inference_selected_scheme": "Bearer",
  "integration_id_source": "copilot_integration_id"
}

No credential material is included in these diagnostics.

Private live evidence

A controlled private GHEC data-residency A/B used the same tenant endpoint, fine-grained PAT, Copilot CLI version, model, and workflow:

• AWF v0.25.55 completed real inference successfully.
• Stock v0.28.10 returned HTTP 400 for /models and inference.
• After the credential-aware scheme fix, the safe diagnostic reported fine_grained_pat with Bearer, but /models still failed.
• After the model-discovery request-shape fix, /models returned HTTP 200 with 26 models, while inference still returned HTTP 400.
• After rejecting the historical provider placeholder, the final canary completed successfully with:
  • /models HTTP 200;
  • 26 available models;
  • 24 successful inference records with HTTP 200;
  • positive aggregate token usage;
  • a validated and successfully published safe output.

The historical placeholder remained present in the final canary, so the run exercised the compatibility branch rather than bypassing it through workflow configuration.

Only status codes, counts, header presence/categories, credential-source enums, and non-sensitive routing fields were recorded.

No token value, request or response body, prompt/message content, or safe-output content was logged.

GitHub.com Copilot Business scope

The complete fix was validated end-to-end with the GHEC data-residency credential and endpoint.

A GitHub.com Copilot Business credential was exercised in earlier live tests, before all three fixes were combined. Those tests confirmed the relevant Authorization and model-discovery behavior, but inference still failed because the historical provider-placeholder precedence bug had not yet been fixed.

Therefore:

• Business and Enterprise applicability is covered by direct-path evidence and automated regression tests.
• Final end-to-end live proof currently covers GHEC data residency.
• A separate live canary using the final combined fix with api.business.githubcopilot.com would be required to claim end-to-end Copilot Business validation.

Regression coverage

Focused tests cover:

• fine-grained PATs on Business and canonical GHEC data-residency targets;
• startup /models, proxied /models, and inference scheme consistency;
• OAuth and classic PAT token behavior;
• standard Copilot endpoint behavior;
• genuine BYOK key precedence;
• current, offline-mode, and historical placeholder rejection;
• Copilot integration identity precedence;
• privacy-safe reflected diagnostics without credential material;
• canonical GHEC header handling.

Validation completed with:

• 4 focused API-proxy suites and 188 tests;
• the full API-proxy suite: 80 suites and 1,660 tests;
• the cumulative branch full repository suite: 338 suites and 5,459 tests.

Confirmed end-to-end result

The final private GHEC canary produced:

{
  "auth_diagnostic": {
    "credential_kind": "fine_grained_pat",
    "selected_scheme": "Bearer",
    "inference_credential_source": "github_token",
    "inference_selected_scheme": "Bearer",
    "integration_id_source": "copilot_integration_id"
  },
  "models_http_status": 200,
  "models_count": 26,
  "inference_statuses": [200],
  "successful_inference_count": 24,
  "tokenCount": 58530,
  "safe_output_types": ["add_comment"]
}

All workflow jobs completed successfully, including inference, safe-output validation, and safe-output publication.

[jaroslawgajewski]

Root cause in code

The regression spans three independent paths in the API proxy:

1. copilotTargetRequiresGitHubTokenPrefix() selects the Authorization scheme
   from the target hostname without considering the resolved credential type.
   The adapter should pass the normalized credential it actually selected rather
   than re-reading an environment variable.

2. GitHub-hosted model discovery and request-header processing do not fully
   recognize canonical copilot-api.<tenant>.ghe.com targets. This can omit the
   required API version or remove the Copilot integration identity before the
   request is sent.

3. resolveApiKey() recognizes the current credential-isolation sentinels but
   not the historical literal
   placeholder-token-for-credential-isolation. Consequently, inference gives
   this placeholder BYOK precedence over COPILOT_GITHUB_TOKEN, while /models
   independently uses the valid GitHub credential.

[jaroslawgajewski]

@lpcox up to my tests that should fix a problem with GHEC copilot tokens I'm facing from afetr version 25.55 😄