Skip to content

Comprehensive overhaul and hardening of MessageDifferencer microbenchmarks in message_differencer_unittest.cc to accurately model production serving conditions and eliminate benchmark gaming vectors: - #29535

Draft
copybara-service[bot] wants to merge 1 commit into
mainfrom
test_974107248

Conversation

@copybara-service

@copybara-service copybara-service Bot commented Aug 31, 2026

Copy link
Copy Markdown

Comprehensive overhaul and hardening of MessageDifferencer microbenchmarks in message_differencer_unittest.cc to accurately model production serving conditions and eliminate benchmark gaming vectors:

  • Equal-Length String Mismatches: Replaced string length alterations (appending "_mismatch" or altering index digit lengths) with in-place character mutation (MutateStringEqualLength) across repeated lists, repeated sets (kDisjoint, kPartialOverlap), and map keys. This prevents string comparisons from early-exiting on string size check (lhs.size() != rhs.size()) without reading memory payloads, properly exercising SIMD/memcmp comparisons.
  • Cardinality Spectrum for Repeated Sets: Expanded repeated set benchmarks from artificial small N <= 15 to a representative cardinality spectrum (N in {2, 10, 50, 200, 1000}). This prevents microbenchmarks from artificially favoring O(N^2) pairwise brute force over scalable O(N) hash/sort algorithms on production-scale collections.
  • Distributed Mismatch Positions:
    • Repeated lists: Mismatches are distributed across indices (first, middle, last) rather than exclusively mutating index 0.
    • Wide schemas: Mismatches exercise multiple field types and tag positions (scalars, strings, enums, nested submessages) rather than exclusively aborting on tag 1 optional_int32.
    • Deep schemas: Mismatch paths ensure full traversal to leaf nodes, evaluating all leaf fields rather than short-circuiting on tag 1.
  • Field Presence Asymmetry: Added dedicated kPresenceAsymmetry benchmarks exercising FieldBefore(field1, field2) and FieldBefore(field2, field1) comparison paths where one message populates fields that the other lacks.
  • Map Field Realism: Separated map field presence asymmetry from key lookup failures (ContainsMapKey) and value comparisons, using equal-length key mutations.
  • Entry Point Ephemerality & Setup Measurement: Added ephemeral TreatAsSet repeated set evaluations alongside long-lived instance benchmarks to capture per-comparison configuration overhead versus steady-state comparison.
  • Benchmark Deduplication: Removed redundant benchmark template instantiations (BM_EntryPoint_Wide_Compare / BM_WideSchema_Identical and BM_EntryPoint_Deep_Compare / BM_DeepSchema_Identical).
  • Pool Integrity Validation: Expanded automated unit tests (ValidateWideMessagePool, ValidateDeepMessagePool, ValidateMapMessagePool, ValidateRepeatedListMessagePool, ValidateRepeatedSetMessagePool) to verify behavioral correctness across all scenarios, field types, and cardinalities up to N=1000.

…marks in message_differencer_unittest.cc to accurately model production serving conditions and eliminate benchmark gaming vectors:

- Equal-Length String Mismatches: Replaced string length alterations (appending "_mismatch" or altering index digit lengths) with in-place character mutation (MutateStringEqualLength) across repeated lists, repeated sets (kDisjoint, kPartialOverlap), and map keys. This prevents string comparisons from early-exiting on string size check (lhs.size() != rhs.size()) without reading memory payloads, properly exercising SIMD/memcmp comparisons.
- Cardinality Spectrum for Repeated Sets: Expanded repeated set benchmarks from artificial small N <= 15 to a representative cardinality spectrum (N in {2, 10, 50, 200, 1000}). This prevents microbenchmarks from artificially favoring O(N^2) pairwise brute force over scalable O(N) hash/sort algorithms on production-scale collections.
- Distributed Mismatch Positions:
  - Repeated lists: Mismatches are distributed across indices (first, middle, last) rather than exclusively mutating index 0.
  - Wide schemas: Mismatches exercise multiple field types and tag positions (scalars, strings, enums, nested submessages) rather than exclusively aborting on tag 1 optional_int32.
  - Deep schemas: Mismatch paths ensure full traversal to leaf nodes, evaluating all leaf fields rather than short-circuiting on tag 1.
- Field Presence Asymmetry: Added dedicated kPresenceAsymmetry benchmarks exercising FieldBefore(field1, field2) and FieldBefore(field2, field1) comparison paths where one message populates fields that the other lacks.
- Map Field Realism: Separated map field presence asymmetry from key lookup failures (ContainsMapKey) and value comparisons, using equal-length key mutations.
- Entry Point Ephemerality & Setup Measurement: Added ephemeral TreatAsSet repeated set evaluations alongside long-lived instance benchmarks to capture per-comparison configuration overhead versus steady-state comparison.
- Benchmark Deduplication: Removed redundant benchmark template instantiations (BM_EntryPoint_Wide_Compare / BM_WideSchema_Identical and BM_EntryPoint_Deep_Compare / BM_DeepSchema_Identical).
- Pool Integrity Validation: Expanded automated unit tests (ValidateWideMessagePool, ValidateDeepMessagePool, ValidateMapMessagePool, ValidateRepeatedListMessagePool, ValidateRepeatedSetMessagePool) to verify behavioral correctness across all scenarios, field types, and cardinalities up to N=1000.

PiperOrigin-RevId: 974107248
@copybara-service copybara-service Bot changed the title Update MessageDifferencer repeated set benchmarks for high-fidelity production modeling. Comprehensive overhaul and hardening of MessageDifferencer microbenchmarks in message_differencer_unittest.cc to accurately model production serving conditions and eliminate benchmark gaming vectors: Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant