Lower post-candidate cache compression from zstd 6 to zstd 1

RedisPostCandidateCacheSideEffect compresses ~2 MB of serde_json on every
cache-miss For You request (MaxPostsToCache = 750 candidates, each a 54-field
PostCandidate). The payload is highly repetitive because all 750 share one
schema, so zstd's fast strategy already captures nearly all of the redundancy
and the higher levels buy very little.

Measured on a reconstructed 750-candidate slate, libzstd 1.5.7:

  level      bytes       ms    vs level 6
  1        349,857     1.63    -6.0% size,  5.9x faster
  2        379,683     2.51    +2.1% size,  3.9x faster
  3        404,066     3.54    +8.6% size,  2.7x faster
  6        372,017     9.72    (current)
  9        354,442    14.22    -4.7% size,  0.7x

Levels 2-4 are both slower and larger than level 1 here, so this is not a
size/speed tradeoff: level 1 dominates them on both axes. Held across three
text-entropy models (24-word, 2000-word, random) and two seeds; level 1 stayed
within -6.0%/+2.0% of level 6 on size at 5.7-7.7x less CPU.

Since the entry only lives for REDIS_TTL_SECONDS = 180, request-path CPU
dominates the value of a marginally smaller blob.

zstd frames are self-describing and the cache key does not encode the
compression level, so CachedPostsQueryHydrator reads entries written by hosts
on either side of a rolling deploy. Added a regression test pinning that so the
level stays retunable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: gauss2302

home-mixer/side_effects/redis_post_candidate_cache_side_effect.rs

@@ -8,7 +8,16 @@ use xai_candidate_pipeline::component_library::utils::is_prod;
 use xai_candidate_pipeline::side_effect::{SideEffect, SideEffectInput};
 
 const REDIS_TTL_SECONDS: u64 = 180;
-const ZSTD_COMPRESSION_LEVEL: i32 = 6;
+// This payload is highly repetitive — ~750 candidates sharing one 54-field schema —
+// so zstd's fast strategy already captures nearly all of the redundancy. Measured on
+// a reconstructed 750-candidate slate (~2.0 MB of serde_json), level 1 compresses
+// 5.7-7.7x faster than level 6 while producing output within a few percent of the
+// same size, and is both faster *and* smaller than the intermediate levels 2-4.
+// Since the entry only lives for REDIS_TTL_SECONDS, request-path CPU dominates the
+// value of a marginally smaller blob. Decompression is level-agnostic, so
+// CachedPostsQueryHydrator reads frames written at any level and no cache key
+// version bump is required.
+const ZSTD_COMPRESSION_LEVEL: i32 = 1;
 
 pub struct RedisPostCandidateCacheSideEffect {
     redis_client: Arc<dyn RedisClient>,
@@ -211,6 +220,33 @@ mod tests {
         assert_eq!(result[max_posts_to_cache - 1].tweet_id, 450);
     }
 
+    /// The cache key is not versioned by compression level, so during a rolling
+    /// deploy a host on the new level reads entries a host on the old level wrote,
+    /// and vice versa. zstd frames are self-describing, so any level decodes with
+    /// the same call — this test pins that so the level can be retuned freely.
+    #[test]
+    fn payload_written_at_any_zstd_level_round_trips() {
+        let candidates = vec![candidate(1, 100, Some(5.0)), candidate(2, 200, Some(3.0))];
+        let json = serde_json::to_vec(&candidates).expect("serialize");
+
+        // 6 is the level this cache was written with before ZSTD_COMPRESSION_LEVEL
+        // was lowered; 1 is the current level.
+        for level in [1, 3, 6, 9] {
+            let compressed = zstd::encode_all(json.as_slice(), level).expect("compress");
+            let decompressed = zstd::decode_all(compressed.as_slice()).expect("decompress");
+            assert_eq!(
+                decompressed, json,
+                "level {level} frame did not decode to the original bytes"
+            );
+
+            let decoded: Vec<PostCandidate> =
+                serde_json::from_slice(&decompressed).expect("deserialize");
+            assert_eq!(decoded.len(), 2);
+            assert_eq!(decoded[0].tweet_id, 1);
+            assert_eq!(decoded[1].tweet_id, 2);
+        }
+    }
+
     #[test]
     fn get_candidates_to_cache_filters_none_and_zero_scores() {
         let selected = vec![