Include labeled post IDs in Under the Hood reports - #92
Open
jnadeau207-collab wants to merge 1 commit into
Open
Conversation
Carry logical post IDs through the daily and backfill post-label rows, persist them in optional Thrift fields, aggregate the newest 1,000 distinct IDs per label into monthly rows, and emit them as strings in reportJson alongside postIdsComplete. IDs use the same logical-post identity (initialTweetId.getOrElse(tweetId)) as the existing carried counts, so an edited post chain stays one post in both the count and the ID list. Aggregation keeps the existing map-side combining: UthPostIds.merge is an associative, commutative reduce that sums carried and removed exactly as the previous sum did, and bounds the ID list at every merge. Peak reducer memory per key stays bounded rather than growing with posts per key, and the daily persisted rows carry the same bound as the monthly rows. Field IDs 4 and 11 are intentional, leaving 3 and 10 for the source-provenance fields proposed in xai-org#52.
jnadeau207-collab
force-pushed
the
fix/uth-labeled-post-ids
branch
from
September 3, 2026 00:32
b6de50b to
50fae33
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #85
Problem
Under the Hood reports aggregate post-label counts but discard the logical post IDs already available in the daily jobs. An account owner can see that posts were labeled, but cannot identify which posts were affected.
Change
reportJson, pluspostIdsCompleteto distinguish complete results from historical or capped resultsThe IDs use the same logical-post identity (
initialTweetId.getOrElse(tweetId)) as the existing carried counts, so an edited post chain remains one post in both the count and the ID list.Field IDs 4 and 11 are intentionally used, leaving 3 and 10 available for the source-provenance fields proposed in #52.
Aggregation shape
The daily rollups keep the map-side combining they already had.
UthPostIds.mergeis an associative, commutative reduce used throughGrouped.reduce, matching the.reduce { ... }.toTypedPipeshape already used forinHorizonin the same file:carriedandremovedare summed exactly as the previous.sumsummed(1L, removed), so both metrics are byte-for-byte unchanged.That last point matters for two reasons. Peak reducer memory per key stays bounded instead of growing with posts-per-key, and the persisted daily rows carry the same 1,000-ID bound as the monthly rows rather than being unbounded. Applying the bound per day cannot change which IDs survive the monthly bound: an ID in the monthly newest-1,000 is necessarily in its own day's newest-1,000, since IDs sort chronologically. That property is covered by a test.
Compatibility
postIdsComplete = falseuntil backfilled.Verification
under-the-hood/src/test/scala/com/twitter/visibility/under_the_hood/UthPostIdsSpec.scalais committed with the change and covers merge associativity and commutativity, identity, removed-count reconciliation, deterministic ordering, the 1,000-ID bound, exactcarriedcounts under capping, and the per-day/monthly cap equivalence above.scalac2.12.20 compiledUthPostIds.scalaand the spec. This is a real typecheck, not a parse.UthPostIds.mergefails 2 of those tests, so the suite is not vacuous.main(85ac72a1bba41f21615e3f0bca56da75970a6633).The public export does not include a runnable Under-the-Hood build or Strato typechecking target, so the Scalding and Strato sources still require a full internal build; only the dependency-free helper and its spec are compiled and executed above.
contributed by Jesse Nadeau