Skip to content

Include labeled post IDs in Under the Hood reports - #92

Open
jnadeau207-collab wants to merge 1 commit into
xai-org:mainfrom
jnadeau207-collab:fix/uth-labeled-post-ids
Open

Include labeled post IDs in Under the Hood reports#92
jnadeau207-collab wants to merge 1 commit into
xai-org:mainfrom
jnadeau207-collab:fix/uth-labeled-post-ids

Conversation

@jnadeau207-collab

@jnadeau207-collab jnadeau207-collab commented Sep 2, 2026

Copy link
Copy Markdown

Fixes #85

Problem

Under the Hood reports aggregate post-label counts but discard the logical post IDs already available in the daily jobs. An account owner can see that posts were labeled, but cannot identify which posts were affected.

Change

  • carry logical post IDs through the daily and backfill post-label rows
  • persist them in optional Thrift fields
  • aggregate the newest 1,000 distinct IDs per label into monthly rows
  • emit IDs as strings in reportJson, plus postIdsComplete to distinguish complete results from historical or capped results

The IDs use the same logical-post identity (initialTweetId.getOrElse(tweetId)) as the existing carried counts, so an edited post chain remains one post in both the count and the ID list.

Field IDs 4 and 11 are intentionally used, leaving 3 and 10 available for the source-provenance fields proposed in #52.

Aggregation shape

The daily rollups keep the map-side combining they already had. UthPostIds.merge is an associative, commutative reduce used through Grouped.reduce, matching the .reduce { ... }.toTypedPipe shape already used for inHorizon in the same file:

  • carried and removed are summed exactly as the previous .sum summed (1L, removed), so both metrics are byte-for-byte unchanged.
  • the ID list is bounded at every merge, not only at the end.

That last point matters for two reasons. Peak reducer memory per key stays bounded instead of growing with posts-per-key, and the persisted daily rows carry the same 1,000-ID bound as the monthly rows rather than being unbounded. Applying the bound per day cannot change which IDs survive the monthly bound: an ID in the monthly newest-1,000 is necessarily in its own day's newest-1,000, since IDs sort chronologically. That property is covered by a test.

Compatibility

  • The new Thrift fields are optional.
  • Existing monthly rows continue to serve; they return an empty ID list with postIdsComplete = false until backfilled.
  • The backfill job is updated to repopulate historical IDs.
  • Monthly IDs are capped at 1,000 per label to bound persisted and response size; exact aggregate counts remain unchanged.

Verification

under-the-hood/src/test/scala/com/twitter/visibility/under_the_hood/UthPostIdsSpec.scala is committed with the change and covers merge associativity and commutativity, identity, removed-count reconciliation, deterministic ordering, the 1,000-ID bound, exact carried counts under capping, and the per-day/monthly cap equivalence above.

  • scalac 2.12.20 compiled UthPostIds.scala and the spec. This is a real typecheck, not a parse.
  • All 14 tests in the committed spec pass.
  • Removing the bound from UthPostIds.merge fails 2 of those tests, so the suite is not vacuous.
  • Apache Thrift 0.19 accepted the schema.
  • Static end-to-end pipeline and report contract checks passed.
  • The branch is one commit directly on current upstream main (85ac72a1bba41f21615e3f0bca56da75970a6633).

The public export does not include a runnable Under-the-Hood build or Strato typechecking target, so the Scalding and Strato sources still require a full internal build; only the dependency-free helper and its spec are compiled and executed above.

contributed by Jesse Nadeau

Carry logical post IDs through the daily and backfill post-label rows,
persist them in optional Thrift fields, aggregate the newest 1,000
distinct IDs per label into monthly rows, and emit them as strings in
reportJson alongside postIdsComplete.

IDs use the same logical-post identity (initialTweetId.getOrElse(tweetId))
as the existing carried counts, so an edited post chain stays one post in
both the count and the ID list.

Aggregation keeps the existing map-side combining: UthPostIds.merge is an
associative, commutative reduce that sums carried and removed exactly as
the previous sum did, and bounds the ID list at every merge. Peak reducer
memory per key stays bounded rather than growing with posts per key, and
the daily persisted rows carry the same bound as the monthly rows.

Field IDs 4 and 11 are intentional, leaving 3 and 10 for the
source-provenance fields proposed in xai-org#52.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Under the Hood JSON should include post IDs for labeled posts

1 participant