test(unixfs): lock in PBNode field order behavior (IPIP-550) - #1212
Merged
Conversation
Opt-in Data-first PBNode field ordering behind the new io.UnixFS_v1_2026 profile, per IPIP-550. Default output is unchanged: all existing profiles keep the legacy Links-first order and their CIDs. - ipld/merkledag: PBNodeFieldOrder global and a Data-first encoder used when a profile opts in (candidate for upstreaming to go-codec-dagpb) - ipld/unixfs/io: PBNodeFieldOrder profile parameter and UnixFS_v1_2026, wired through ApplyGlobals - tests assert byte-exact fixtures from the IPIP table Refs ipfs/specs#550
Temporary pin so the PBNode field ordering tests from ipfs/gateway-conformance#304 run against boxo gateway backends. Switch back to a tagged release once one ships.
Codecov Report✅ All modified and coverable lines are covered by tests. @@ Coverage Diff @@
## main #1212 +/- ##
==========================================
- Coverage 64.34% 64.29% -0.05%
==========================================
Files 269 270 +1
Lines 27258 27270 +12
==========================================
- Hits 17539 17534 -5
- Misses 8004 8018 +14
- Partials 1715 1718 +3
... and 10 files with indirect coverage changes 🚀 New features to boost your workflow:
|
v0.14.1 shipped the ipfs/gateway-conformance#304 tests, so the moving v0.14 tag covers them again.
lidel
marked this pull request as ready for review
August 27, 2026 20:06
lidel
marked this pull request as draft
August 30, 2026 11:31
Derive PBNodeDataFirst bytes from dagpb.AppendEncode by moving the trailing Data field to the front, instead of a second hand-written encoder. One encoder owns link sorting and field presence, so a decoded block with unsorted links now re-encodes sorted under both orders, and byte parity with links-first holds by construction. Unknown PBNodeFieldOrder values return an error instead of silently encoding links-first. - fieldorder.go: moveDataFirst; godoc spells out the process-wide nature of the setting, the set-once-at-startup constraint, and cites the DAG-PB strictness section - coding.go: switch on the order after AppendEncode, error on unknown - fieldorder_test.go: unknown-order test; property test over random nodes against a protowire-based links-first oracle, covering unsorted links, nil and empty Data, multi-byte length prefixes, and CIDv0, identity, and CIDv1 link hashes
Build a three-chunk file and a sharded directory from fixed inputs under UnixFS_v1_2025 and UnixFS_v1_2026 and pin the root CIDs; the file CIDs match `ipfs add --chunker=size-1000` output under each profile. Every dag-pb block in both DAGs must lead with the profile's first field, and the two DAGs must decode to the same nodes. - profile.go: UnixFS_v1_2026 written as a full literal so the test asserts each parameter instead of reconstructing the copy - profile_test.go: per-field asserts for all three profiles; ApplyGlobals subtest checks all six globals; saveAndRestoreGlobals also restores chunk.DefaultBlockSize and helpers.DefaultLinksPerBlock so applied profiles no longer leak into later tests
Every dag-pb node with both Data and Links gets a new CID, files larger than one chunk included; single-chunk raw-leaf files keep theirs. Existing links-first directories are re-encoded when reopened through the directory API (MFS directories on their next access), a sharded root first and each child shard as it is loaded. The godocs also state why the setting is a process-wide global and that it must be applied once at startup. - profile.go: field, profile and ApplyGlobals godoc - doc.go: UnixFS_v1_2026 in the profile list, Global Settings section - CHANGELOG.md: scope, MFS re-encode, ✨ marker, PR link
lidel
added a commit
to ipfs/kubo
that referenced
this pull request
Aug 30, 2026
Pins the ipfs/boxo#1212 branch tip: data-first bytes derive from dagpb.AppendEncode so link sorting is inherited, unknown field order values return an error, and pinned end-to-end CIDs cover the profile.
A dated successor profile invites unintentional adoption and a de facto new CIDv1 default. PBNodeFieldOrder stays as a documented low-level opt-in; UnixFS_v0_2015 and UnixFS_v1_2025 now pin PBNodeLinksFirst explicitly. Tests derive data-first from UnixFS_v1_2025 plus the knob and keep asserting the same IPIP-550 fixture bytes and CIDs. Refs ipfs/specs#550
lidel
changed the title
lidel
added a commit
to ipfs/kubo
that referenced
this pull request
Sep 2, 2026
A dated successor profile invites unintentional adoption and a de facto new CIDv1 default. Import.UnixFSPBNodeFieldOrder stays as the documented low-level opt-in; unixfs-v0-2015 and unixfs-v1-2025 now pin links-first explicitly. - deps: boxo bump to the ipfs/boxo#1212 commit that drops UnixFS_v1_2026 - test/cli: field order exercised via the config knob, same IPIP-550 fixture bytes and CIDs Refs ipfs/specs#550
lidel
added a commit
to ipfs/kubo
that referenced
this pull request
Sep 4, 2026
Squash-merge of ipfs/boxo#1212 on main.
lidel
added a commit
to ipfs/kubo
that referenced
this pull request
Sep 4, 2026
- tighter CID profile entry; MFS caveat now says re-encode happens on rewrite, not read - deps: boxo pseudo-version with ipfs/boxo#1212, gateway-conformance v0.14.1
lidel
added a commit
to ipfs/kubo
that referenced
this pull request
Sep 4, 2026
* feat: opt-in unixfs-v1-2026 profile (IPIP-550) Opt-in Data-first PBNode field ordering via the new unixfs-v1-2026 config profile, per IPIP-550. Defaults and the preexisting unixfs-v0-2015 and unixfs-v1-2025 profiles are unchanged and keep their CIDs. - config: Import.UnixFSPBNodeFieldOrder (links-first default, data-first) and the unixfs-v1-2026 profile applying unixfs-v1-2025 settings plus data-first - core/node: wires merkledag.DefaultPBNodeFieldOrder from config - deps: boxo bump to the ipfs/boxo#1212 encoder commit - test/cli: byte-exact fixtures from the IPIP-550 table; pinned CIDs for existing profiles unchanged Refs ipfs/specs#550 * ci: run gateway-conformance from ipip-550 commit Temporary pin to the ipfs/gateway-conformance#304 head so the PBNode field ordering tests run against kubo. Switch back to a tagged release once one ships. * ci: gateway-conformance back to v0.14 v0.14.1 shipped the ipfs/gateway-conformance#304 tests, so the moving v0.14 tag covers them again. * docs: unixfs-v1-2026 scope and MFS re-encode State what the profile actually changes: every dag-pb node with both Data and Links gets a new CID, files larger than one chunk included, and data already in MFS is upgraded to the new order on first read (plain `ipfs files ls` or `stat` included), a sharded root before its child shards; the MFS root is re-encoded by any command that starts a node. - config/profile.go, docs/config.md: profile description and the unixfs-v1-2026 section - docs/changelogs/v0.43.md: highlight leads with the need (readers get the HAMT layout before links) and the upgrade-on-first-use behavior * chore: update boxo to 04a079ec27b1 Pins the ipfs/boxo#1212 branch tip: data-first bytes derive from dagpb.AppendEncode so link sorting is inherited, unknown field order values return an error, and pinned end-to-end CIDs cover the profile. * refactor: drop unixfs-v1-2026, keep opt-in knob A dated successor profile invites unintentional adoption and a de facto new CIDv1 default. Import.UnixFSPBNodeFieldOrder stays as the documented low-level opt-in; unixfs-v0-2015 and unixfs-v1-2025 now pin links-first explicitly. - deps: boxo bump to the ipfs/boxo#1212 commit that drops UnixFS_v1_2026 - test/cli: field order exercised via the config knob, same IPIP-550 fixture bytes and CIDs Refs ipfs/specs#550 * chore: update boxo to 02026ddcf262 Squash-merge of ipfs/boxo#1212 on main. * docs: tighten v0.43.1 changelog - tighter CID profile entry; MFS caveat now says re-encode happens on rewrite, not read - deps: boxo pseudo-version with ipfs/boxo#1212, gateway-conformance v0.14.1 * docs: tighten Import.UnixFSPBNodeFieldOrder docs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The ecosystem decodes both PBNode field orders, but by accident: nothing in boxo's tests said so, and nothing pinned which order the profiles write. ipfs/specs#550 formalizes both sides.
Fix
UnixFS_v0_2015andUnixFS_v1_2025pinPBNodeLinksFirstexplicitly, so a future refactor cannot silently change their bytes or CIDsData-first fixtures needed an encoder, kept as the documented low-levelUnixFSProfile.PBNodeFieldOrderopt-in; no profile enables it, and enabling it changes every written CIDDefaults are unchanged everywhere. Kubo wires the same knob as
Import.UnixFSPBNodeFieldOrder(ipfs/kubo#11439).