Skip to content

Speed up mixed-line-ending scans with memchr2 - #2391

Merged
j178 merged 1 commit into
masterfrom
simd-eol
Jul 22, 2026
Merged

Speed up mixed-line-ending scans with memchr2#2391
j178 merged 1 commit into
masterfrom
simd-eol

Conversation

@j178

@j178 j178 commented Jul 22, 2026

Copy link
Copy Markdown
Owner

Use memchr2 to skip non-line-ending bytes when counting and normalizing line endings, while continuing to treat CRLF as one ending.

Performance

Measured on an Apple M3 Pro with macOS 15.7.7 and rustc 1.97.0. Both origin/master (6c898499) and this change (64daab4d) were built with --profile profiling, then compared with hyperfine using:

prek run mixed-line-ending --all-files

The table uses pooled medians from two order-reversed runs to reduce ordering bias:

Workload Runs per binary master This PR Change
prek repo: 309 files, 3.97 MB 100 44.65 ms 44.70 ms no measurable change
20 × 1 MiB LF text files, ~83-byte lines 80 120.05 ms 100.54 ms 16.3% faster (1.19×)
20 × 1 MiB mixed LF/CRLF files, including rewrite 60 287.30 ms 274.51 ms 4.5% faster (1.05×)

The normal repository case remains dominated by process startup, Git enumeration, and file I/O. The improvement becomes visible as the amount of text scanned per invocation grows; rewriting sees a smaller gain because output I/O dominates.

Copilot AI review requested due to automatic review settings July 22, 2026 18:51
@j178 j178 added the enhancement New feature or request label Jul 22, 2026
Copilot AI reviewed Jul 22, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@prek-ci-bot

prek-ci-bot Bot commented Jul 22, 2026

Copy link
Copy Markdown

📦 Cargo Bloat Comparison

Binary size change: +0.00% (29.2 MiB → 29.2 MiB)

Expand for cargo-bloat output

Head Branch Results

 File  .text     Size             Crate Name
 1.1%   2.3% 332.0KiB        aws_lc_sys aws_lc_0_42_0_aes_gcm_encrypt_avx512
 1.1%   2.3% 332.0KiB        aws_lc_sys aws_lc_0_42_0_aes_gcm_decrypt_avx512
 0.2%   0.5%  65.4KiB              prek <prek::cli::Command as clap_builder::derive::Subcommand>::augment_subcommands
 0.2%   0.4%  55.1KiB              prek <<prek::config::Config as serde_core::de::Deserialize>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::<<serde_saphyr::de::deserializer::YamlDeserializer as serde_core::de::Deserializer>::deserialize_map::MA>
 0.2%   0.3%  49.4KiB annotate_snippets annotate_snippets::renderer::render::render
 0.2%   0.3%  47.0KiB              prek prek::run::{closure#0}
 0.1%   0.3%  38.4KiB              prek prek::cli::run::run::run::{closure#0}
 0.1%   0.2%  34.3KiB              prek <prek::cli::RunOptions as clap_builder::derive::Args>::augment_args
 0.1%   0.2%  30.1KiB     granit_parser <granit_parser::scanner::Scanner<granit_parser::input::str::StrInput>>::fetch_more_tokens
 0.1%   0.2%  28.0KiB        aws_lc_sys aws_lc_0_42_0_edwards25519_scalarmuldouble_alt
 0.1%   0.2%  27.5KiB        aws_lc_sys aws_lc_0_42_0_edwards25519_scalarmuldouble
 0.1%   0.2%  27.2KiB              prek <<prek::config::RemoteHook as serde_core::de::Deserialize>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::<<serde_saphyr::de::deserializer::YamlDeserializer as serde_core::de::Deserializer>::deserialize_map::MA>
 0.1%   0.2%  26.2KiB              prek prek::cli::try_repo::try_repo::{closure#0}
 0.1%   0.2%  24.9KiB               xml <xml::reader::parser::PullParser>::dispatch_token
 0.1%   0.2%  24.6KiB              prek <prek::languages::node::installer::NodeInstaller>::install::{closure#0}
 0.1%   0.2%  24.3KiB              prek <<prek::config::LocalHook as serde_core::de::Deserialize>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::<<serde_saphyr::de::deserializer::YamlDeserializer as serde_core::de::Deserializer>::deserialize_map::MA>
 0.1%   0.2%  24.0KiB     granit_parser <granit_parser::scanner::Scanner<serde_saphyr::de::buffered_input::ReaderInput>>::fetch_more_tokens
 0.1%   0.2%  23.7KiB              prek <<prek::config::UpdateOptions as serde_core::de::Deserialize>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::<<serde_saphyr::de::deserializer::YamlDeserializer as serde_core::de::Deserializer>::deserialize_map::MA>
 0.1%   0.2%  22.8KiB              prek prek::archive::unzip::<fs_err::tokio::file::File, &std::path::PathBuf>::{closure#0}
 0.1%   0.2%  22.3KiB         [Unknown] Lp384_montjscalarmul_alt_p384_montjadd
42.3%  88.4%  12.4MiB                   And 25902 smaller methods. Use -n N to show more.
47.9% 100.0%  14.0MiB                   .text section size, the file size is 29.2MiB

Base Branch Results

 File  .text     Size             Crate Name
 1.1%   2.3% 332.0KiB        aws_lc_sys aws_lc_0_42_0_aes_gcm_encrypt_avx512
 1.1%   2.3% 332.0KiB        aws_lc_sys aws_lc_0_42_0_aes_gcm_decrypt_avx512
 0.2%   0.5%  65.4KiB              prek <prek::cli::Command as clap_builder::derive::Subcommand>::augment_subcommands
 0.2%   0.4%  55.1KiB              prek <<prek::config::Config as serde_core::de::Deserialize>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::<<serde_saphyr::de::deserializer::YamlDeserializer as serde_core::de::Deserializer>::deserialize_map::MA>
 0.2%   0.3%  49.4KiB annotate_snippets annotate_snippets::renderer::render::render
 0.2%   0.3%  46.8KiB              prek prek::run::{closure#0}
 0.1%   0.3%  38.4KiB              prek prek::cli::run::run::run::{closure#0}
 0.1%   0.2%  34.3KiB              prek <prek::cli::RunOptions as clap_builder::derive::Args>::augment_args
 0.1%   0.2%  30.1KiB     granit_parser <granit_parser::scanner::Scanner<granit_parser::input::str::StrInput>>::fetch_more_tokens
 0.1%   0.2%  28.0KiB        aws_lc_sys aws_lc_0_42_0_edwards25519_scalarmuldouble_alt
 0.1%   0.2%  27.5KiB        aws_lc_sys aws_lc_0_42_0_edwards25519_scalarmuldouble
 0.1%   0.2%  27.2KiB              prek <<prek::config::RemoteHook as serde_core::de::Deserialize>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::<<serde_saphyr::de::deserializer::YamlDeserializer as serde_core::de::Deserializer>::deserialize_map::MA>
 0.1%   0.2%  25.7KiB              prek prek::cli::try_repo::try_repo::{closure#0}
 0.1%   0.2%  24.9KiB               xml <xml::reader::parser::PullParser>::dispatch_token
 0.1%   0.2%  24.6KiB              prek <prek::languages::node::installer::NodeInstaller>::install::{closure#0}
 0.1%   0.2%  24.3KiB              prek <<prek::config::LocalHook as serde_core::de::Deserialize>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::<<serde_saphyr::de::deserializer::YamlDeserializer as serde_core::de::Deserializer>::deserialize_map::MA>
 0.1%   0.2%  24.0KiB     granit_parser <granit_parser::scanner::Scanner<serde_saphyr::de::buffered_input::ReaderInput>>::fetch_more_tokens
 0.1%   0.2%  23.7KiB              prek <<prek::config::UpdateOptions as serde_core::de::Deserialize>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::<<serde_saphyr::de::deserializer::YamlDeserializer as serde_core::de::Deserializer>::deserialize_map::MA>
 0.1%   0.2%  23.0KiB              prek prek::archive::unzip::<fs_err::tokio::file::File, &std::path::PathBuf>::{closure#0}
 0.1%   0.2%  22.3KiB         [Unknown] Lp384_montjscalarmul_alt_p384_montjadd
42.3%  88.3%  12.4MiB                   And 25905 smaller methods. Use -n N to show more.
47.9% 100.0%  14.0MiB                   .text section size, the file size is 29.2MiB

@prek-ci-bot

prek-ci-bot Bot commented Jul 22, 2026

Copy link
Copy Markdown

⚡️ Hyperfine Benchmarks

Summary: 0 regressions, 0 improvements above the 10% threshold.

Environment
  • OS: Linux 6.17.0-1020-azure
  • CPU: 4 cores
  • prek version: prek 0.4.10+29 (559b99f 2026-07-23)
  • Rust version: rustc 1.97.1 (8bab26f4f 2026-07-14)
  • Hyperfine version: hyperfine 1.20.0
CLI Commands

Benchmarking basic commands in the main repo:

prek --version

Command Mean [ms] Min [ms] Max [ms] Relative
prek-base --version 2.2 ± 0.1 2.1 2.4 1.00
prek-head --version 2.3 ± 0.2 2.1 3.2 1.05 ± 0.10

prek list

Command Mean [ms] Min [ms] Max [ms] Relative
prek-base list 9.6 ± 0.2 9.2 10.0 1.00 ± 0.03
prek-head list 9.5 ± 0.2 9.1 10.4 1.00

prek validate-config .pre-commit-config.yaml

Command Mean [ms] Min [ms] Max [ms] Relative
prek-base validate-config .pre-commit-config.yaml 3.3 ± 0.5 3.1 6.7 1.07 ± 0.16
prek-head validate-config .pre-commit-config.yaml 3.1 ± 0.1 3.0 3.3 1.00

prek sample-config

Command Mean [ms] Min [ms] Max [ms] Relative
prek-base sample-config 2.5 ± 0.1 2.4 3.1 1.02 ± 0.05
prek-head sample-config 2.4 ± 0.0 2.4 2.6 1.00
Cold vs Warm Runs

Comparing first run (cold) vs subsequent runs (warm cache):

prek run --all-files (cold - no cache)

Command Mean [ms] Min [ms] Max [ms] Relative
prek-base run --all-files 67.3 ± 1.4 65.2 70.1 1.01 ± 0.02
prek-head run --all-files 66.9 ± 0.9 65.5 68.4 1.00

prek run --all-files (warm - with cache)

Command Mean [ms] Min [ms] Max [ms] Relative
prek-base run --all-files 66.9 ± 1.1 65.7 70.2 1.00
prek-head run --all-files 67.4 ± 2.2 65.1 74.1 1.01 ± 0.04
Full Hook Suite

Running the builtin hook suite on the benchmark workspace:

prek run --all-files (full builtin hook suite)

Command Mean [ms] Min [ms] Max [ms] Relative
prek-base run --all-files 67.4 ± 1.4 64.5 70.8 1.01 ± 0.03
prek-head run --all-files 66.6 ± 1.2 63.9 70.0 1.00
Individual Hook Performance

Benchmarking each hook individually on the test repo:

prek run trailing-whitespace --all-files

Command Mean [ms] Min [ms] Max [ms] Relative
prek-base run trailing-whitespace --all-files 20.7 ± 0.5 19.9 22.2 1.02 ± 0.03
prek-head run trailing-whitespace --all-files 20.2 ± 0.4 19.6 21.3 1.00

prek run end-of-file-fixer --all-files

Command Mean [ms] Min [ms] Max [ms] Relative
prek-base run end-of-file-fixer --all-files 19.8 ± 0.6 19.0 21.2 1.00
prek-head run end-of-file-fixer --all-files 19.9 ± 0.5 19.2 21.3 1.00 ± 0.04

prek run check-json --all-files

Command Mean [ms] Min [ms] Max [ms] Relative
prek-base run check-json --all-files 7.8 ± 0.2 7.5 8.4 1.02 ± 0.05
prek-head run check-json --all-files 7.7 ± 0.3 7.3 8.6 1.00

prek run check-yaml --all-files

Command Mean [ms] Min [ms] Max [ms] Relative
prek-base run check-yaml --all-files 7.6 ± 0.1 7.4 7.9 1.00
prek-head run check-yaml --all-files 7.6 ± 0.3 7.4 8.8 1.00 ± 0.04

prek run check-toml --all-files

Command Mean [ms] Min [ms] Max [ms] Relative
prek-base run check-toml --all-files 7.8 ± 0.3 7.2 8.8 1.02 ± 0.05
prek-head run check-toml --all-files 7.6 ± 0.2 7.2 8.3 1.00

prek run check-xml --all-files

Command Mean [ms] Min [ms] Max [ms] Relative
prek-base run check-xml --all-files 7.5 ± 0.3 7.2 8.4 1.00
prek-head run check-xml --all-files 7.6 ± 0.5 7.1 8.8 1.01 ± 0.07

prek run detect-private-key --all-files

Command Mean [ms] Min [ms] Max [ms] Relative
prek-base run detect-private-key --all-files 11.0 ± 0.7 9.8 13.2 1.00
prek-head run detect-private-key --all-files 11.2 ± 0.6 10.3 13.0 1.01 ± 0.09

prek run fix-byte-order-marker --all-files

Command Mean [ms] Min [ms] Max [ms] Relative
prek-base run fix-byte-order-marker --all-files 18.1 ± 0.9 16.8 20.2 1.01 ± 0.07
prek-head run fix-byte-order-marker --all-files 17.9 ± 0.9 17.0 21.1 1.00
Installation Performance

Benchmarking hook installation (fast path hooks skip Python setup):

prek install-hooks (cold - no cache)

Command Mean [ms] Min [ms] Max [ms] Relative
prek-base install-hooks 4.5 ± 0.0 4.5 4.6 1.00
prek-head install-hooks 4.5 ± 0.1 4.5 4.6 1.00 ± 0.02

prek install-hooks (warm - with cache)

Command Mean [ms] Min [ms] Max [ms] Relative
prek-base install-hooks 4.6 ± 0.1 4.5 4.7 1.01 ± 0.03
prek-head install-hooks 4.5 ± 0.1 4.4 4.6 1.00
File Filtering/Scoping Performance

Testing different file selection modes:

prek run (staged files only)

Command Mean [ms] Min [ms] Max [ms] Relative
prek-base run 36.3 ± 0.5 35.4 37.6 1.00
prek-head run 36.5 ± 0.6 34.8 37.7 1.01 ± 0.02

prek run --files '*.json' (specific file type)

Command Mean [ms] Min [ms] Max [ms] Relative
prek-base run --files '*.json' 7.9 ± 0.1 7.8 8.2 1.00
prek-head run --files '*.json' 8.0 ± 0.1 7.8 8.2 1.00 ± 0.02
Workspace Discovery & Initialization

Benchmarking hook discovery and initialization overhead:

prek run --dry-run --all-files (measures init overhead)

Command Mean [ms] Min [ms] Max [ms] Relative
prek-base run --dry-run --all-files 6.8 ± 0.1 6.7 7.0 1.01 ± 0.03
prek-head run --dry-run --all-files 6.7 ± 0.2 6.6 7.6 1.00
Meta Hooks Performance

Benchmarking meta hooks separately:

prek run check-hooks-apply --all-files

Command Mean [ms] Min [ms] Max [ms] Relative
prek-base run check-hooks-apply --all-files 11.9 ± 0.2 11.6 12.4 1.08 ± 0.07
prek-head run check-hooks-apply --all-files 11.0 ± 0.7 10.5 12.9 1.00

prek run check-useless-excludes --all-files

Command Mean [ms] Min [ms] Max [ms] Relative
prek-base run check-useless-excludes --all-files 10.7 ± 0.1 10.5 10.9 1.00
prek-head run check-useless-excludes --all-files 10.8 ± 0.3 10.6 11.7 1.01 ± 0.03

prek run identity --all-files

Command Mean [ms] Min [ms] Max [ms] Relative
prek-base run identity --all-files 10.0 ± 0.2 9.7 10.7 1.01 ± 0.03
prek-head run identity --all-files 9.9 ± 0.1 9.8 10.1 1.00

@j178
j178 merged commit 928dc5c into master Jul 22, 2026
34 checks passed
@j178
j178 deleted the simd-eol branch July 22, 2026 19:01
@j178 j178 added performance Performance improvements and removed enhancement New feature or request labels Jul 22, 2026
@codecov

codecov Bot commented Jul 22, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.11765% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 93.38%. Comparing base (6c89849) to head (64daab4).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
...ek/src/hooks/pre_commit_hooks/mixed_line_ending.rs 94.11% 1 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #2391      +/-   ##
==========================================
- Coverage   93.39%   93.38%   -0.01%     
==========================================
  Files         129      129              
  Lines       27642    27637       -5     
==========================================
- Hits        25816    25810       -6     
- Misses       1826     1827       +1

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@BrewTestBot BrewTestBot mentioned this pull request Jul 24, 2026
Merged
This was referenced Jul 27, 2026