Skip to content

ZipArchiveOutputStream.addRawArchiveEntry() should check is2PhaseSource - #571

Merged
garydgregory merged 2 commits into
apache:masterfrom
yujincheng08:is2PhaseSource
Sep 11, 2024
Merged

ZipArchiveOutputStream.addRawArchiveEntry() should check is2PhaseSource#571
garydgregory merged 2 commits into
apache:masterfrom
yujincheng08:is2PhaseSource

Conversation

@yujincheng08

@yujincheng08 yujincheng08 commented Sep 10, 2024

Copy link
Copy Markdown
Contributor

This allows users to add manually compressed raw data to the zip file under an unsupported compression method (e.g., XZ) or encrypt method.

Sample usage:

val data = "hello world".toByteArray()
ZipArchiveOutputStream(File("test.zip")).use { zip ->
    val entry = ZipArchiveEntry("test")
    entry.method = ZipMethod.XZ.code
    entry.crc = CRC32().apply { update(data) }.value
    ByteArrayOutputStream().use { os ->
        XZCompressorOutputStream(os).use {
            it.write(data)
        }
        entry.size = data.size.toLong()
        entry.compressedSize = os.size().toLong()
        zip.addRawArchiveEntry(entry, ByteArrayInputStream(os.toByteArray()))
    }
}

Previously these codes would throw an exception about the unsupported compression method.

This allow user to add manually compressed raw data to the zip file
under unsupported compression method (e.g., XZ) or encrypt method

@garydgregory garydgregory left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello @yujincheng08

Thank you for your PR.

This PR is missing tests.

Where is this feature documented?

@yujincheng08

Copy link
Copy Markdown
Contributor Author

Hi, @garydgregory,

Test added.

And this feature is already documented:

/**
* Adds an archive entry with a raw input stream.
* <p>
* If crc, size and compressed size are supplied on the entry, these values will be used as-is. Zip64 status is re-established based on the settings in this
* stream, and the supplied value is ignored.
* </p>
* <p>
* The entry is put and closed immediately.
* </p>
*
* @param entry The archive entry to add
* @param rawStream The raw input stream of a different entry. May be compressed/encrypted.
* @throws IOException If copying fails
*/

According to the document, the rawStream may be compressed/encrypted, and if crc, size and compressed size are supplied on the entry, these values will be used as-is. So there's no reason to check if the compression method or encryption method is supported or not when crc, size and compressed size are supplied (we don't need to calculate these values and we don't need to compress or encrypt the data).

@garydgregory

Copy link
Copy Markdown
Member

@yujincheng08
Please do not use star imports, this causes the build checks to fail.
Run mvn by itself before you push to catch all build errors.
TY!

@yujincheng08

Copy link
Copy Markdown
Contributor Author

@garydgregory Thank you. Fixed.

@garydgregory garydgregory changed the title Don't check requested features for is2PhaseSource ZipArchiveOutputStream.addRawArchiveEntry should check is2PhaseSource Sep 11, 2024
@garydgregory garydgregory changed the title ZipArchiveOutputStream.addRawArchiveEntry should check is2PhaseSource ZipArchiveOutputStream.addRawArchiveEntry() should check is2PhaseSource Sep 11, 2024
@garydgregory
garydgregory merged commit 6bed6b7 into apache:master Sep 11, 2024
@garydgregory

Copy link
Copy Markdown
Member

@yujincheng08

Thank you for your work. Merged!

@yujincheng08
yujincheng08 deleted the is2PhaseSource branch September 12, 2024 05:29
dongjoon-hyun added a commit to apache/spark that referenced this pull request Aug 10, 2025
### What changes were proposed in this pull request?

This PR aims to upgrade `commons-compress` to 1.28.0 which is tested with **Java 24 and 25-ea** officially.

### Why are the changes needed?

To bring the latest improvements and bug fixes.

- https://commons.apache.org/proper/commons-compress/changes.html#a1.28.0 (2025-07-26)
  - [Java 24 and 25-ea](apache/commons-compress@6be46a4)
  - apache/commons-compress#571
  - apache/commons-compress#624
  - apache/commons-compress#649
  - apache/commons-compress#663
  - apache/commons-compress#655

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass the CIs.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #51947 from dongjoon-hyun/SPARK-53222.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants