[Python] type checkers broken for pyarrow.compute and other sub-modules since PyArrow 24.0+

[jameslamb]

Describe the bug, including details regarding any error messages, version, and platform.

Consider the following snippet:

docker run \
  --rm \
  -it python:3.13 \
  bash

pip install 'pyarrow>=24.0' 'mypy'

cat > test.py <<EOF
import pyarrow as pa
import pyarrow.compute as pa_compute

def all_ones(arr) -> bool:
    return pa_compute.all(pa_compute.equal(arr, 1))

if __name__ == "__main__":
    X = pa.array([1, 1, 1])
    print(f"all ones?: {all_ones(X)}")
EOF

This is totally plausible pyarrow code and works as expected.

$ python ./test.py
all ones?: True

But mypy complains about it referencing things that don't exist in pyarrow

$ mypy --version
mypy 1.20.2 (compiled: yes)

$ mypy ./test.py
test.py:5: error: Module has no attribute "all"  [attr-defined]
test.py:5: error: Module has no attribute "equal"  [attr-defined]
Found 2 errors in 1 file (checked 1 source file)

Notes

Similar errors are not surfaced by type checkers like mypy on pyarrow<24, so I strongly suspect this is related to something new in the 24.0 release. Possibly the introduction of type-checking infrastructure (a py.typed file and .pyi stubs), around #48618

Maybe the .pyi stub file(s) are incomplete? At https://mypy.readthedocs.io/en/stable/stubs.html#creating-a-stub, I see:

If a directory contains both a .py and a .pyi file for the same module, the .pyi file takes precedence.

Or maybe mypy does not handle well situations where things are added to globals at import time like this?

arrow/python/pyarrow/compute.py

Line 343 in acd8e44

_make_global_functions()

Thanks for your time and consideration.

Component(s)

Python

[AlenkaF]

Thank you for opening an issue!

Looking at the ongoing work in #32609 the sub-issue #49194 will handle this correctly. I am not fully sure how this works currently, but after triaging with Copilot's help it seems to me the workaround that already exists on the pyarrow level doesn't help work on pyarrow's submodules. As a quick fix a similar workaround might also be added for each submodule.

@rok do you think this needs a quick fix or should we wait for #49194?

[raulcd]

Is there a way to use pyarrow-stubs to fix those in the interim? or are pyarrow-stubs outdated at the moment.

[jorisvandenbossche]

Running into similar CI failures in pandas. It might have been a mistake to already include a py.typed file in the pyarrow package, while the stub files are not yet complete?

And note that it are not only the cython submodules having no attributes. For example also seeing "Call to untyped function" errors for eg parquet and feather functions.

the workaround that already exists on the pyarrow level doesn't help work on pyarrow's submodules.

That file also points to a TODO #48970 to remove that before the release (no idea if that is related)

[jmahlik]

I ended up working around it with this in the pyproject.toml.

[[tool.mypy.overrides]]
module = [
    "pyarrow",
    "pyarrow.*",
]
follow_imports = "skip"

Not the best solution but it allows upgrading while keeping mypy happy until the stubs are fully defined.