Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bug and false-positive: pylint blows up with implicit namespaces submodules with relative imports in src-layout in parallel mode #10147

Open
webknjaz opened this issue Dec 22, 2024 · 2 comments
Open

bug and false-positive: pylint blows up with implicit namespaces submodules with relative imports in src-layout in parallel mode #10147

webknjaz opened this issue Dec 22, 2024 · 2 comments
Labels
Needs triage 📥 Just created, needs acknowledgment, triage, and proper labelling

Comments

@webknjaz
Copy link
Contributor

webknjaz commented Dec 22, 2024 •
edited
Loading

Bug description

$sbj. It either tracebacks or produces false-positives depending on the settings + CLI args. I faced this when replacing a try-except ImportError with just an import in https://github.com/sphinx-contrib/sphinxcontrib-towncrier.git that has both an implicit namespace and an src layout.

When I tried adding source-roots, it was giving me more import-related errors which confused me and lead down the wrong rabbit hole. At some point, I realized that source-roots does work (one of the complications was the lack of a correct usage example in the docs!), and proceeded bisecting other aspects of the project.

Eventually, I ended up with a reproducer with no deps or config. It turned out that when --jobs is anything other than 1, pylint tracebacks somewhere in concurrent.futures. And it seems like depending on what path is passed via CLI, it may either dump that trace or show some import rule violation that is untrue.

See the repro with the console logs below.

Configuration

N/A

Command used

pylint --verbose --disable=missing-function-docstring,missing-module-docstring,unused-import --output-format=colorized --source-roots=src/ --recursive=y --jobs=2 src/ns/concurrent/

pylint --verbose --disable=missing-function-docstring,missing-module-docstring,unused-import --output-format=colorized --source-roots=src/ --recursive=y --jobs=2 src/ns/concurrent/relative_module.py

pylint --verbose --disable=missing-function-docstring,missing-module-docstring,unused-import --output-format=colorized --source-roots=src/ --recursive=y --jobs=2 src/ns/concurrent/__init__.py

pylint --verbose --disable=missing-function-docstring,missing-module-docstring,unused-import --output-format=colorized --source-roots=src/ --recursive=y --jobs=2 src/ns/concurrent/another_relative_module.py

pylint --verbose --disable=missing-function-docstring,missing-module-docstring,unused-import --output-format=colorized --source-roots=src/ --recursive=y --jobs=2 src/

Pylint output

$ ls -RhalF
.:
total 44K
drwxr-xr-x  3 wk wk  103 Dec 22 05:20  ./
drwxr-xr-x 63 wk wk 8.0K Dec 22 04:49  ../
drwxr-xr-x  3 wk wk   16 Dec 22 04:49  src/

./src:
total 0
drwxr-xr-x 3 wk wk  16 Dec 22 04:49 ./
drwxr-xr-x 3 wk wk 103 Dec 22 05:20 ../
drwxr-xr-x 3 wk wk  24 Dec 22 05:08 ns/

./src/ns:
total 0
drwxr-xr-x 3 wk wk 24 Dec 22 05:08 ./
drwxr-xr-x 3 wk wk 16 Dec 22 04:49 ../
drwxr-xr-x 2 wk wk 85 Dec 22 04:55 concurrent/

./src/ns/concurrent:
total 12K
drwxr-xr-x 2 wk wk 85 Dec 22 04:55 ./
drwxr-xr-x 3 wk wk 24 Dec 22 05:08 ../
-rw-r--r-- 1 wk wk 15 Dec 22 04:54 another_relative_module.py
-rw-r--r-- 1 wk wk 74 Dec 22 04:55 __init__.py
-rw-r--r-- 1 wk wk 41 Dec 22 04:55 relative_module.py

$ cat src/ns/concurrent/__init__.py
from concurrent.futures import Executor

from .relative_module import fun

$ cat src/ns/concurrent/relative_module.py 
from .another_relative_module import fun

$ cat src/ns/concurrent/another_relative_module.py 
def fun(): ...

$ python3.12 -Im venv some-venv-py312

$ some-venv-py312/bin/python -Im pip install pylint
Collecting pylint
  Using cached pylint-3.3.2-py3-none-any.whl.metadata (12 kB)
Collecting dill>=0.3.6 (from pylint)
  Using cached dill-0.3.9-py3-none-any.whl.metadata (10 kB)
Collecting platformdirs>=2.2.0 (from pylint)
  Using cached platformdirs-4.3.6-py3-none-any.whl.metadata (11 kB)
Collecting astroid<=3.4.0-dev0,>=3.3.5 (from pylint)
  Using cached astroid-3.3.7-py3-none-any.whl.metadata (4.5 kB)
Collecting isort!=5.13.0,<6,>=4.2.5 (from pylint)
  Using cached isort-5.13.2-py3-none-any.whl.metadata (12 kB)
Collecting mccabe<0.8,>=0.6 (from pylint)
  Using cached mccabe-0.7.0-py2.py3-none-any.whl.metadata (5.0 kB)
Collecting tomlkit>=0.10.1 (from pylint)
  Using cached tomlkit-0.13.2-py3-none-any.whl.metadata (2.7 kB)
Using cached pylint-3.3.2-py3-none-any.whl (521 kB)
Using cached astroid-3.3.7-py3-none-any.whl (275 kB)
Using cached dill-0.3.9-py3-none-any.whl (119 kB)
Using cached isort-5.13.2-py3-none-any.whl (92 kB)
Using cached mccabe-0.7.0-py2.py3-none-any.whl (7.3 kB)
Using cached platformdirs-4.3.6-py3-none-any.whl (18 kB)
Using cached tomlkit-0.13.2-py3-none-any.whl (37 kB)
Installing collected packages: tomlkit, platformdirs, mccabe, isort, dill, astroid, pylint
Successfully installed astroid-3.3.7 dill-0.3.9 isort-5.13.2 mccabe-0.7.0 platformdirs-4.3.6 pylint-3.3.2 tomlkit-0.13.2

$ some-venv-py312/bin/python -Im pylint --verbose --disable=missing-function-docstring,missing-module-docstring,unused-import --output-format=colorized --source-roots=src/ --recursive=y --jobs=1 src/
No config file found, using default configuration

----------------------------------------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 8.00/10, +2.00)
Checked 3 files, skipped 3 files


$ some-venv-py312/bin/python -Im pylint --verbose --disable=missing-function-docstring,missing-module-docstring,unused-import --output-format=colorized --source-roots=src/ --recursive=y --jobs=0 src/
No config file found, using default configuration
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/__main__.py", line 10, in <module>
    pylint.run_pylint()
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/__init__.py", line 34, in run_pylint
    PylintRun(argv or sys.argv[1:])
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/run.py", line 215, in __init__
    linter.check(args)
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/pylinter.py", line 679, in check
    check_parallel(
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/parallel.py", line 162, in check_parallel
    ) in executor.map(_worker_check_single_file, files):
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/process.py", line 851, in map
    results = super().map(partial(_process_chunk, fn),
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/_base.py", line 608, in map
    fs = [self.submit(fn, *args) for args in zip(*iterables)]
                                             ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/process.py", line 198, in _get_chunks
    chunk = tuple(itertools.islice(it, chunksize))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/pylinter.py", line 874, in _iterate_file_descrs
    for descr in self._expand_files(files_or_modules).values():
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/pylinter.py", line 883, in _expand_files
    result, errors = expand_modules(
                     ^^^^^^^^^^^^^^^
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/expand_modules.py", line 158, in expand_modules
    modpath = _modpath_from_file(
              ^^^^^^^^^^^^^^^^^^^
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/expand_modules.py", line 22, in _modpath_from_file
    return modutils.modpath_from_file_with_callback(  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/astroid/modutils.py", line 294, in modpath_from_file_with_callback
    raise ImportError(
ImportError: Unable to find module for src/ns/concurrent/another_relative_module.py in /usr/lib/python312.zip, 
/usr/lib/python3.12, 
/usr/lib/python3.12/lib-dynload, 
~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages

$ some-venv-py312/bin/python -Im pylint --verbose --disable=missing-function-docstring,missing-module-docstring,unused-import --output-format=colorized --source-roots=src/ --recursive=y --jobs=2 src/
No config file found, using default configuration
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/__main__.py", line 10, in <module>
    pylint.run_pylint()
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/__init__.py", line 34, in run_pylint
    PylintRun(argv or sys.argv[1:])
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/run.py", line 215, in __init__
    linter.check(args)
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/pylinter.py", line 679, in check
    check_parallel(
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/parallel.py", line 162, in check_parallel
    ) in executor.map(_worker_check_single_file, files):
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/process.py", line 851, in map
    results = super().map(partial(_process_chunk, fn),
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/_base.py", line 608, in map
    fs = [self.submit(fn, *args) for args in zip(*iterables)]
                                             ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/process.py", line 198, in _get_chunks
    chunk = tuple(itertools.islice(it, chunksize))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/pylinter.py", line 874, in _iterate_file_descrs
    for descr in self._expand_files(files_or_modules).values():
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/pylinter.py", line 883, in _expand_files
    result, errors = expand_modules(
                     ^^^^^^^^^^^^^^^
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/expand_modules.py", line 158, in expand_modules
    modpath = _modpath_from_file(
              ^^^^^^^^^^^^^^^^^^^
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/expand_modules.py", line 22, in _modpath_from_file
    return modutils.modpath_from_file_with_callback(  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/astroid/modutils.py", line 294, in modpath_from_file_with_callback
    raise ImportError(
ImportError: Unable to find module for src/ns/concurrent/another_relative_module.py in /usr/lib/python312.zip, 
/usr/lib/python3.12, 
/usr/lib/python3.12/lib-dynload, 
~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages

$ some-venv-py312/bin/python -Im pylint --verbose --disable=missing-function-docstring,missing-module-docstring,unused-import --output-format=colorized --source-roots=src/ --recursive=y --jobs=2 src/ns/concurrent 
No config file found, using default configuration
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/__main__.py", line 10, in <module>
    pylint.run_pylint()
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/__init__.py", line 34, in run_pylint
    PylintRun(argv or sys.argv[1:])
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/run.py", line 215, in __init__
    linter.check(args)
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/pylinter.py", line 679, in check
    check_parallel(
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/parallel.py", line 162, in check_parallel
    ) in executor.map(_worker_check_single_file, files):
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/process.py", line 851, in map
    results = super().map(partial(_process_chunk, fn),
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/_base.py", line 608, in map
    fs = [self.submit(fn, *args) for args in zip(*iterables)]
                                             ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/process.py", line 198, in _get_chunks
    chunk = tuple(itertools.islice(it, chunksize))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/pylinter.py", line 874, in _iterate_file_descrs
    for descr in self._expand_files(files_or_modules).values():
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/pylinter.py", line 883, in _expand_files
    result, errors = expand_modules(
                     ^^^^^^^^^^^^^^^
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/expand_modules.py", line 158, in expand_modules
    modpath = _modpath_from_file(
              ^^^^^^^^^^^^^^^^^^^
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/expand_modules.py", line 22, in _modpath_from_file
    return modutils.modpath_from_file_with_callback(  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/astroid/modutils.py", line 294, in modpath_from_file_with_callback
    raise ImportError(
ImportError: Unable to find module for src/ns/concurrent/another_relative_module.py in /usr/lib/python312.zip, 
/usr/lib/python3.12, 
/usr/lib/python3.12/lib-dynload, 
~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages

$ some-venv-py312/bin/python -Im pylint --verbose --disable=missing-function-docstring,missing-module-docstring,unused-import --output-format=colorized --source-roots=src/ --recursive=y --jobs=2 src/ns/concurrent/another_relative_module.py 
No config file found, using default configuration

-----------------------------------------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)
Checked 1 files, skipped 1 files


$ some-venv-py312/bin/python -Im pylint --verbose --disable=missing-function-docstring,missing-module-docstring,unused-import --output-format=colorized --source-roots=src/ --recursive=y --jobs=2 src/ns/concurrent/__init__.py 
No config file found, using default configuration
************* Module __init__
src/ns/concurrent/__init__.py:3:0: E0401: Unable to import '__init__.relative_module' (import-error)

---------------------------------------------------------------------------------------------------
Your code has been rated at 0.00/10 (previous run: 0.00/10, +0.00)
Checked 1 files, skipped 1 files


$ some-venv-py312/bin/python -Im pylint --verbose --disable=missing-function-docstring,missing-module-docstring,unused-import --output-format=colorized --source-roots=src/ --recursive=y --jobs=2 src/ns/concurrent/relative_module.py 
No config file found, using default configuration
************* Module relative_module
src/ns/concurrent/relative_module.py:1:0: E0402: Attempted relative import beyond top-level package (relative-beyond-top-level)

--------------------------------------------------------------------
Your code has been rated at 0.00/10
Checked 1 files, skipped 1 files


$ some-venv-py312/bin/python -Im pylint --verbose --disable=missing-function-docstring,missing-module-docstring,unused-import --output-format=colorized --source-roots=src/ --recursive=y --jobs=2 src/ns/concurrent/                  
No config file found, using default configuration
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/__main__.py", line 10, in <module>
    pylint.run_pylint()
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/__init__.py", line 34, in run_pylint
    PylintRun(argv or sys.argv[1:])
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/run.py", line 215, in __init__
    linter.check(args)
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/pylinter.py", line 679, in check
    check_parallel(
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/parallel.py", line 162, in check_parallel
    ) in executor.map(_worker_check_single_file, files):
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/process.py", line 851, in map
    results = super().map(partial(_process_chunk, fn),
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/_base.py", line 608, in map
    fs = [self.submit(fn, *args) for args in zip(*iterables)]
                                             ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/process.py", line 198, in _get_chunks
    chunk = tuple(itertools.islice(it, chunksize))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/pylinter.py", line 874, in _iterate_file_descrs
    for descr in self._expand_files(files_or_modules).values():
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/pylinter.py", line 883, in _expand_files
    result, errors = expand_modules(
                     ^^^^^^^^^^^^^^^
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/expand_modules.py", line 158, in expand_modules
    modpath = _modpath_from_file(
              ^^^^^^^^^^^^^^^^^^^
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/expand_modules.py", line 22, in _modpath_from_file
    return modutils.modpath_from_file_with_callback(  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/astroid/modutils.py", line 294, in modpath_from_file_with_callback
    raise ImportError(
ImportError: Unable to find module for src/ns/concurrent/another_relative_module.py in /usr/lib/python312.zip, 
/usr/lib/python3.12, 
/usr/lib/python3.12/lib-dynload, 
~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages

➜ some-venv-py312/bin/python -Im pip install 'astroid @ git+https://github.com/pylint-dev/astroid.git' 'pylint @ git+https://github.com/pylint-dev/pylint.git'
Collecting astroid@ git+https://github.com/pylint-dev/astroid.git
  Cloning https://github.com/pylint-dev/astroid.git to /tmp/pip-install-p0wd_ffk/astroid_2dbb18c524b0442d8ef658bf73fbf2d1
  Running command git clone --filter=blob:none --quiet https://github.com/pylint-dev/astroid.git /tmp/pip-install-p0wd_ffk/astroid_2dbb18c524b0442d8ef658bf73fbf2d1
  Resolved https://github.com/pylint-dev/astroid.git to commit 30ddb250a4c15aadf5d397fa42a96a2ac108c8d8
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting pylint@ git+https://github.com/pylint-dev/pylint.git
  Cloning https://github.com/pylint-dev/pylint.git to /tmp/pip-install-p0wd_ffk/pylint_77f85aaa0c9c491e85b3679b3d637a5f
  Running command git clone --filter=blob:none --quiet https://github.com/pylint-dev/pylint.git /tmp/pip-install-p0wd_ffk/pylint_77f85aaa0c9c491e85b3679b3d637a5f
  Resolved https://github.com/pylint-dev/pylint.git to commit be1968e6aa59c8de19a778c280b192e53471a96f
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: dill>=0.3.6 in ./some-venv-py312/lib/python3.12/site-packages (from pylint@ git+https://github.com/pylint-dev/pylint.git) (0.3.9)
Requirement already satisfied: platformdirs>=2.2.0 in ./some-venv-py312/lib/python3.12/site-packages (from pylint@ git+https://github.com/pylint-dev/pylint.git) (4.3.6)
Requirement already satisfied: isort!=5.13.0,<6,>=4.2.5 in ./some-venv-py312/lib/python3.12/site-packages (from pylint@ git+https://github.com/pylint-dev/pylint.git) (5.13.2)
Requirement already satisfied: mccabe<0.8,>=0.6 in ./some-venv-py312/lib/python3.12/site-packages (from pylint@ git+https://github.com/pylint-dev/pylint.git) (0.7.0)
Requirement already satisfied: tomlkit>=0.10.1 in ./some-venv-py312/lib/python3.12/site-packages (from pylint@ git+https://github.com/pylint-dev/pylint.git) (0.13.2)
Building wheels for collected packages: astroid, pylint
  Building wheel for astroid (pyproject.toml) ... done
  Created wheel for astroid: filename=astroid-4.0.0.dev0-py3-none-any.whl size=275991 sha256=79a792879efe3abbd977258a2c851d750004182961472a970701cea1dceb983b
  Stored in directory: /tmp/pip-ephem-wheel-cache-cx4l0ij1/wheels/ee/2d/6e/311cb12113446220dc04dc7a7284d966fd348a5fa7f7948766
  Building wheel for pylint (pyproject.toml) ... done
  Created wheel for pylint: filename=pylint-4.0.0.dev0-py3-none-any.whl size=522853 sha256=9326efdf2ad5219ddeb659b9269616afc9f3796762e11f9651ba8de512fa88bd
  Stored in directory: /tmp/pip-ephem-wheel-cache-cx4l0ij1/wheels/61/54/d0/bfb7060be86de9deff91d06ca91aa55805e79a8c9e4fa59f6d
Successfully built astroid pylint
Installing collected packages: astroid, pylint
  Attempting uninstall: astroid
    Found existing installation: astroid 3.3.7
    Uninstalling astroid-3.3.7:
      Successfully uninstalled astroid-3.3.7
  Attempting uninstall: pylint
    Found existing installation: pylint 3.3.2
    Uninstalling pylint-3.3.2:
      Successfully uninstalled pylint-3.3.2
Successfully installed astroid-4.0.0.dev0 pylint-4.0.0.dev0

src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout took 8.9s …
➜ some-venv-py312/bin/python -m pylint --verbose --disable=missing-function-docstring,missing-module-docstring,unused-import --output-format=colorized --source-roots=src/ --recursive=y --jobs=2 src/ns/concurrent/
No config file found, using default configuration
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/__main__.py", line 10, in <module>
    pylint.run_pylint()
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/__init__.py", line 34, in run_pylint
    PylintRun(argv or sys.argv[1:])
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/run.py", line 215, in __init__
    linter.check(args)
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/pylinter.py", line 679, in check
    check_parallel(
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/parallel.py", line 162, in check_parallel
    ) in executor.map(_worker_check_single_file, files):
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/process.py", line 851, in map
    results = super().map(partial(_process_chunk, fn),
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/_base.py", line 608, in map
    fs = [self.submit(fn, *args) for args in zip(*iterables)]
                                             ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/process.py", line 198, in _get_chunks
    chunk = tuple(itertools.islice(it, chunksize))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/pylinter.py", line 874, in _iterate_file_descrs
    for descr in self._expand_files(files_or_modules).values():
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/pylinter.py", line 883, in _expand_files
    result, errors = expand_modules(
                     ^^^^^^^^^^^^^^^
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/expand_modules.py", line 158, in expand_modules
    modpath = _modpath_from_file(
              ^^^^^^^^^^^^^^^^^^^
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/pylint/lint/expand_modules.py", line 22, in _modpath_from_file
    return modutils.modpath_from_file_with_callback(  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages/astroid/modutils.py", line 294, in modpath_from_file_with_callback
    raise ImportError(
ImportError: Unable to find module for src/ns/concurrent/another_relative_module.py in ., 
~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/src, 
/usr/lib/python312.zip, 
/usr/lib/python3.12, 
/usr/lib/python3.12/lib-dynload, 
~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages, 
/usr/lib/python312.zip, 
/usr/lib/python3.12, 
/usr/lib/python3.12/lib-dynload, 
~/src/experiments/pylint-repro-relative-import-error-implicit-namespace-src-layout/some-venv-py312/lib/python3.12/site-packages

Expected behavior

No false positives (import-error, no-name-in-module, relative-beyond-top-level seem to be caused by this bug) or tracebacks.

Pylint version

$ some-venv-py312/bin/python -Im pylint --version
pylint 3.3.2
astroid 3.3.7
Python 3.12.8 (main, Dec 11 2024, 12:13:45) [GCC 14.2.1 20241116]

$ some-venv-py312/bin/python -Im pip list
Package      Version
------------ -------
astroid      3.3.7
dill         0.3.9
isort        5.13.2
mccabe       0.7.0
pip          24.3.1
platformdirs 4.3.6
pylint       3.3.2
tomlkit      0.13.2

OS / Environment

Only checked under GNU/Linux. It's Ubuntu that GHA provides and my laptop with Gentoo Linux. Python 3.9, 3.10, 3.11, 3.12. I haven't checked others, but I imagine it'd be the same.

Additional dependencies

None.
@webknjaz webknjaz added the Needs triage 📥 Just created, needs acknowledgment, triage, and proper labelling label Dec 22, 2024
webknjaz added a commit to sphinx-contrib/sphinxcontrib-towncrier that referenced this issue Dec 22, 2024
@webknjaz
🧪 Disable pylint parallelism
ab37df5 
This turned out to be necessary as pylint tracebacks in
`concurrent.futures` and sometimes reports false-positive import rule
violations instead of printing the trace.

Ref: pylint-dev/pylint#10147
@webknjaz webknjaz mentioned this issue Dec 23, 2024
Open
@webknjaz
Copy link
Contributor Author

webknjaz commented Jan 8, 2025

I'm also getting a no-name-in-module w/o parallelism. Haven't yet figured out the pattern...

@b-morgenthaler b-morgenthaler mentioned this issue Jan 14, 2025
Closed
@Julfried
Copy link
Contributor

As discussed here in #10185 I think the problem is likely the way you call pylint. By setting source roots you tell pylint that the namespace starts inside the folder you specified, so you file paths need to start from there:

pylint --verbose --disable=missing-function-docstring,missing-module-docstring,unused-import --output-format=colorized --source-roots=src/ --recursive=y --jobs=2 ns/concurrent/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Needs triage 📥 Just created, needs acknowledgment, triage, and proper labelling
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@webknjaz @Julfried