Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

dmypy crashes from RecursionError for some complex expressions (fastparse) #17706

Open
kcdodd opened this issue Aug 23, 2024 · 1 comment · May be fixed by #17707
Open

dmypy crashes from RecursionError for some complex expressions (fastparse) #17706

kcdodd opened this issue Aug 23, 2024 · 1 comment · May be fixed by #17707
Labels
crash topic-daemon dmypy

Comments

@kcdodd
Copy link

kcdodd commented Aug 23, 2024

Crash Report

I have been unable to use dmypy for a while due to recursion error, but finally took the time to figure out why. This seems to happen when processing a dependency, SymPy source file+line: https://github.com/sympy/sympy/blob/e676df8ded885babe62bd31bec5db8c96b7480e7/sympy/polys/numberfields/resolvent_lookup.py#L248.
It seems that the AST of the expression is just simply too deep to process before hitting recursion limit, estimating its a couple thousand nodes deep.

Honestly, I wouldn't consider this a bug, except that it crashes the daemon and prevents it from processing anything else. The ast module clearly warns "It is possible to crash the Python interpreter with a sufficiently large/complex string due to stack depth limitations in Python’s AST compiler.", so it might even be close to that limit as well.

But if there was a way of simply not processing such large nodes, or recovering, I think that would deserve a "fix".

Traceback

version: 1.12.0+dev.fe15ee69b9225f808f8ed735671b73c31ae1bed8
Daemon crashed!
Traceback (most recent call last):
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/dmypy_server.py", line 236, in serve
    resp = self.run_command(command, data)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/dmypy_server.py", line 285, in run_command
    ret = method(self, **data)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/dmypy_server.py", line 353, in cmd_run
    return self.check(sources, export_types, is_tty, terminal_width)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/dmypy_server.py", line 427, in check
    res = self.initialize_fine_grained(sources, is_tty, terminal_width)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/dmypy_server.py", line 466, in initialize_fine_grained
    result = mypy.build.build(sources=sources, options=self.options, fscache=self.fscache)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/build.py", line 193, in build
    result = _build(
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/build.py", line 268, in _build
    graph = dispatch(sources, manager, stdout)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/build.py", line 2897, in dispatch
    graph = load_graph(sources, manager)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/build.py", line 3083, in load_graph
    st = State(
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/build.py", line 2000, in __init__
    self.parse_file(temporary=temporary)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/build.py", line 2176, in parse_file
    self.tree = manager.parse_file(
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/build.py", line 841, in parse_file
    tree = parse(source, path, id, self.errors, options=options)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/parse.py", line 27, in parse
    tree = mypy.fastparse.parse(source, fnam=fnam, module=module, errors=errors, options=options)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/fastparse.py", line 242, in parse
    ).visit(ast)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/fastparse.py", line 417, in visit
    return visitor(node)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/fastparse.py", line 876, in visit_Module
    body = self.fix_function_overloads(self.translate_stmt_list(mod.body, ismodule=True))
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/fastparse.py", line 491, in translate_stmt_list
    node = self.visit(stmt)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/fastparse.py", line 417, in visit
    return visitor(node)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/fastparse.py", line 889, in visit_FunctionDef
    return self.do_func_def(n)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/fastparse.py", line 1023, in do_func_def
    body = self.as_required_block(n.body, can_strip=True, is_coroutine=is_coroutine)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/fastparse.py", line 606, in as_required_block
    self.translate_stmt_list(stmts, can_strip=can_strip, is_coroutine=is_coroutine)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/fastparse.py", line 491, in translate_stmt_list
    node = self.visit(stmt)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/fastparse.py", line 417, in visit
    return visitor(node)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/fastparse.py", line 1443, in visit_Expr
    value = self.visit(n.value)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/fastparse.py", line 417, in visit
    return visitor(node)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/fastparse.py", line 1497, in visit_BinOp
    e = OpExpr(op, self.visit(n.left), self.visit(n.right))
...
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/fastparse.py", line 417, in visit
    return visitor(node)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/fastparse.py", line 1497, in visit_BinOp
    e = OpExpr(op, self.visit(n.left), self.visit(n.right))
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/fastparse.py", line 417, in visit
    return visitor(node)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/fastparse.py", line 1492, in visit_BinOp
    op = self.from_operator(n.op)
RecursionError: maximum recursion depth exceeded

To Reproduce

Run dymy on https://github.com/sympy/sympy/blob/e676df8ded885babe62bd31bec5db8c96b7480e7/sympy/polys/numberfields/resolvent_lookup.py.

Your Environment

  • Mypy version used: dmypy 1.12.0+dev.fe15ee69b9225f808f8ed735671b73c31ae1bed8
  • Mypy command-line flags: dmypy run -- sympy/polys/numberfields/resolvent_lookup.py
  • Python version used: 3.10
@kcdodd kcdodd added the crash label Aug 23, 2024
@kcdodd kcdodd linked a pull request Aug 23, 2024 that will close this issue
Open
@kcdodd kcdodd closed this as completed Aug 25, 2024
@kcdodd
Copy link
Author

kcdodd commented Aug 25, 2024

As a small update, I was wondering why the non-daemon mypy command did not crash even though it should be going through the same analysis. Apparently mypy.main sets the recursion limit to 2**14 (or 16384) but mypy.dmypy leaves the default limit (1000 in this case).

I have linked a pull request that makes these consistent, which might address this by itself. But it also includes a proposed method of handling such expressions, even though its probably up for debate what the proper handling of it should be.

@kcdodd kcdodd reopened this Aug 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
crash topic-daemon dmypy
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Initial attempt at fixing issue-17706 kcdodd/mypy
2 participants
@kcdodd @hauntsaninja