emacsPackages.tree-sitter-langs: Python grammar uses incompatible ABI version

[mcwitt]

Describe the bug

Trying to use elisp-tree-sitter in Python buffers fails with the following error:

Language’s ABI is too new: 14, (13 . 13), "/nix/store/y5yzbhx8k4c5svc86aq9mrmljqciijbj-emacs-tree-sitter-grammars/langs/bin/python.so"

Steps To Reproduce

Steps to reproduce the behavior:

nix-shell -I nixpkgs=. -p 'emacsWithPackages (ps: with ps; [ tree-sitter tree-sitter-langs ])' --run "emacs -Q"

Execute the elisp commands:

(require 'tree-sitter)
(require 'tree-sitter-langs)

Finally, open a Python buffer and do M-x tree-sitter-hl-mode to reproduce the above error.

Expected behavior

elisp-tree-sitter should work in Python buffers. (This issue does not occur with Haskell or Nix buffers, for example).

Additional context

The cause seems to be that the upstream https://github.com/tree-sitter/tree-sitter-python generated the parser using ABI version 14 (which is now the default, since tree-sitter/tree-sitter@e2fe380), but elisp-tree-sitter does not support this since it uses an older tree-sitter version.

I'm able to work around this with the following nixpkgs overlay:

final: prev:
{
  tree-sitter-grammars = prev.tree-sitter-grammars // {
    tree-sitter-python = prev.tree-sitter-grammars.tree-sitter-python.overrideAttrs (_: {
      nativeBuildInputs = [ final.nodejs final.tree-sitter ];
      configurePhase = ''
        tree-sitter generate --abi 13 src/grammar.json
      '';
    });
  };
}

Related issue: emacs-tree-sitter/elisp-tree-sitter#247

Notify maintainers

@pimeys , @adisbladis

Metadata

• system: "x86_64-linux"
• host os: Linux 5.15.85, NixOS, 22.11 (Raccoon), 22.11.20230103.a9eedea
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.11.1
• channels(matt): ""
• channels(root): "nixos, nixpkgs"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

[clhodapp]

I'll raise that it seems like this comment is relevant to this situation: #230547 (comment)

[xinyifly]

I think this can be solved more or less by installing emacs29 and using built-in treesit now.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tree-sitter-grammars-collide-with-each-other/41805/10