Support local compilation of dynamic module and grammar binaries

[ubolonton]

Most importantly, this would allow M1 macOS users to use the package, without waiting for us to set up all the CI infrastructure (#88). It's also necessary for users who want increased control over their setup.

• Support local compilation in tsc-dyn-get.el: Compilation Utilities #190. (Note that the Rust source is included in the package.)
• Support better local compilation in tree-sitter-langs-build.el. It already has functions to compile the grammars, but they assume a checked-out git repository, instead of downloading the source: Automatically build tree-sitter-langs from source instead of downloading binaries on arm64 (m1) machines tree-sitter-langs#36.

[nowislewis]

How is everything going? And thanks for your work

[ubolonton]

Local compilation for tsc is almost done in the compilation-utilities branch.

[ckruse]

Good to read! Wasn't able to get it working, yet.

[ubolonton]

Good to read! Wasn't able to get it working, yet.

What issues did you encounter? Local compilation should be more or less working. I'm just improving error reporting and fixing CI.

[ckruse]

Sorry that it took me so long to answer, I had to reproduce everything.

The problem I encounter is that I am not able to compile the grammars (they only exist as x86 binaries: bin/tsx.dylib: Mach-O 64-bit dynamically linked shared library x86_64). I fail to get the tree-parser binary at the correct version and the correct architecture. Installing it via NPM installs the x86 binary.

[ckruse]

Ok, after compiling tree-sitter myself and installing the CLI tool on my machine via cargo install --path . I am now able to generate the languages via ./script/compile all.

[Tass0sm]

I've used the work you've done for local compilation of the dynamic module to package emacs tree-sitter for GNU Guix. You can find that linked here if anyone is interested. I'm hoping to contribute this package to Guix (or NonGuix) upstream at some point.

[aaronjensen]

After I've compiled a language, where is the dylib? Alternatively, are there step-by-step instructions for doing a full build on this branch? I'm trying to get it working on a mac M1. So far I've got tsc-dyn.dylib compiled for arm, but I can't figure out how to get language bindings.

Thanks!

[ubolonton]

After I've compiled a language, where is the dylib? Alternatively, are there step-by-step instructions for doing a full build on this branch?

This branch is only for the core package itself. I haven't started on the grammar bundle yet.

I'm trying to get it working on a mac M1. So far I've got tsc-dyn.dylib compiled for arm, but I can't figure out how to get language bindings.

Maybe this would work https://github.com/emacs-tree-sitter/tree-sitter-langs#building-grammars-from-source. I don't have an M1 to check, though.

[aaronjensen]

I’ve run those scripts but I don’t know what it is doing. I don’t end up with a dylib so I can’t tell what it’s actually producing.

[ubolonton]

The dylib should be under the bin directory if the build succeeds.

[aaronjensen]

I must be missing something. The only thing I can see that actually builds binaries is here: https://github.com/emacs-tree-sitter/tree-sitter-langs/blob/master/tree-sitter-langs-build.el#L293-L303

And that's disabled on macOS, and only works for cc files. Typescript has a scanner.c, for example. The build completes successfully on my machine and emits nothing to bin.

[aaronjensen]

I was able to build them with:

(if (and ;; (memq system-type '(gnu/linux))
                   (file-exists-p "src/scanner.c"))
              ;; Modified from
              ;; https://github.com/tree-sitter/tree-sitter/blob/v0.20.0/cli/loader/src/lib.rs#L351
              (tree-sitter-langs--call
               "g++" "-shared" "-fPIC" "-fno-exceptions" "-g" "-O2"
               "-static-libstdc++"
               "-I" "src"
               "-xc" "-std=c99" "src/scanner.c"
               "-xc" "src/parser.c"
               "-o" (format "%sbin/%s.so" tree-sitter-langs-grammar-dir lang-symbol))
            (tree-sitter-langs--call "tree-sitter" "test"))```

[theHamsta]

Nice idea to use -xc to convince g++ to compile C. You might also try https://github.com/nvim-treesitter/nvim-treesitter to compile parser binaries for more languages or export the compile commands for additional languages. We could add a pipe line for arm64 Mac.

[aaronjensen]

It wasn't my idea, I was just expanding on what was in https://github.com/tree-sitter/tree-sitter/blob/v0.20.0/cli/loader/src/lib.rs#L351. It looks like the current build assumes c++ and linux. I have no idea how it ever build any libs for typescript...

[ubolonton]

It looks like the current build assumes c++ and linux. I have no idea how it ever build any libs for typescript...

It special-cases C++ on Linux so that stdc++ is linked statically, to ease distribution. It calls the tree-sitter CLI tool otherwise.

The build completes successfully on my machine and emits nothing to bin.

It's probably because you have a new version of the CLI tool, which doesn't allow customizing the output directory. See tree-sitter/tree-sitter#1336.

I think we will eventually remove the reliance on the CLI tool to build the binaries.

[aaronjensen]

It's probably because you have a new version of the CLI tool

Arg, you are right. Thank you. That makes it work better.

[ethan-leba]

On the tree-sitter-langs side of this feature, is it planned to allow installation for abritrary parsers via github (something like a straight.el for ts parsers), or is the focus more on supporting new architectures?

[ubolonton]

On the tree-sitter-langs side of this feature, is it planned to allow installation for abritrary parsers via github (something like a straight.el for ts parsers)

Yes, that's an important goal.

[the-moriarty]

Can you bump tree-sitter?

https://github.com/emacs-tree-sitter/elisp-tree-sitter/pull/206/files

They fixed the download for m1

[ckruse]

@the-moriarty wrong issue? :-)

[LunNova]

Support local compilation in tsc-dyn-get.el: #190. (Note that the Rust source is included in the package.)

This bit doesn't seem to work, or I'm using it wrong. doom config.el:

;; workaround for https://github.com/emacs-tree-sitter/elisp-tree-sitter/issues/242
;; but it doesn't seem to do anything, still get x86_64 binaries
 (use-package tsc
   :init (setq tsc-dyn-get-from '(:compilation)))

Still tries to use pre-built x86_64-linux so on aarch64-linux.