Making eldev work for Org mode

[yantar92]

Hello,

In Org mode, we are currently exploring ways to add third-party packages for Org mode version distributed via GNU Elpa (see https://list.orgmode.org/[email protected]/ where we discuss adding compat.el dependency). However, our build system is Makefile-based and does not support compiling/testing when package dependencies are not optional.

As part of the discussion, we arrived to a need to automatically download and manage dependecies.

This can be done either using some custom Makefile scripts or using an existing cross-platform solution. eldev is indeed one such existing solution.

Now, I am playing around with eldev to setup the existing Org mode compilation and testing setups.
However, when trying to run eldev compile, I am running into issues related to eldev somehow loading built-in version of Org mode rather than the git version:

> ../eldev compile
ELC      lisp/ob-C.el
ELC      lisp/ob-R.el

In end of data:
ob-R.el:518:19: Warning: the function ‘org-id-uuid’ is not known to be defined.
ob-R.el:279:12: Warning: the function ‘org-require-package’ is not known to be defined.
ELC      lisp/ob-awk.el
ELC      lisp/ob-calc.el
ELC      lisp/ob-clojure.el
...

The above warnings indicate that eldev loads an older Org mode version.

Eldev file is

(setf eldev-project-loading-mode 'byte-compiled)
(setf eldev-project-source-dirs "lisp")

Any ideas?

[doublep]

For me to better understand, does this happen in some simple setup, e.g. check out Org's Git repository, try to compile — error, or is there something else involved? Does it depend on Emacs version (if you can't test easily, just tell me with which version you see this failure)? Does it fail for anything else (tests, most importantly) or do you get this error only when byte-compiling?

[doublep]

Can you test if the recent commit to Eldev fixes the problem?

You can e.g. run eldev --unstable upgrade-self to pull an unreleased version from MELPA, or check out the Git repository and use ELDEV_LOCAL=~/git/eldev eldev ....

[yantar92]

Can you test if the recent commit to Eldev fixes the problem?

Yes, I think. Although I am getting a slightly different problem (also with compilation though):

ELC      lisp/org-loaddefs.el
Cannot open load file: No such file or directory, /home/yantar92/tmp/org-mode/lisp/org-loaddefs.elc

This file has "no-byte-compile: t" and eldev appears to expect that every single .el file can be byte-compiled, producing .elc file. With no-byte-compile, it is not the case.

(if you want to test this yourself, you will need to run make autoloads - Org needs to compute org-version and autolodas file manually, according to git output; I am yet to figure out how to port this logic to eldoc).

[doublep]

Should be fixed now. I will release 1.9.1 soon.

Note that Eldev has built-in support for autoloads (search for it in the documentation). I don't know if you have something special in that make autoloads, but if it is only to emulate how installed package works without actually installing it, you can likely replace that with Eldev's autoloads plugin.

[yantar92]

We emulate (or use, in newer Emacs) loaddefs-generate and we do not use org-autoloads.el. Rather org-loaddefs.el. Can <package name>-autoloads name be customized in eldev? (it looks like it is hardcoded)

In addition, we have to generate org-version.el programatically to define Org version in git repository - see the output of M-x org-version. Do I understand correctly that I will need to define a new builder via eldev-defbuilder for a task of generating a file?

Finally, I am observing eldev compile reaching to mk/org-fixup.el:

ELC      mk/org-fixup.el

In toplevel form:
../mk/org-fixup.el:1:1: Warning: file has no ‘lexical-binding’ directive on its first line
Cannot open load file: No such file or directory, /home/yantar92/tmp/org-mode/mk/org-fixup.elc

This is despite (setf eldev-project-source-dirs "lisp"). I feel that I miss something.

[doublep]

we do not use org-autoloads.el. Rather org-loaddefs.el.

Hm, what is the difference? I may need to update Eldev to support whatever Emacs has invented in the meantime...

Can -autoloads name be customized in eldev? (it looks like it is hardcoded)

It is hardcoded currently. I wanted to use a function from Emacs source code to compute it, but there was nothing suitable in public interface, so I ended up with what's in the source code.

Do I understand correctly that I will need to define a new builder via eldev-defbuilder for a task of generating a file?

That would be the best way. Another option is to write a command to do that explicitly, but formulating it as a builder is better. If you do it like this, you won't need a separate invocation to generate the file. Just define dependencies correctly and Eldev will automatically build (and even rebuild/update) the file whenever needed.

This is despite (setf eldev-project-source-dirs "lisp"). I feel that I miss something.

By default Eldev handles all files everywhere in the project, including subdirectories. With the layout you have you'd be able to load lisp/org.el using (require 'lisp/org), for example. When you add "lisp" as source directory, file instead becomes accessible just as (require 'org), which is what you need. However, the rest of the project is not ignored and files in other subdirectories are still considered to be part of source code. The best (from memory) way to let Eldev skip them would be

(setf eldev-standard-excludes `(:or ,eldev-standard-excludes "./mk/**"))

[doublep]

Yup: 1. git clone git://git.savannah.gnu.org/emacs/org-mode.git ...

Thanks, will try it locally.

and the test files are not named in the way eldev expects

At least this is not a problem, just set `eldev-test-fileset' to whatever is used in the project. Use `eldev-add-loading-roots' if needed (see documentation).

…

On Sat, 16 Mar 2024 at 19:43, Ihor Radchenko ***@***.***> wrote: does this happen in some simple setup Yup: 1. git clone git://git.savannah.gnu.org/emacs/org-mode.git 2. cd org-mode 3. eldev init 4. y y 5. Add (setf eldev-project-source-dirs "lisp") to Eldev file 6. eldev compile 7. Observe a lot of errors complaining about new functions introduced on the latest development Org version not being defined Does it depend on Emacs version Does not look like, other than errors are different when using different Emacs with different built-in version of Org mode. I tried Emacs master and Emacs 28. Does it fail for anything else (tests, most importantly) or do you get this error only when byte-compiling? I am not yet there to run tests - they require special configuration to work (need to load org-test library) and the test files are not named in the way eldev expects. — Reply to this email directly, view it on GitHub <#101 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABYPBIVF3GOMZKLWNBYA6LYYSHF5AVCNFSM6AAAAABEZEGVU2VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DQMJTGU2TI> . You are receiving this because you commented.Message ID: ***@***.***>

[yantar92]

Paul Pogonyshev ***@***.***> writes:

> we do not use org-autoloads.el. Rather org-loaddefs.el. Hm, what is the difference? I may need to update Eldev to support whatever Emacs has invented in the meantime...

package-loaddefs.el (or loaddefs.el) is the default name for autoload definitions. Check out 16.5 Autoload section of Elisp manual and the top commentary of loaddefs-gen.el library. <library name>-loaddefs.el is also used across Emacs source code.

…

-- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92>

[doublep]

I use Emacs 29, in my ~/.emacs.d/elpa I see that every package has PKG-autoloads.el, but there is not a single PKG-loaddefs.el. Has that changed in Emacs 30?

Basically, these files are exactly what Eldev reproduces in its plugin autoloads. The idea is that during development your package behaves as close as possible to one installed in the "standard way". Do you need org-loaddefs.el for some other purpose? In other words, would something break if the file is called differently?

[yantar92]

Paul Pogonyshev ***@***.***> writes:

I use Emacs 29, in my `~/.emacs.d/elpa` I see that every package has `PKG-autoloads.el`, but there is not a single `PKG-loaddefs.el`. Has that changed in Emacs 30?

I was talking about Emacs source code. Unlike external packages, packages shipped with Emacs use pkg-loaddefs.el. I am not 100% sure why there is a difference - we may ask on emacs-devel.

Basically, these files are exactly what Eldev reproduces in its plugin `autoloads`. The idea is that during development your package behaves as close as possible to one installed in the "standard way". Do you need `org-loaddefs.el` for some other purpose? In other words, would something break if the file is called differently?

org-loaddefs is what Org mode uses. Changing it may break some user scripts and thus cannot be done without justification.

…

-- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92>

[doublep]

org-loaddefs is what Org mode uses

I now installed Org from GNU ELPA, overriding the built-in package. In ~/.emacs.d/elpa/org-9.6.21 I see both org-autoloads.el and org-loaddefs.el. The first is apparently created automatically by Emacs package manager on installation, while the second is distributed with the package (I did check https://elpa.gnu.org/packages/org-9.6.21.tar). In this case, the plugin should not be used to create org-loaddefs.el: it is really only meant to emulate what package manager does, but without actually installing packages (as a speedup, Eldev normally operates without creating and installing a package from your project, though you can also request that).

I'd then suggest that you formulate another builder for org-loaddefs.el, as for org-version.el, so that Eldev generates and updates the file automatically as needed, instead of requiring this as a separate step (i.e. that make autoloads). Since you already create the files using Elisp code, it should not be difficult.

[yantar92]

Paul Pogonyshev ***@***.***> writes:

> org-loaddefs is what Org mode uses I now installed Org from GNU ELPA, overriding the built-in package. In `~/.emacs.d/elpa/org-9.6.21` I see _both_ `org-autoloads.el` and `org-loaddefs.el`. The first is apparently created automatically by Emacs package manager on installation, while the second is _distributed_ with the package (I did check `https://elpa.gnu.org/packages/org-9.6.21.tar`).

This is funny. Let me ask Emacs devs. ***@***.***/T/#u

…

-- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92>

[yantar92]

I have managed to write the "org-loaddefs" builder and eldev compile is finishing cleanly now.

However, it looks like it is going to be quite an effort to port the existing Makefile-based Org mode build system to use Eldev.
I think that the bare minimum I actually need for my purposes is an ability to (1) download and manage external dependencies; (2) run Emacs with these downloaded external dependencies available and sandboxes.

A short testing showed that Package-Requires: in org.el is quite sufficient for eldev to catch what needs to be downloaded.

However, to actually include eldev as a part of Org build system, it would be ideal if we do not add a requirement for users to install eldev manually. I'd like to have some kind of automatic downloading and setup. But not from github.

Would it be something feasible to ask distributing eldev via trusted service - GNU ELPA, so that we can automatically download eldev from there rather than relying upon sources GNU Emacs has no control of?

[doublep]

(2) run Emacs with these downloaded external dependencies available and

sandboxes. That should be achievable with simple $ eldev emacs as Eldev is already able to build the project. (Also many ways to customize what exactly you get, see documentation.)

Would it be something feasible to ask distributing eldev via trusted

service - GNU ELPA, so that we can automatically download eldev from there rather than relying upon sources GNU Emacs has no control of? Most likely no. I don't want to sign legal papers, go through the review process and listen to all the helpful comments how I should reimplement this and that. In general, I prefer to keep communication with emacs-devel as close to zero as possible, I'm fed up with it. You can maybe set up a mirror that delivers Eldev script (replacing the URL to retrieve the package from MELPA to whatever you want) + the package itself, but that's likely too much hassle. And also won't make original disappear, leading to confusion for users as to which copy to use. Or you can look into alternatives, e.g. Eask (though AFAIK it's also only on GitHub.) Paul

…

On Mon, 25 Mar 2024 at 11:16, Ihor Radchenko ***@***.***> wrote: I have managed to write the "org-loaddefs" builder and eldev compile is finishing cleanly now. However, it looks like it is going to be quite an effort to port the existing Makefile-based Org mode build system to use Eldev. I think that the bare minimum I actually need for my purposes is an ability to (1) download and manage external dependencies; (2) run Emacs with these downloaded external dependencies available and sandboxes. A short testing showed that Package-Requires: in org.el is quite sufficient for eldev to catch what needs to be downloaded. However, to actually include eldev as a part of Org build system, it would be ideal if we do not add a requirement for users to install eldev manually. I'd like to have some kind of automatic downloading and setup. But not from github. Would it be something feasible to ask distributing eldev via trusted service - GNU ELPA, so that we can automatically download eldev from there rather than relying upon sources GNU Emacs has no control of? — Reply to this email directly, view it on GitHub <#101 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABYPBLAJ3LBKWOIX2LO3ITYZ72Q5AVCNFSM6AAAAABEZEGVU2VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DSMBQGQZDA> . You are receiving this because you commented.Message ID: ***@***.***>

[suhail-singh]

I believe inclusion into NonGNU elpa wouldn't require paperwork. Said inclusion could also be done by someone other than the original author.

@yantar92 would nongnu elpa be sufficient for what you had in mind?

[yantar92]

suhail-singh ***@***.***> writes:

I believe inclusion into NonGNU elpa wouldn't require paperwork. Said inclusion could also be done by someone other than the original author. @yantar92 would nongnu elpa be sufficient for what you had in mind?

The idea is to switch to eldev to build and test Org mode. Such a switch means that Org mode development, as a GNU project, will depend on the availability of eldev - something that may be questionable in the long term (5+ years) if its copyright status is not rock-solid. Or we may have to write a fallback package management, which kinda defeats the purpose of using eldev. (Unless we use something really simple, with just the basic features.) Further, we do not want to force users who are using Org mode sources to interact with non-libre services. Some users frown upon a need to download staff from untrusted services like GitHub. This last point is quite important and having a non-GNU package is a must-have. Still, I need to consult Emacs devs to see if using non-GNU package is acceptable for a project which is a part of Emacs.

…

-- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92>

[suhail-singh]

I see. Thank you for elaborating on the material considerations.