v3.2.1 release

[jameslamb]

CRAN is asking for a new submission of the R package because it is failing checks on an environment we were not testing before. See #4164 (comment) for the request and #4105 for a description of the failing check.

We have until April 21st to submit a new version, or {lightgbm} will be removed from CRAN.

This PR proposes doing a 3.2.1 release of LightGBM to avoid that situation. No breaking changes have been merged since #3872, so I think this should be ok. Maintainers can see the full set of changes at this draft release: https://github.com/microsoft/LightGBM/releases/tag/untagged-e297143c87782ac26583.

Release checklist:

• Update VERSION.txt number.
• Update version in Appveyor config file.
• Update version in configure file of R-package: /gha run r-configure.
• Run R valgrind checks after all PRs are merged: /gha run r-valgrind.
• Run R Solaris checks after all PRs are merged: /gha run r-solaris.
• Push commit with v* tag to trigger GitHubRelease action at Azure Pipelines.
• Convert automatic GitHub release from Draft to normal one.
• Update stable tag at GitHub.
• Trigger RTD build for new tag.
• Upload release to PyPI.
• Upload release to CRAN.
• Update cran-comments.md when new release is accepted on CRAN.
• Upload release to NuGet.
• Update version and hash in Homebrew formula.

Necessary PRs

This is a small maintenance release and the CRAN issue makes it time-sensitive, so I don't think we should wait for any in-progress features to be merged. However, a few bug fixes do need to be done before this can be released.

• [ci] Restrict breathe version at CI #4168
• [ci] Stop running CUDA tests at CI #4167
• [ci] Add debian-clang-devel CI job for the R package #4164

[jameslamb]

/gha run r-configure

[jameslamb]

@StrikerRUS the automatic configure update worked perfectly, great work! 🎉

1bdf35f

[StrikerRUS]

@jameslamb I believe you should use your admin power and merge all CI-fixing PRs one by one. I don't want to squash them into one messy PR fixing everything. WDYT?

[jameslamb]

@jameslamb I believe you should use your admin power and merge all CI-fixing PRs one by one. I don't want to squash them into one messy PR fixing everything. WDYT?

yes you're definitely right, I'll keep doing that! I just merged #4158 and #4167 , will do #4168 next

[jameslamb]

OK all the blocking CI changes (#4167, #4168, #4158) have now been merged. Not merging them into this branch yet since this release won't be "ready" until #4164, so we might as well not spend the CI cycles until merging that in.

[jameslamb]

/gha run r-valgrind

Workflow R valgrind tests has been triggered! 🚀
https://github.com/microsoft/LightGBM/actions/runs/739877877

Status: success ✔️.

[jameslamb]

/gha run r-solaris

Workflow Solaris CRAN check has been triggered! 🚀
https://github.com/microsoft/LightGBM/actions/runs/739878031

solaris-x86-patched: https://builder.r-hub.io/status/lightgbm_3.2.1.tar.gz-390bafe6704c4081bc07e4c74026a742
solaris-x86-patched-ods: https://builder.r-hub.io/status/lightgbm_3.2.1.tar.gz-27a0c3f2bb0e49a19597c914a01d21fe
Reports also have been sent to LightGBM public e-mail: http://www.yopmail.com/lightgbm_rhub_checks
Status: success ✔️.

[jameslamb]

Last night, I updated this to the latest master and triggered the Solaris and valgrind checks for the R package. Those and all other CI checks look like they succeeded.

I'd like to submit to CRAN, what do you think @StrikerRUS ?

[StrikerRUS]

@jameslamb OK, let's merge then! CRAN package for submission will be attached to the GitHub release after passing CI.

[StrikerRUS]

Release will be delayed (we should re-run Azure Pipelines later) due to unhealthy Homebrew:

==> Downloading https://homebrew.bintray.com/bottles/libomp-11.1.0.mojave.bottle.tar.gz
curl: (22) The requested URL returned error: 403 Forbidden

All AppleClang macOS jobs failed.

[StrikerRUS]

Release will be delayed ...

GitHub release is ready: https://github.com/microsoft/LightGBM/releases/tag/v3.2.1.

[jameslamb]

Great, thanks very much for re-running! I'll get the CRAN package from there and submit it right now. And I'll email Guolin to let him know that he should expect to receive an email from CRAN.

[jameslamb]

Ok I just submitted to CRAN and emailed Guolin.

By the way, I want to be sure to tell you, in case I'm ever unavailable and you do a submission to CRAN. The file name has to be changed to exactly the format lightgbm_{VERSION}.tar.gz before submitting.

mv lightgbm-3.2.1-r-cran.tar.gz lightgbm_3.2.1.tar.gz

From https://cran.r-project.org/web/packages/policies.html#Source-packages

Uploads must be source tarballs created by R CMD build and following the PACKAGE_VERSION.tar.gz naming scheme.

[StrikerRUS]

Can we change artifact name here then? It will allow to drop one manual step of renaming before submission. I believe that that name shouldn't conflict with sdist artifact for PyPI: - vs _.

LightGBM/.vsts-ci.yml

Line 290 in b8a6f80

mv lightgbm_${LGB_VER}.tar.gz $(Build.ArtifactStagingDirectory)/lightgbm-${LGB_VER}-r-cran.tar.gz

[jameslamb]

I'm concerned that it would be confusing for users to see a file just called lightgbm_3.2.1.tar.gz on the release and have to "just know" that that one is the R package

[StrikerRUS]

I think ordinary users should install directly from CRAN and not use Releases page at all. More advanced users can ask us in issues in case of any troubles.

[jameslamb]

I'm personally opposed to having two artifacts whose names are separated by the difference between a - and a _. The first time that the Python package is accidentally uploaded to CRAN instead of the R one, any tiny time savings from not having to rename the file will have been lost.

The CRAN upload process is already manual anyway and happens so infrequently that I don't mind changing the file name manually during that process.

[StrikerRUS]

Homebrew/homebrew-core#75186

[jameslamb]

I've been refreshing the CRAN checks at https://cran.r-project.org/web/checks/check_results_lightgbm.html for 3.2.1 every few hours.

Every check run on version 3.2.1 has passed so far. There are 4 left to be run. Notably, the r-devel-linux-x86_64-debian-clang test (the one that originally caused this new release) has not been run yet.

I don't think we need to worry about the ERROR on Windows for 3.2.0 in that screenshot. Looks like a problem with CRAN's environment.

cannot open file 'D:\temp\RtmpGe6aNA\ltxbff048a61bc9/lgb.load.tex': No space left on device

[guolinke]

Now the clang also pass.

[jameslamb]

excellent!

[StrikerRUS]

Final CRAN results:

[jameslamb]

Great! Thanks to all of you for the help, acting quickly to be sure the package stays on CRAN.

[github-actions]

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.