v1.1.2

b2luigi v1.1.2

Summary

This patch release includes minor bug fixes and additions to b2luigi mostly targeting batch execution. A new option is also added to build the automatically created output directory structure.

Changes

• Parameter Name in Output Directory Structure

Added the setting use_parameter_name_in_output to False (the default is True). The directory structure will be built as "parameter_value1/parameter_value2" instead of "parameter_key1=parameter_value1/parameter_key2=parameter_value2.

• env_script in create_executable_wrapper Method

Removed the existence check of the environment script (env_script setting) when building the executable wrapper. The main reason for this change is the possibility of a specialised script for remote execution.

• XRootD Support

Removed potential memory leak in XRootDTargets. Per default, suppress listdir printout for XRootDTargets. Additionally, clarified descriptions in the documentation.

• Apptainer Support

Fixed a bug in which the mount list would catch all inputs from all tasks due to it being passed by reference.

• Gbasf2 Support

Added gbasf2_jinja_template_path setting to pass custom jinja templets to be used as steering files for gbasf2 submission. The benefit of this method is that it conserves the way that the main basf2 path is still contained in the create_path method and the custom jinja template only contains additional stuff that cannot be pickled.
The specific use case for this is using and manipulating Python-based basf2 modules that are not pickable.

• LSF Support
  Added retry decorator to LSF job status.

Compatibility

• No changes

Contributors

Belle II collaboration.

We appreciate the continued support and contributions from the community. Thank you! 🙌

Full Changelog: v1.1.1...v1.1.2

v1.1.1

b2luigi v1.1.1

Summary

This patch release fixes the issue of the env_script being sourced in batch execution when run with Apptainer support. Additionally, the XRootD import is now guarded for the respective targets and filesystem imports, and classes are renamed. Finally, AttributeErrors are no longer passed when using task-specific htcondor_settings.

Changes

• Apptainer Support with Batch Execution

  Previously, the env_script was sourced in the executable wrapper submitted by the batch system. This source is not necessary for the execution with Apptainer since the environment during the execution does not need to agree with the one in the container. The env_script source was moved to the case in which no Apptainer is used.

• XRootD Import

  The XRootD import for the XRootDTarget and XRootDSystem were moved into the respective classes to avoid breaking b2luigi when no XRootD is set up. Additionally, the classes were re-named: XrootDTarget -> XRootDTarget, XrootDSystem ->�XRootDSystem .

• AttributeError in htcondor_settings

  Previously, AttributeErrors were passed when updating the HTCondor settings with the task-specific htcondor_settings. This leads to confusing error messages when the missing htcondor_setting object does not cause the AttributeError. Now, the htcondor_settings are updated by get_setting.

Compatibility

• XrootDTarget -> XRootDTarget, XrootDSystem -> XRootDSystem

Contributors

Belle II collaboration.

We appreciate the continued support and contributions from the community. Thank you! 🙌

Full Changelog: v1.1.0...v1.1.1

v1.1.0

b2luigi v1.1.0

Summary

The latest b2luigi updates introduce several enhancements. The new XrootDTarget feature simplifies grid file management by using XRootD-based targets. Apptainer support allows tasks to run in containers both locally and in batch modes. Batch system discovery is now automated when set batch_system="auto". Task-specific propagation of gbasf2 settings is supported for proxy groups, project paths, and proxy lifetimes. The removal of the git_hash parameter from Basf2Task ensures stable task IDs and output paths, with the option to reintroduce it manually if needed. Finally, the new B2LUIGI_SETTINGS_JSON environment variable allows users to specify a custom path for the settings.json file, providing more flexibility in configuration management.

Changes

• XrootDTarget

  b2luigi simplifies working with grid files by offering XRootD-based targets. The XRootD client, pre-installed in basf2 environments on CVMFS, requires a valid VOMS proxy for access. To use this feature, override the _get_output_file_target function in your task and set up a XrootDSystem object with the server address (root://<server_address>). Optionally, define a scratch_dir to handle temporary files using the temporary_path context. Files are first stored in the scratch directory and copied to their final XRootD location after execution, streamlining grid-based file management.

• Apptainer Support

  b2luigi now supports the execution of jobs in an Apptainer container. Tasks can be run both locally and in batch mode for the lsf and the htcondor batch systems. However, for the latter batch system, it is not recommended to use Apptainer images since HTCondor provides its own mechanisms for containerised execution (steerable with the htcondor_settings). To achieve the execution of tasks, an apptainer exec command is built within this class and executed in a subprocess. To steer the execution, one can steer the following settings:

  • apptainer_image: The image to use for the Apptainer container.
  • apptainer_mounts: A list of directories to mount into the Apptainer container.
  • apptainer_mount_defaults: Boolean parameter to mount log_dir and result_dir by default.
  • apptainer_additional_params: Additional parameters to pass to the apptainer exec command.

• Automatic Batch System Discovery

  It is now possible to set batch_system="auto" which will check if bsub or condo_submit commands are available on the system (in that order).

• gbasf2 settings propagation

  The gbasf2 settings for gbasf2_proxy_group, gbasf2_project_lpn_path and gbasf2_proxy_lifetime are now propagated on the task level, introducing the possibility to set these settings task-specific.

• Removal of git_hash

  This release removes the git_hash parameter from the Basf2Task significant parameters. Task IDs and automatic output paths created with the add_to_output will no longer contain the git_hash parameter. This change was implemented to avoid the task ID changing every time the basf2 git hash changes. If the git_hash is needed for the current pipeline's execution or output structure, please add git_hash = b2luigi.Parameter(default=get_basf2_git_hash()) to Basf2Task.

• Settings

  The newly introduced environment variable B2LUIGI_SETTINGS_JSON can be used to steer the path to the settings.json file.

Compatibility

• Workflows including Basf2Task will no longer use the git_hash for building the task graph.
• Workflows including Basf2Task which use the add_to_output method will no longer store the output in directories containing the git_hash. Current workflows might be affected.
• Workflows including Basf2Task will no longer store the log output in directories containing the git_hash.

Contributors

Belle II collaboration.

We appreciate the continued support and contributions from the community. Thank you! 🙌

Full Changelog: v1.0.2...v1.1.1

v1.0.2

b2luigi v1.0.2

Summary

The main goal of this release is the implementation of the --new flag for gb2_ds_get in the Gbasf2Process. With the new changes, the usage of the --new is no longer restricted to the belle accounting group. Additional changes include the beginning of the depreciation processes for very user-specific implementations in b2luigi such as b2luigi/basf_helper/data.py and the Basf2PathTask, and more documentation updates.

Changes

• Gbasf2Process

  The addition of the method search_pattern_in_folder now provides a way to search for the project name in the accounting group folder. This addition is necessary due to Belle II-user-based repositories' different directory structures (accounting group belle) and group-based repositories. From a user's point of view, nothing has to be changed. Downloads from the gabsf2 Grid will always use the --new flag.

• Deprecation

  • b2luigi/basf_helper/data.py will be entirely depricated in the future. The tasks in this file are very tailored to user-specific problems and are no longer maintained. All the default paths given for specific file structures are no longer existent.
  • The Basf2PathTask would assert the basf2 git hash upon running the task. Since this can lead to errors when running tasks from different environments and different basf2 releases, the assertion is now deleted. The git_hash parameter is kept for the Basf2PathTask.

• Documentation

  • Adjusted documentation links to point to https://b2luigi.belle2.org//
  • Instructions on how to utilise the git_hash parameter
  • External package linking
  • Update the developer instructions
  • More links to the luigi documentation

Compatibility

The git_hash parameter in Basf2PathTask is no longer asserted. Tasks specifically depending on this assertion might display unexpected behaviour.

Contributors

Belle II collaboration.

We appreciate the continued support and contributions from the community. Thank you! 🙌

Full Changelog: v1.0.1...v1.0.2

v1.0.1

b2luigi v1.0.1

Summary

The main goal of this release is to restore the functionality of the gbasf2_proxy_info.py tool due to changes in the output format of the grid certificate readout with gabsf2 release 6.0.0. Additionally, parts of the documentation have been improved and caching was added for gb2_job_status.

Changes

• gbasf2_proxy_info.py

  The X509Chain certificate chain value now returns the string value of the certificate content instead of a bytes-typed output. To ensure compatibility with older gabsf2 releases, it is checked if the value type is bytes and then decoded. If the type is not bytes, it is assumed that the value is already in string format.

• Documentation

  • Addition of the Zenodo link
  • Update the documentation link
  • Update of the release procedure
  • Several small cosmetic changes

• gb2_job_status Caching

  b2luigi now caches the state of a gbasf2 Task for 5 minutes before checking again. This reduces the number of checks to Dirac and should improve the performance, especially for large numbers of gbasf2 Tasks run with one scheduler.

Compatibility

No breaking changes from the previous version.

Contributors

Belle II collaboration.

We appreciate the continued support and contributions from the community. Thank you! 🙌

Full Changelog: v1.0.0...v1.0.1

v1.0.0

b2luigi v1.0.0

Summary

This release marks the transition of authorship for b2luigi from the previous maintainers on GitHub to the Belle II collaboration. While there were no major changes to the functionality of b2luigi in this version, the change in authorship signifies a new phase for the project.

Changes

Change of authorship to the Belle II collaboration.

Compatibility

No breaking changes from the previous version.

Contributors

Belle II collaboration.

We appreciate the continued support and contributions from the community. Thank you! 🙌

Full Changelog: v0.10.2...v1.0.0