Setup for running Quantum ESPRESSO#
+Aiida#
import subprocess
@@ -543,9 +536,8 @@ Setup for running Quantum ESPRESSO
-Equation of State curve - basic QE#
+
+Equation of State curve - basic QE#
Running an EOS without all the fancy features in the aiida-quantumespresso
plugin.
@@ -564,7 +556,7 @@ Equation of State curve - basic QE
-Importing a structure#
+Importing a structure#
from ase.build import bulk
@@ -576,7 +568,7 @@ Importing a structure
-Relaxing the geometry#
+Relaxing the geometry#
resources = {
@@ -630,7 +622,7 @@ Relaxing the geometry
-Calc function to rescale structures#
+Calc function to rescale structures#
The calcfunction
below takes an input structure and rescales it to different volumes.
@@ -680,6 +672,7 @@ Calc function to rescale structures
EOS: Work function version#
@@ -1032,7 +1025,7 @@ Using the build
title="next page">
Using the build
@@ -1052,18 +1045,16 @@
diff --git a/book/challenges.html b/book/challenges.html
index 53b942e..011d541 100644
--- a/book/challenges.html
+++ b/book/challenges.html
@@ -179,14 +179,9 @@
diff --git a/book/introduction.html b/book/introduction.html
index e0cb17e..8c86adf 100644
--- a/book/introduction.html
+++ b/book/introduction.html
@@ -179,14 +179,9 @@
diff --git a/challenges.html b/challenges.html
index d2d00aa..d2f9d8f 100644
--- a/challenges.html
+++ b/challenges.html
@@ -60,7 +60,7 @@
-
+
@@ -179,14 +179,9 @@
@@ -493,7 +488,7 @@ Universal Interface
diff --git a/genindex.html b/genindex.html
index 0488180..d608a7e 100644
--- a/genindex.html
+++ b/genindex.html
@@ -178,14 +178,9 @@
diff --git a/introduction.html b/introduction.html
index b3931cd..079634f 100644
--- a/introduction.html
+++ b/introduction.html
@@ -179,14 +179,9 @@
diff --git a/jobflow.html b/jobflow.html
index d2e2b64..8fc5b13 100644
--- a/jobflow.html
+++ b/jobflow.html
@@ -8,7 +8,7 @@
- Functions — ADIS2023
+ jobflow — ADIS2023
@@ -60,8 +60,8 @@
-
-
+
+
@@ -179,14 +179,9 @@
@@ -354,7 +349,7 @@
- Functions
+ jobflow
@@ -368,7 +363,9 @@ Functions
-
+
+jobflow#
+
import subprocess
import os
@@ -400,8 +397,6 @@ Functions
-
-Functions#
def generate_structures(structure, strain_lst):
@@ -746,7 +741,7 @@ Functions
Functions
diff --git a/objects.inv b/objects.inv
index 2419227..5a7155d 100644
Binary files a/objects.inv and b/objects.inv differ
diff --git a/pyiron_base.html b/pyiron_base.html
index cc6947a..e8d810f 100644
--- a/pyiron_base.html
+++ b/pyiron_base.html
@@ -8,7 +8,7 @@
-
Functions — ADIS2023
+ pyiron — ADIS2023
@@ -60,7 +60,7 @@
-
+
@@ -178,14 +178,9 @@
@@ -355,7 +350,7 @@
- Functions
+ pyiron
@@ -365,13 +360,12 @@ Contents
@@ -382,61 +376,23 @@ Contents
-
-
-import os
-
-
-
-
+
+pyiron#
+The integrated development environment (IDE) for computational materials science pyiron
accelerates the rapid prototyping and up-scaling of simulation protocols. Internally, it consists of two primary components the pyiron_atomistics
package, which provides the interfaces for atomistic simulations codes and atomistic simulation workflows and the pyiron_base
package, which defines the job management and data storage interface. The latter is independent of the atomistic scale and addresses the general challenge of coupling simulation codes in reproducible workflows. Simulation codes can be integrated in the pyiron_base
package by either using existing python bindings or alternatively by writing the input files, executing the simulation code and parsing the output files. The following explanations focus on the pyiron_base
package as a workflow manager, which constructs simulation workflows by combining job
objects like building blocks.
+
+Installation / Setup#
+The pyiron_base
workflow manager can be installed via the python package index or the conda package manager. While no additional configuration is required to use pyiron_base
on a workstation, the connection to an high performance computing (HPC) cluster requires some additional configuration. The .pyiron
configuration file in the users home directory is used to specify the resource directory, which contains the configuration of the queuing system.
+
+
+Implementation of a new simulation code#
+The pyiron_base
workflow manager provides two interfaces to implement new simulation codes or simulation workflows. For simulation codes which already provide a python interface the wrap_python_function()
function is used to convert any python function into a pyiron job object. In analogy external executables can be wrapped using the wrap_executable()
. Based on these two functions any executable can be wrapped as Job
object. By naming the Job
object the user can easily reload the same calculation at any time. Furthermore, pyiron_base
internally uses the name to generate a directory for each Job
object to simplify locating the input and output of a given calculation for debugging:
-import matplotlib.pyplot as plt
+import os
+import matplotlib.pyplot as plt
import numpy as np
-
-
-
-
-
-
-from ase.build import bulk
from ase.io import write
-
-
-
-
-
-
-from adis_tools.parsers import parse_pw
-
-
-
-
-
-Functions#
-
-
-def generate_structures(structure, strain_lst):
- structure_lst = []
- for strain in strain_lst:
- structure_strain = structure.copy()
- structure_strain.set_cell(
- structure_strain.cell * strain**(1/3),
- scale_atoms=True
- )
- structure_lst.append(structure_strain)
- return structure_lst
-
-
-
-
-
-
-def plot_energy_volume_curve(volume_lst, energy_lst):
- plt.plot(volume_lst, energy_lst)
- plt.xlabel("Volume")
- plt.ylabel("Energy")
- plt.savefig("evcurve.png")
+from adis_tools.parsers import parse_pw
@@ -477,6 +433,23 @@ Functions
+
+def generate_structures(structure, strain_lst):
+ structure_lst = []
+ for strain in strain_lst:
+ structure_strain = structure.copy()
+ structure_strain.set_cell(
+ structure_strain.cell * strain**(1/3),
+ scale_atoms=True
+ )
+ structure_lst.append(structure_strain)
+ return structure_lst
+
+
+
+
def workflow(project, structure, pseudopotentials):
@@ -527,12 +500,12 @@ Functions
-Setup#
+As the quantum espresso calculations are the computationally expensive steps they are combined in a python function to be submitted to dedicated computing resources. In contrast the creation of the atomistic structure and the plotting of the energy volume curve are executed in the users process.
+The remaining simulation protocol, can be summarized in a few lines. The required modules are imported, a Project
object is created which represents a folder on the filesystem, the wrap_python_function()
is used to convert the computationally expensive steps of the workflow into a single Job
object and the resulting energy volume curve is plotted:
-from pyiron_base import Project
+from ase.build import bulk
+from pyiron_base import Project
@@ -540,19 +513,7 @@ Setup#<
pr = Project("test")
-pr.remove_jobs(recursive=True, silently=True)
-
-
-
-
-
-
-
-
-Workflow#
-
-
-
-The job workflowdbcdde11bde789bfe23b268a60c426c2 was saved and received the ID: 1
+The job workflow895ba469e3d888839622dab8177e3746 was saved and received the ID: 1
The job job_qe_minimize was saved and received the ID: 2
-The job generate_structuresffd5c51457c48a1cff1923a2d98eb48c was saved and received the ID: 3
+The job generate_structures81144f1592dde5715ec257eb7f425177 was saved and received the ID: 3
The job job_strain_0 was saved and received the ID: 4
The job job_strain_1 was saved and received the ID: 5
The job job_strain_2 was saved and received the ID: 6
@@ -573,9 +534,18 @@ Workflow
-
-
-Result#
+
+
+def plot_energy_volume_curve(volume_lst, energy_lst):
+ plt.plot(volume_lst, energy_lst)
+ plt.xlabel("Volume")
+ plt.ylabel("Energy")
+ plt.savefig("evcurve.png")
+
+
+
+
+This concludes the first version of the simulation workflow, in the following the submission to HPC resources, the different options for data storage and the publication of the workflow are briefly discussed.
plot_energy_volume_curve(
@@ -586,203 +556,37 @@ Result#
-
+
-
-Summary#
-
-
-pr.job_table()
+
+Submission to an HPC / Check pointing / Error handling#
+While the local installation of the pyiron_base
workflow manager requires no additional configuration, the connection to an HPC system is more evolved. The existing examples provided for specific HPC systems can be converted to jinja2 templates, by defining variables with double curly brackets. A minimalist template could be:
+#!/bin/bash
+#SBATCH --job-name={{job_name}}
+#SBATCH --chdir={{working_directory}}
+#SBATCH --cpus-per-task={{cores}}
+
+{{command}}
+Here the job_name
, the working_directory
and the number of compute cores
can be specified as parameters. In the pyiron_base
workflow manager such a submission script can then be selected based on its name as parameter of the server
object:
+job_workflow.server.queue = "my_queue"
+job_workflow.server.cores = 64
+
-
-
-
-
-
-
-
- id
- status
- chemicalformula
- job
- subjob
- projectpath
- project
- timestart
- timestop
- totalcputime
- computer
- hamilton
- hamversion
- parentid
- masterid
-
-
-
-
- 0
- 1
- finished
- None
- workflowdbcdde11bde789bfe23b268a60c426c2
- /workflowdbcdde11bde789bfe23b268a60c426c2
- None
- /home/jovyan/test/
- 2024-03-27 18:09:12.391144
- NaT
- NaN
- pyiron@jupyter-jan-2djanssen-2dqua-2dsso-5fpyiron-5fbase-2dtqm3r7gv#1
- PythonFunctionContainerJob
- 0.4
- None
- None
-
-
- 1
- 2
- finished
- None
- job_qe_minimize
- /job_qe_minimize
- None
- /home/jovyan/test/
- 2024-03-27 18:09:12.461873
- 2024-03-27 18:09:58.263945
- 45.0
- pyiron@jupyter-jan-2djanssen-2dqua-2dsso-5fpyiron-5fbase-2dtqm3r7gv#1
- ExecutableContainerJob
- 0.4
- None
- None
-
-
- 2
- 3
- finished
- None
- generate_structuresffd5c51457c48a1cff1923a2d98eb48c
- /generate_structuresffd5c51457c48a1cff1923a2d98eb48c
- None
- /home/jovyan/test/
- 2024-03-27 18:09:58.313609
- NaT
- NaN
- pyiron@jupyter-jan-2djanssen-2dqua-2dsso-5fpyiron-5fbase-2dtqm3r7gv#1
- PythonFunctionContainerJob
- 0.4
- None
- None
-
-
- 3
- 4
- finished
- None
- job_strain_0
- /job_strain_0
- None
- /home/jovyan/test/
- 2024-03-27 18:09:58.477003
- 2024-03-27 18:10:07.059195
- 8.0
- pyiron@jupyter-jan-2djanssen-2dqua-2dsso-5fpyiron-5fbase-2dtqm3r7gv#1
- ExecutableContainerJob
- 0.4
- None
- None
-
-
- 4
- 5
- finished
- None
- job_strain_1
- /job_strain_1
- None
- /home/jovyan/test/
- 2024-03-27 18:10:07.429814
- 2024-03-27 18:10:17.057758
- 9.0
- pyiron@jupyter-jan-2djanssen-2dqua-2dsso-5fpyiron-5fbase-2dtqm3r7gv#1
- ExecutableContainerJob
- 0.4
- None
- None
-
-
- 5
- 6
- finished
- None
- job_strain_2
- /job_strain_2
- None
- /home/jovyan/test/
- 2024-03-27 18:10:17.833435
- 2024-03-27 18:10:28.145417
- 10.0
- pyiron@jupyter-jan-2djanssen-2dqua-2dsso-5fpyiron-5fbase-2dtqm3r7gv#1
- ExecutableContainerJob
- 0.4
- None
- None
-
-
- 6
- 7
- finished
- None
- job_strain_3
- /job_strain_3
- None
- /home/jovyan/test/
- 2024-03-27 18:10:28.190237
- 2024-03-27 18:10:40.343077
- 12.0
- pyiron@jupyter-jan-2djanssen-2dqua-2dsso-5fpyiron-5fbase-2dtqm3r7gv#1
- ExecutableContainerJob
- 0.4
- None
- None
-
-
- 7
- 8
- finished
- None
- job_strain_4
- /job_strain_4
- None
- /home/jovyan/test/
- 2024-03-27 18:10:40.400240
- 2024-03-27 18:10:53.349634
- 12.0
- pyiron@jupyter-jan-2djanssen-2dqua-2dsso-5fpyiron-5fbase-2dtqm3r7gv#1
- ExecutableContainerJob
- 0.4
- None
- None
-
-
-
-
-
+These lines are inserted before calling the run()
function. The rest of the simulation protocol remains the same.
+When simulation protocols are up-scaling and iterated over a large number of parameters, certain parameter combinations might lead to poor conversion or even cause simulation code crashes. In the pyiron_base
workflow manager these calculation are marked as aborted
. This gives the user to inspect the calculation and in case the crash was not related to the parameter combination, individual jobs can be removed with the remove_job()
function. Afterwards, the simulation protocol can be executed again. In this case the pyiron_base
workflow manager recognizes the already completed calculation and only re-evaluates the removed broken calculation.
+
+
+Data Storage / Data Sharing#
+In the pyiron_base
workflow manager the input of the calculation as well as the output are stored in the hierachical data format (HDF). In addition, pyiron_base
can use a Structured Query Language (SQL) database, which acts as an index of all the Job
objects and their HDF5 files. This file-based approach allows the user easily to browse through the results and at the same time the compressed storage in HDF5 and the internal hierarchy of the data format, enable the efficient storage of large tensors, like atomistic trajectories.
+
+
+Publication of the workflow#
+The pyiron_base
workflow manager provides a publication template to publish simulation workflows on Github. This template enables both the publication of the workflow as well as the publication of the results generated with a given workflow. For reproduciblity this publication template is based on sharing a conda environment file environment.yml
in combination with the Jupyter notebook containing the simulation protocol and the archived Job
objects. The Jupyter notebook is then rendered as static website with links to enable interactive execution using Jupyterbook and the mybinder service. As the pyiron_base
workflow manager reloads existing calculation from the archive, a user of the interactive mybinder environment does not have to recompute the computationally expensive steps and still has the opportunity to interact with the provided workflow and data.
+