mbx: Automate Storage management - SysVM template download & seeding (#…

…19) mbx launch now creates systemvmtemplate cache and seeds template to secondary storage if necessary before launching zone Signed-off-by: Rohit Yadav <[email protected]> Co-authored-by: Rohit Yadav <[email protected]>
shapeblue · Oct 7, 2021 · 08d5fff · 08d5fff
1 parent b00a810
commit 08d5fff
Show file tree

Hide file tree

Showing 7 changed files with 164 additions and 102 deletions.
diff --git a/.gitignore b/.gitignore
@@ -1,3 +1,4 @@
 boxes/boxes.list
 boxes/**
 templates/*.qcow2
+templates/cache/*
diff --git a/README.md b/README.md
@@ -17,7 +17,6 @@ Table of Contents
     * [Setup NFS Storage](#setup-nfs-storage)
     * [Setup KVM](#setup-kvm)
     * [Setup mbx](#setup-mbx)
-    * [Create Storage Gold-Masters](#create-storage-gold-masters)
 * [Using mbx](#using-mbx)
 * [CloudStack Development](#cloudstack-development)
     * [Install Development Tools](#install-development-tools)
@@ -40,21 +39,23 @@ nested guest VMs and virtual router are plugged in nested-virtual networks that
 run in a nested KVM host VM.
 
 To learn more about CloudStack as a user, you may read:
+
 https://github.com/shapeblue/hackerbook/blob/main/0-init.md
+
 https://github.com/shapeblue/hackerbook/blob/main/1-user.md
 
 ### Storage
 
 `mbx` requires NFS storage to be setup and exported for the base path
 `/export/testing` for environment-specific primary and secondary storages.
 
-A typical `mbx` environment deployment makes copy of a CloudStack
-version-specific gold-master directory that generally contains two empty primary
-storage directories (`primary1` and `primary2`) and one secondary storage
-directory (`secondary`). The secondary storage directory must be seeded
-with CloudStack version-specific `systemvmtemplates`. The `systemvmtemplate` is
-then used to create system VMs such as the Secondary-Storage VM, Console-Proxy
-VM and Virtual Router in an `mbx` environment.
+A typical `mbx` environment upon deployment creates two primary storage
+directories (`primary1` and `primary2`) and one secondary storage directory
+(`secondary`). The secondary storage directory is seeded with supported
+CloudStack version-specific `systemvmtemplates` if necessary when the env is
+launched using `mbx launch`. The seeded `systemvmtemplate` is then used to
+create system VMs such as the Secondary-Storage VM, Console-Proxy VM and Virtual
+Router in an `mbx` environment.
 
 ### Networking
 
@@ -126,6 +127,8 @@ Tested CloudStack versions:
 - 4.15.2.0
 - 4.16.0.0-SNAPSHOT (main branch)
 
+Supported CloudStack versions: 4.9, 4.11, 4.12, 4.13, 4.14, 4.15 and later
+
 Note: legacy CloudStack releases older than v4.11 that don't have
 `cloudstack-marvin` package will not work.
 
@@ -255,82 +258,29 @@ NAT-enabled virtual network in 172.20.0.0/16.
 
     Your workstation/host IP address is `172.20.0.1`.
 
-### Create Storage Gold-Masters
-
-Note: This is required to be done only once for a specific version of CloudStack and
-is only required for CloudStack **4.15 or below**.
-
-After setting up NFS on the workstation host, you need to create a
-CloudStack-version specific storage golden master directory that contains two
-primary storage folders and a secondary storage folder with the systemvmtemplate for the
-specific version of CloudStack seeded. The storage golden master is used as
-storage source of an mbx environment during `mbx deploy` command execution.
-
-For example, the following is needed only one-time for creating a golden master
-storage directory for CloudStack 4.14 version:
-
-    mkdir -p /export/testing
-    # Create directory layout for a specific ACS version under /export/testing
-    mkdir -p /export/testing/4.14/{primary1,primary2,secondary}
-    # Get the systemvm templates
-    cd /export/testing/4.14
-    wget http://packages.shapeblue.com/systemvmtemplate/4.14/systemvmtemplate-4.14.0-kvm.qcow2.bz2
-    wget http://packages.shapeblue.com/systemvmtemplate/4.14/systemvmtemplate-4.14.0-vmware.ova
-    wget http://packages.shapeblue.com/systemvmtemplate/4.14/systemvmtemplate-4.14.0-xen.vhd.bz2
-    wget http://packages.shapeblue.com/systemvmtemplate/4.14/md5sum.txt
-    # Check the downloaded templates, it should say OK for the three templates
-    md5sum --check md5sum.txt
-    # Seed template in the secondary folder for 4.14
-    /export/monkeybox/files/setup-systemvmtemplate.sh -m /export/testing/4.14/secondary -f systemvmtemplate-4.14.0-kvm.qcow2.bz2 -h kvm
-    /export/monkeybox/files/setup-systemvmtemplate.sh -m /export/testing/4.14/secondary -f systemvmtemplate-4.14.0-vmware.ova -h vmware
-    /export/monkeybox/files/setup-systemvmtemplate.sh -m /export/testing/4.14/secondary -f systemvmtemplate-4.14.0-xen.vhd.bz2 -h xenserver
-    # Cleanup downloaded files
-    rm -fv md5sum.txt systemvmtemplate*
-
-For example, the following is needed only one-time for creating a golden master
-storage directory for CloudStack 4.15 version:
-
-    mkdir -p /export/testing
-    # Create directory layout for a specific ACS version under /export/testing
-    mkdir -p /export/testing/4.15/{primary1,primary2,secondary}
-    # Get the systemvm templates
-    cd /export/testing/4.15
-    wget http://packages.shapeblue.com/systemvmtemplate/4.15/systemvmtemplate-4.15.1-kvm.qcow2.bz2
-    wget http://packages.shapeblue.com/systemvmtemplate/4.15/systemvmtemplate-4.15.1-vmware.ova
-    wget http://packages.shapeblue.com/systemvmtemplate/4.15/systemvmtemplate-4.15.1-xen.vhd.bz2
-    wget http://packages.shapeblue.com/systemvmtemplate/4.15/md5sum.txt
-    # Check the downloaded templates, it should say OK for the three templates
-    md5sum --check md5sum.txt
-    # Seed template in the secondary folder for 4.15
-    /export/monkeybox/files/setup-systemvmtemplate.sh -m /export/testing/4.15/secondary -f systemvmtemplate-4.15.1-kvm.qcow2.bz2 -h kvm
-    /export/monkeybox/files/setup-systemvmtemplate.sh -m /export/testing/4.15/secondary -f systemvmtemplate-4.15.1-vmware.ova -h vmware
-    /export/monkeybox/files/setup-systemvmtemplate.sh -m /export/testing/4.15/secondary -f systemvmtemplate-4.15.1-xen.vhd.bz2 -h xenserver
-    # Cleanup downloaded files
-    rm -fv md5sum.txt systemvmtemplate*
-
 ## Using `mbx`
 
 The `mbx` tool can be used to build CloudStack packages, deploy dev or QA
 environments with KVM, VMware, XenServer and XCP-ng hypervisors, and run
 smoketests on them.
 
     $ mbx
-    MonkeyBox 🐵 v0.1
+    MonkeyBox 🐵 v0.2
     Available commands are:
       init: initialises monkeynet and mbx templates
       package: builds packages from a git repo and sha/tag/branch
       list: lists available environments
-      deploy: deploys QA env with two monkeybox VMs, configures storage, creates marvin cfg file
+      deploy: creates QA env with two monkeybox VMs and creates marvin cfg file
       launch: launches QA env zone using environment's marvin cfg file
       test: start marvin tests
-      dev: deploys dev env with a single monkeybox VM, configures storage, creates marvin cfg file
+      dev: creates dev env with a single monkeybox VM and creates marvin cfg file
       agentscp: updates KVM agent in dev environment using scp and restarts it
       ssh: ssh into a mbx VM
       stop: stop all env VMs
       start: start all env VMs
       destroy: destroy environment
 
-0. On first run, initialise networking and templates, run:
+0. On first run or when upgrading `mbx`, please run:
 
     mbx init
 
@@ -340,17 +290,17 @@ smoketests on them.
 
 2. To deploy an environment, run:
 
-    mbx deploy <name of env, default: mbxe> <mgmt server template, default: mbxt-kvm-centos7> <hypervisor template, default: mbxt-kvm-centos7> <repo, default: http://packages.shapeblue.com/cloudstack/upstream/centos7/4.15> <storage source, default: /export/testing/4.15>
+    mbx deploy <name of env, default: mbxe> <mgmt server template, default: mbxt-kvm-centos7> <hypervisor template, default: mbxt-kvm-centos7> <repo, default: http://packages.shapeblue.com/cloudstack/upstream/centos7/4.15>
 
 Example to deploy test matrix (kvm, vmware, xenserver) environments:
 
     mbx deploy 415-kenv mbxt-kvm-centos7 mbxt-kvm-centos7 # deploys 4.15 + KVM CentOS7 env
     mbx deploy 415-venv mbxt-kvm-centos7 mbxt-vmware67u3  # deploys 4.15 + VMware67u3 env
     mbx deploy 415-xenv mbxt-kvm-centos7 mbxt-xenserver71 # deploys 4.15 + XenServer71 env
 
-More examples with specific repositories and custom storage source: (custom storage source must exist)
+More examples with custom packages repositories:
 
-    mbx deploy 416-snapshot mbxt-kvm-centos7 mbxt-kvm-centos7 http://download.cloudstack.org/testing/nightly/latest/centos7/4.16 /export/testing/4.16.0
+    mbx deploy 416-snapshot mbxt-kvm-centos7 mbxt-kvm-centos7 https://download.cloudstack.org/centos/7/4.15/
 
 3. Once `mbx` environment is deployed, to launch a zone run:
 
@@ -360,7 +310,7 @@ More examples with specific repositories and custom storage source: (custom stor
 
     mbx list # find your environment
     mbx ssh <name of the mbx VM>
-    cd /marvin # here you'll find smoketests.sh to run smoketests
+    cd /marvin
     bash -x smoketests.sh
 
 5. To destroy your mbx environment, run:

diff --git a/files/setup-systemvmtemplate.sh b/files/setup-systemvmtemplate.sh
@@ -36,6 +36,9 @@ failed() {
   fi
   if [[ $returnval -eq 0 ]]; then
     return 0
+  elif [[ -n "(echo $returnmsg | grep "already")" ]]; then
+    echo $returnmsg
+    exit 0
   else
     echo "Installation failed"
     exit $returnval

diff --git a/files/sudoer.mbx b/files/sudoer.mbx
@@ -1,4 +1,4 @@
-Cmnd_Alias CLOUDSTACK = /bin/mkdir, /bin/mount, /bin/umount, /bin/cp, /bin/chmod, /usr/bin/keytool, /bin/keytool, /usr/bin/virsh, /usr/bin/rm -fr /export/testing/*
+Cmnd_Alias CLOUDSTACK = /bin/mkdir, /bin/mount, /bin/umount, /bin/cp, /bin/chmod, /usr/bin/keytool, /bin/keytool, /usr/bin/virsh, /usr/bin/rm -fr /export/testing/*, /export/monkeybox/files/setup-systemvmtemplate.sh
 
 Defaults:%sudo !requiretty
 

diff --git a/files/systemvm-map.json b/files/systemvm-map.json
@@ -0,0 +1,86 @@
+{
+    "4.9": {
+        "kvm": {
+          "url": "https://download.cloudstack.org/systemvm/4.6/dnsmasq/systemvm64template-4.6-kvm.qcow2.bz2",
+          "checksum": "c713c00fcca7e01f089a1962237e1cd8"
+         },
+        "xen": {
+          "url": "https://download.cloudstack.org/systemvm/4.6/dnsmasq/systemvm64template-4.6-xen.vhd.bz2",
+          "checksum": "80c072799ac094fddfb8cba9bd0305e9"
+        },
+        "vmw": {
+          "url": "https://download.cloudstack.org/systemvm/4.6/dnsmasq/systemvm64template-4.6-vmware.ova",
+          "checksum": "d91212b6c96b98bfd1559ef2cc2891e1"
+        }
+    },
+    "4.11": {
+        "kvm": {
+          "url": "https://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.3-kvm.qcow2.bz2",
+          "checksum": "d40bce40b2d5bb4ba73e56d1e95aeae5"
+         },
+        "xen": {
+          "url": "https://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.3-xen.vhd.bz2",
+          "checksum": "1566dcbcc3806755d0012d1619bd4210"
+        },
+        "vmw": {
+          "url": "https://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.3-vmware.ova",
+          "checksum": "d695376be20929d323adfaa5410c093f"
+        }
+    },
+    "4.12": {
+        "kvm": {
+          "url": "https://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.3-kvm.qcow2.bz2",
+          "checksum": "d40bce40b2d5bb4ba73e56d1e95aeae5"
+         },
+        "xen": {
+          "url": "https://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.3-xen.vhd.bz2",
+          "checksum": "1566dcbcc3806755d0012d1619bd4210"
+        },
+        "vmw": {
+          "url": "https://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.3-vmware.ova",
+          "checksum": "d695376be20929d323adfaa5410c093f"
+        }
+    },
+    "4.13": {
+        "kvm": {
+          "url": "https://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.3-kvm.qcow2.bz2",
+          "checksum": "d40bce40b2d5bb4ba73e56d1e95aeae5"
+         },
+        "xen": {
+          "url": "https://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.3-xen.vhd.bz2",
+          "checksum": "1566dcbcc3806755d0012d1619bd4210"
+        },
+        "vmw": {
+          "url": "https://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.3-vmware.ova",
+          "checksum": "d695376be20929d323adfaa5410c093f"
+        }
+    },
+    "4.14": {
+        "kvm": {
+          "url": "https://download.cloudstack.org/systemvm/4.14/systemvmtemplate-4.14.0-kvm.qcow2.bz2",
+          "checksum": "4978e6e6140d167556f201496549a498"
+         },
+        "xen": {
+          "url": "https://download.cloudstack.org/systemvm/4.14/systemvmtemplate-4.14.0-xen.vhd.bz2",
+          "checksum": "2e3078de2ccce760d537e06fd9b4c7c7"
+        },
+        "vmw": {
+          "url": "https://download.cloudstack.org/systemvm/4.14/systemvmtemplate-4.14.0-vmware.ova",
+          "checksum": "33cad72f858aef11c95df486b0f21938"
+        }
+    },
+    "4.15": {
+        "kvm": {
+          "url": "https://download.cloudstack.org/systemvm/4.15/systemvmtemplate-4.15.1-kvm.qcow2.bz2",
+          "checksum": "0e9f9a7d0957c3e0a2088e41b2da2cec"
+         },
+        "xen": {
+          "url": "https://download.cloudstack.org/systemvm/4.15/systemvmtemplate-4.15.1-xen.vhd.bz2",
+          "checksum": "86373992740b1eca8aff8b08ebf3aea5"
+        },
+        "vmw": {
+          "url": "https://download.cloudstack.org/systemvm/4.15/systemvmtemplate-4.15.1-vmware.ova",
+          "checksum": "4006982765846d373eb3719b2fe4d720"
+        }
+    }
+}