Add backups to workspaces.

[james-annages]

Resolves #4362

This pull request introduces significant updates to the base template for workspaces, primarily focusing on adding backup capabilities and enhancing the cleanup process for Azure Recovery Services Vaults. The key changes include the addition of new parameters and outputs, updates to the porter.yaml file for handling backups, and the creation of new Terraform resources for managing backups.

Backup and Recovery Enhancements:

• templates/workspaces/base/cleanup_vault.sh: Added a new script to handle the cleanup of Azure Recovery Services Vaults, including disabling soft delete and removing protected items.
• templates/workspaces/base/terraform/backup/backup.tf: Introduced new Terraform resources to create and manage Azure Recovery Services Vaults, VM backup policies, and file share backup policies.
• templates/workspaces/base/porter.yaml: Updated to include new parameters and outputs related to backup configuration, and added steps to handle backup vault cleanup during uninstallation. [1] [2] [3] [4] [5] [6] [7] [8]

Parameter and Schema Updates:

• templates/workspaces/base/parameters.json: Added new parameters enable_backup and shared_storage_name to support backup configurations.
• templates/workspaces/base/template_schema.json: Updated the schema to include the enable_backup parameter, allowing backups to be enabled or disabled for the workspace. [1] [2]

Terraform Configuration:

• templates/workspaces/base/terraform/backup/variables.tf: Defined new variables for backup configurations, including location, tre_id, resource_group_name, and shared_storage_name.
• templates/workspaces/base/terraform/backup/outputs.tf: Added new outputs for backup vault and policy names to be used in other parts of the configuration.
• templates/workspaces/base/terraform/storage.tf: Updated the shared storage name variable to be configurable.

Role Assignments:

• templates/workspaces/base/terraform/api-permissions.tf: Added new role assignments for Backup Contributor and Site Recovery Contributor to manage backup and site recovery permissions.

These changes collectively enhance the robustness of the workspace by adding comprehensive backup and recovery functionalities, ensuring that critical data can be protected and restored as needed.

What is being addressed

Added in a boolen for enable_backup that is set in the workspace config window. The system will deploy a recovery vault and the needed policy's.
It passes the names of the polices back out so they can be used by other services (sql, vm, etc).

How is this addressed

• Added terrafrom code to deploy the needed objects. Changed some of the settings for deploying shared vm storage.
• still needs a management ui for recovery, ability to set the retention and backup policy. And possibly give the ability to move the backups to a diffrent storage account for archive.
• Workspace documentation needs to be updated. NOT COMPLETED YET!
• base workspace have been bumped up. This needs to be fixed as it was changed for my testing.

[marrobi]

Thanks for the PR! Couple of initial comments/questions.

[marrobi]

@james-annages is there a reason that you closed the PR?

[james-annages]

@marrobi Sorry for closeing the pull request, i messed up my commits and ended up with a 3 way split, that was not resolving corretly, i have fixed the merge and can repopen.

[marrobi]

@marrobi Sorry for closeing the pull request, i messed up my commits and ended up with a 3 way split, that was not resolving corretly, i have fixed the merge and can repopen.

Be great if can reopen, as then the comments are preserved.

[marrobi]

@james-annages let me know when you are sorted and will have another run through. Thanks.

[marrobi]

Think getting closer, if can make the changes, and update the CHANGELOG.md file I will give it a test run.

Thank you!

[marrobi]

I'm thinking , should the backup policy live with the workspace service, so for VMs, guacamole workspace service? Then each use resource is going to need to be protected?

[james-annages]

humm possibly a good call. I was thinking of doing it this way they they are there allready.

[marrobi]

Maybe, then the frequency etc coudl be configured across the worksapce. Hmm.

[marrobi]

My worry is people think by ticking this box, everthing is being backed up. Many need a note to say "provided supported by the workspace services". The docs need to be clear.

[james-annages]

Aggreed. Would it be a possiblity to show a icon on ether user objets or workspace servise that show what is backed up. Also can add a tickbox per service that can enable/disable.
My hope was to add a tab that would show some details about backups/what is backed up. For now have it as show only.

[james-annages]

found a fun bug when trying to clean up a workspace. It soft deletes the Azure storage container befor it deletes the backups so that failes.
protecteditems.ProtectedItemsClient#Delete: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="BMSUserErrorContainerNotUndeleted" Message="Container is not yet undeleted. Please undelete the container that contains this datasource before attempting to undelete the datasource."
Might need to purge a diffrent way? Any idea's?

[marrobi]

found a fun bug when trying to clean up a workspace. It soft deletes the Azure storage container befor it deletes the backups so that failes. protecteditems.ProtectedItemsClient#Delete: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="BMSUserErrorContainerNotUndeleted" Message="Container is not yet undeleted. Please undelete the container that contains this datasource before attempting to undelete the datasource." Might need to purge a diffrent way? Any idea's?

A depends_on might work...

[marrobi]

@james-annages I'm going to give this a test today. Let me know if feel I shouldn't, will try look at the delete container issues.

[james-annages]

@james-annages I'm going to give this a test today. Let me know if feel I shouldn't, will try look at the delete container issues.

@marrobi Give it a go, i think im allready using the the depends_on tags? but may see something i have missed.

[marrobi]

@james-annages made a few changes, have put them in my branch here - f425e34

Can you pull them into yours?

Still testing delete, its getting further though. Thanks!

[marrobi]

Ok, now in the place where as soft_delete_enabled = true is on the vault cannot be deleted until after the retention period So options are turn soft_delete_enabled off, or leave the recovery vault when delete the workspace.

[james-annages]

Ok, now in the place where as soft_delete_enabled = true is on the vault cannot be deleted until after the retention period So options are turn soft_delete_enabled off, or leave the recovery vault when delete the workspace.

OK my script may be of help then.
If we leave the vault it will need the storage account and the rg to stay as well.
If we keep soft delete on it can protect from accidental deletion. I also think that it get readable on changes/additions to the vault (need to check the azure docs later).
It may be better to disable softdelete and see what happens when we backup a storage share then try and remove the workspace with the vault.

A possible middle ground could be a few commands in the uninstall step:

• az login
• turn off the soff delete setting
• ???possible purge all backups???
• terraform destroy.

[marrobi]

I'm not sure you can disable soft delete.

If it's turned off, the delete seems to work, but i my test is hanging on private endpoint deleting, not sure if it's related.

I think if it works, we disable soft delete, get the PR in, and can start another discussion on if people think that should be changed.

[marrobi]

If we were to leave the storage accounts we would start to hit quotas, especially if people have workspaces coming and going.

Can you test with the changes in my branch, with soft delete disabled.

It would actually be much easier with a vault in the core, btu then we have the multi subscription challenge. And it is quite tidy having a vault per workspace.

[james-annages]

If we were to leave the storage accounts we would start to hit quotas, especially if people have workspaces coming and going.

Can you test with the changes in my branch, with soft delete disabled.

It would actually be much easier with a vault in the core, btu then we have the multi subscription challenge. And it is quite tidy having a vault per workspace.

In our TRE we have a vault in the core and are haveing problems with storage accounts with diffrent CMK on them to what the Vault has. (may be a isolated problem.) We can do storage account across diffrtent subs but not the vms or SQL.
possibly use a core vault for the shares? but then its how to you manage the backups and policys...
Not sure on this one.

[marrobi]

If disable soft delete, can you get the workspace to deploy, do a backup, disable and delete?

I think that works for now.

I think you can change soft delete as long as it's not AlwaysOn.

Maybe as a next step we have it on, and turn it off as part of the workspace disable process.

[james-annages]

@marrobi Just tested the changes from your branch and they look to have worked.
I think it might be worth looking at switching it off as part of the uninstall.
As we have needed the soft delete on the storage accounts when updates have redeployed the share.

[marrobi]

Hmm, when have you seen a share get redeployed by an update?

[james-annages]

@marrobi had it happen when we moved from v0.19.1 to v0.20.0 it deleted the shares then redeployed them.

[marrobi]

😕

That's not good. So you upgraded the workspace and it deleted the shares? Any idea from what version to what?

Would like to track that down.

[marrobi]

Did you turn CMK on maybe?

Can you create an issue for that, as we need to make sure it doesn't happen to others

[james-annages]

I will do when I get in to the office tomorrow.