feat(rfc): make operator the default install for Spinnaker

[mashby2022]

No description provided.

[jasonmcintosh]

Going to ask: SHOULD the operator be moved to the spinnaker project instead of the armory project as part of this?

[mattgogerly]

Going to ask: SHOULD the operator be moved to the spinnaker project instead of the armory project as part of this?

I think that probably makes sense. I've not kept up to date with the maintenance of it - does the Armory Enterprise operator depend on the open source one, or are they completely independent?

[jasonmcintosh]

Going to ask: SHOULD the operator be moved to the spinnaker project instead of the armory project as part of this?

I think that probably makes sense. I've not kept up to date with the maintenance of it - does the Armory Enterprise operator depend on the open source one, or are they completely independent?

Completely independent :)

[dogonthehorizon]

@jasonmcintosh yes, we intend to donate the armory/spinnaker-operator project to OSS as a requirement for this work :D There's some evolving thinking internally on how we manage the two operators going forward, but not ready to talk about it publicly quite yet.

[jvz]

Got a link to this operator? This space has a few competing choices there.

[dogonthehorizon]

@jvz Here you go:
https://github.com/armory/spinnaker-operator

This space == Spinnaker operators, or install and configure? The intent of this RFC is to decide on the default while allowing competing options to continue :)

[jvz]

This looks like a solid project. The space was more so on the operators side, though it'd be great to work on standardizing one. The install and configure story is overly complex, and I'd love to see this simplify for common use cases like running in Kubernetes.

[jvz]

So I've got a question about this operator: would this make it easier for us to eventually introduce more microservices and potentially split up existing ones into more well-defined domains? If this makes the operational aspects of configuration and deployment of new services more straightforward than the status quo, then I'd be highly supportive of this operator idea. Sometimes, I worry that our microservices are getting monolithic in parts which has been a result of the difficulty in introducing new services to the project whether they be official services in this project or custom ones related to plugins, extensions, or whatever. If this operator approach can help address this issue, then you've got my strong support here!

[kskewes-sf]

Thank you for raising this RFC!

It would be great to progress the Spinnaker installation and day2 management story.

The Operator looks to be a potential candidate.
Though I wonder if we're going far enough with the Operator?

RE: Halyard
Halyard's main features are: templating of Kubernetes manifests, validation of some configuration values, deploying to Kubernetes. Will talk to these separately in following sections.
Halyard has served us well but is difficult for users to run in a modern git based workflow.
It is also a maintenance burden for the community and complicates the Spinnaker release process.
It would be great to archive the Halyard repository providing there is another on-rails option for the community. All as per the linked Halyard-Lite RFC (thank you), this RFC, comments here, in our issues and other community locations.
The Operator currently takes a dependency on Halyard I believe which is something we should avoid going forward.

RE: Templating
The Spinnaker Operator CRD's are a kind of template.
When deploying a single Spinnaker, the user can keep these raw yaml manifests in git and deploy them with Spinnaker, kubectl, etc. I've done this with Kind.
When deploying multiple Spinnakers, the user will probably template the yaml manifests using their choice of tooling, e.g: Helm, Kustomize, etc.
By the time a user templates the Spinnaker Operator CRD's they could potentially template the raw Kubernetes Deployment/ConfigMap/etc.

So one alternative to Operator here would be for a base Kustomize installation to be provided.
Potentially a refactor of https://github.com/spinnaker/kustomization-base, especially now Spring Boot 2.4 has landed, simplifying *.yml file management.
The https://github.com/armory/spinnaker-kustomize-patches repo provide "recipes" in Halyard form for a lot of common configuration but could be converted to their native spinnaker.yml form.
Again with Spring Boot 2.4 we can drop each spinnaker-kustomize-patch straight into the config dir, eg: /opt/spinnaker/config/<local|spring_profile>/<patch>.yml

RE: Validation
Halyard validates some configuration in ~/.hal/config and other files.
Kleat also does this but is unmaintained.
With kubeval, checkov, and other Kubernetes yaml linters used in the community, we don't need to duplicate Kubernetes related validation in Spinnaker project.
The Spinnaker services also perform some validation on startup.
If the configuration is invalid in some ways the services won't start and in Kubernetes this will likely result in the existing Pods with valid configuration continuing to handle requests.
It would be wonderful to have a schema validator (and generated docs?) for configuration that hooks into the codebases but I would consider these nice to haves (we don't have them now really).

RE: Deploying to Kubernetes
This is handled by Spinnaker, kubectl, terraform or any other tool used in the Kubernetes community.

RE: Other
Are there other features or elements of the Operator that I have missed? Please advise. :)

RE: Debians
Whilst it's not the subject of this RFC it is part of Halyard.
A basic Ansible playbook or Chef cookbook with the bare minimum could be provided as an alternative to Halyard. We could poll the community.

[kskewes-sf]

How would a Spinnaker user manage configuration for multiple Spinnaker installations?

I see Armory has a collection of kustomize patches for the Operator here: https://github.com/armory/spinnaker-kustomize-patches#kustomize-patches-for-armory

Is this repo involved in this RFC at all?

[kskewes-sf]

Whilst kleat is available, the current official installation process is Halyard per the Spinnaker docs - https://spinnaker.io/docs/setup/install/

[kskewes-sf]

Earlier in the doc we have:

The current recommended way to deploy Spinnaker is by using Kleat. While Kleat has improved some aspects of installing and configuring Spinnaker, it is currently not being maintained since Google has left the project. This document proposes that Operator should replace Kleat as the default install method.

1. What functionality does/would kleat provide to the Operator?
2. If kleat is taken as a dependency then it will need security uplift and ongoing maintenance and development. Does the effort required for an initial uplift and then ongoing look reasonable?

[kskewes-sf]

I'm not familiar with kleat adoption.
I see very little action in the #kleat slack channel. Some questioning the state of the project and also comments on bypassing kleat and using kustomization-base directly.
Do you have an idea?

[kskewes-sf]

Halyard is also used to install Spinnaker via Debian packages.

If we do not provide an alternative for Debian users then they will continue to use Halyard and thus we need to continue to support Halyard + "new thing (operator/other)".

I'm not familiar enough with kleat to comment but do you know if it's possible to use kleat with a Debian install process?

Overall I'm onboard with replacing the Halyard Kubernetes offering with something and tackling Debian installation separately with whatever is appropriate in that space (Ansible, chef, etc).

[mattgogerly]

The problem I've found, and that I have with operator/Kleat/any other Kustomize based install, is that when you're operating multiple instances of Spinnaker the file structure can become extremely unwieldy with overrides across each instance.

There are other issues with the Operator (and also the Operator/Halyard combo) approach that I've encountered, that I think would need solving before it can be considered an optimal install process:

1. The operator is reliant on installing CRD(s) to the cluster on which it's running. This isn't always possible, especially for teams running on multi-tenant infrastructure where access is restricted. This is more an issue with operators in general rather than being specific to this one.

2. The operator as far as I know is still just using Halyard underneath. It's therefore still reliant on having an object store to upload profile/service settings files to, which seems a) a barrier to entry and b) somewhat unnecessary since the config files are available in the repo for each service anyway. This also means teams have to ingest multiple images to use operator.

[ksrinimba]

IMO, Operator as a concept has not taken off as much as it was anticipated. Originally, from what I understood at that time, operator was thought as a mechanism that allowed users/groups to install/upgrade software that is controlled at org-level from the UI with a click of a button.

In Spinnaker context, it kind of adds another layer of complexity and, IMHO, the benefits provided do not seem to add-up, particularly in the context of the alternatives.

All said and done, Halyard is pretty good for most of the use-cases- it processes .hal/config and other files and creates the yamls. The problem is if we need something that is not supported by Halyard, we need to workaround.

An alternative, IMO, would be what someone already mentioned: Kustomize. Gives full controll over each of the yamls. We could break-up the .hal/config into multiple files, based on purpose (accounts, artifacts, etc.) making it lot more maintainable..not everyone needs all of the account-types, artifact-types, etc. My 2 cents.