sysbox misc hosts: how to connect them between each others?

[dblas]

Bravo for the job accomplished. It's now easy to create a few KinD nodes. But is it easy to do it when these nodes are on different hosts?
Are Cilium, Weave or Calico still possible in the sysbox context?
Thank you!
db

[rodnymolina]

@dblas, thanks!

Just to make sure I understood your question, you are referring to the capability of instantiating k8s-nodes in multiple servers (bare-metal / VMs) and interconnecting them through traditional CNI's. These k8s-nodes would be created by a regular K8s-controller, not by the KinD tool, right? If that's the case, yes, this is a key component of our roadmap.

Now, I'm curious, what's the use-case that you have in mind? Are you thinking about deploying these in the cloud or on-prem? When would you use a Sysbox-based node vs a regular VM?

[dblas]

I'd like to build that configuration.

With Docker and the sysbox runtime instead of a VM whatever it is built on.
Each column is a bare host.
Each set of rows of the same colour is a cluster: a production one and a test one.
The drawing then shows 10 nodes (6 for production and 4 for testing). These nodes will be created before.
That doesn't prevent me to create other nodes at cloud providers' using a tool like Rancher. But I'm looking here to reuse current hosts in order to be efficient.

It is perfectly feasible using up to 4 QEMU on a host as well as a CNI-compliant component like weave, calico, etc.
But I'd like to keep going on with a runc-like runtime in order to ensure transition with the current Docker containers that are in production while keeping a minimum level of security (so no DoutD there).
And, even (Christmas is coming soon :)) having a double view on such containers (a view through the common k8s tools as well as a view through docker ps).
Why? Because all the production procedures are written supposing a Docker environment but I need to go to k8s the smoothest way possible because it becomes impossible to manually manage all these containers (> 40).

Can you see what I mean?
Thank you for your answers.
db

[ctalledo]

Hi @dblas,

A couple of months ago I played around with deploying K8s-in-Docker across several hosts, but could not get it fully working yet.

Basically, I deployed a couple of K8s-in-Docker nodes across different machines, where each node was deployed with Docker + Sysbox. These were connected using a Docker overlay network.

I got as far as forming the K8s cluster (i.e., the k8s master and worker joined the cluster), but then hit some networking problem when the k8s master tried to deploy a pod to the k8s worker node.

In general, there is nothing in Sysbox that should prevent creating K8s-in-Docker nodes across several hosts, and it's definitely something we want to support as @rodnymolina mentioned.

The problems mostly show up because of the nesting of networks (e.g., there is one container network connecting the k8s-in-docker nodes across machines, and there is the (inner) networking setup by k8s inside the container-based k8s nodes). It's here that problems arise, and we need to look at these and understand how Sysbox can help fix them.

We don't have cycles to work on this config right now, but if you give it a shot we can definitely support you as you find issues.

[rodnymolina]

Yes, that sounds more feasible to me right now, even though we haven't tried it. Question, if you are considering to go down this Docker&Weave path as a temp workaround, what about just regular Docker-Swarm to instantiate your k8s-nodes and build a manual overlay for the apps within the k8s-nodes? I think that should work too.

[ctalledo]

I did not try Weave as a way of connecting the K8s-in-Docker containers across machines. I only tried with Docker overlay networks.

In general Sysbox is agnostic to the CNI, so using Weave or any other CNI ** should ** work. But as I mentioned, the problems typically arise when nesting CNIs.

[ctalledo]

I got as far as forming the K8s cluster (i.e., the k8s master and worker joined the cluster), but then hit some networking problem when the k8s master tried to deploy a pod to the k8s worker node.

There is a mistake there: In fact, I was able to deploy a k8s pod in the worker node, but hit a problem when exposing that pod via a K8s service (clusterIP), meaning that I could not access the pod via the service when trying the access on the K8s-master. The service access did work when done from the K8s-worker.

[dblas]

Hum. Things to test. I'm currently busy upgrading the hosts to kernel 5 as you requested. Will take a few days.
Stay tuned!
db

[ctalledo]

Cool! Once you are ready to try, ping us (via our Slack channel) and we can assist you with the setup and debug.

[ctalledo]

Forgot: I would also start with a simple network (e.g., flannel), both for the overlay network that connects the K8s-in-Docker nodes across machines, as well as for the K8s CNI itself. Once this is working we can try more advanced CNIs.

[ctalledo]

One more thing: K8s-in-Docker with Sysbox is definitely useful for testing at this time, but I would not use it for production K8s yet. It's not yet a certified K8s platform, and there are still some limitations. See here:

https://github.com/nestybox/sysbox/blob/master/docs/user-guide/kind.md#preliminary-support--known-limitations