Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

external-ip of loadbalancer service in wsl2 is pending #189

Open
d2461795341 opened this issue Jan 16, 2025 · 13 comments
Open

external-ip of loadbalancer service in wsl2 is pending #189

d2461795341 opened this issue Jan 16, 2025 · 13 comments

Comments

@d2461795341
Copy link

d2461795341 commented Jan 16, 2025 •
edited
Loading

I am using Ubuntu 22.04 on WSL2 and Docker Desktop on Windows, and I created a cluster using Kind within WSL2.
And the networkmode is mirrored, which means wsl2 and windows have the same ip.

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
  - role: worker

Then I created a service in the cluster.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:latest
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
      nodePort: 30080
  type: LoadBalancer

And I run cloud-provider-kind in WSL2, but I get

I0117 02:07:48.384702   33952 node_controller.go:271] Update 3 nodes status took 61.104576ms.
I0117 02:08:04.807284   33952 controller.go:401] Ensuring load balancer for service default/nginx
I0117 02:08:04.807365   33952 loadbalancer.go:28] Ensure LoadBalancer cluster: dkj-cluster service: nginx
I0117 02:08:04.807458   33952 event.go:389] "Event occurred" object="default/nginx" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"
I0117 02:08:04.850248   33952 server.go:113] updating loadbalancer
I0117 02:08:04.850306   33952 proxy.go:236] address type Hostname, only InternalIP supported
I0117 02:08:04.850325   33952 proxy.go:236] address type Hostname, only InternalIP supported
I0117 02:08:04.850330   33952 proxy.go:271] envoy config info: &{HealthCheckPort:10256 ServicePorts:map[IPv4_80_TCP:{Listener:{Address:0.0.0.0 Port:80 Protocol:TCP} Cluster:[{Address:172.18.0.2 Port:30080 Protocol:TCP} {Address:172.18.0.3 Port:30080 Protocol:TCP}]}] SessionAffinity:None SourceRanges:[]}
I0117 02:08:04.850467   33952 proxy.go:289] updating loadbalancer with config 
resources:
- "@type": type.googleapis.com/envoy.config.listener.v3.Listener
  name: listener_IPv4_80_TCP
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 80
      protocol: TCP
  filter_chains:
  - filters:
    - name: envoy.filters.network.tcp_proxy
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
        access_log:
        - name: envoy.file_access_log
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
        stat_prefix: tcp_proxy
        cluster: cluster_IPv4_80_TCP
I0117 02:08:04.905330   33952 proxy.go:300] updating loadbalancer with config 
resources:
- "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster
  name: cluster_IPv4_80_TCP
  connect_timeout: 5s
  type: STATIC
  lb_policy: RANDOM
  health_checks:
  - timeout: 5s
    interval: 3s
    unhealthy_threshold: 2
    healthy_threshold: 1
    no_traffic_interval: 5s
    always_log_health_check_failures: true
    always_log_health_check_success: true
    event_log_path: /dev/stdout
    http_health_check:
      path: /healthz
  load_assignment:
    cluster_name: cluster_IPv4_80_TCP
    endpoints:
      - lb_endpoints:
        - endpoint:
            health_check_config:
              port_value: 10256
            address:
              socket_address:
                address: 172.18.0.2
                port_value: 30080
                protocol: TCP
      - lb_endpoints:
        - endpoint:
            health_check_config:
              port_value: 10256
            address:
              socket_address:
                address: 172.18.0.3
                port_value: 30080
                protocol: TCP
I0117 02:08:05.021652   33952 server.go:121] updating loadbalancer tunnels on userspace
I0117 02:08:05.040583   33952 tunnel.go:34] found port maps map[10000:60733 80:60805] associated to container kindccm-RZTQCN5REF5VESVEBSRFF5NKA2S54JVJFWGT3EOM
I0117 02:08:05.060601   33952 tunnel.go:41] setting IPv4 address 172.18.0.5 associated to container kindccm-RZTQCN5REF5VESVEBSRFF5NKA2S54JVJFWGT3EOM
E0117 02:08:05.072794   33952 controller.go:301] "Unhandled Error" err="error processing service default/nginx (retrying with exponential backoff): failed to ensure load balancer: exit status 2" logger="UnhandledError"
I0117 02:08:05.072864   33952 event.go:389] "Event occurred" object="default/nginx" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message="Error syncing load balancer: failed to ensure load balancer: exit status 2"
I0117 02:08:18.384799   33952 instances.go:47] Check instance metadata for dkj-cluster-control-plane
I0117 02:08:18.384820   33952 instances.go:47] Check instance metadata for dkj-cluster-worker2
I0117 02:08:18.384821   33952 instances.go:47] Check instance metadata for dkj-cluster-worker
I0117 02:08:18.437675   33952 instances.go:75] instance metadata for dkj-cluster-control-plane: &cloudprovider.InstanceMetadata{ProviderID:"kind://dkj-cluster/kind/dkj-cluster-control-plane", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"dkj-cluster-control-plane"}, v1.NodeAddress{Type:"InternalIP", Address:"172.18.0.4"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::4"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I0117 02:08:18.439573   33952 instances.go:75] instance metadata for dkj-cluster-worker: &cloudprovider.InstanceMetadata{ProviderID:"kind://dkj-cluster/kind/dkj-cluster-worker", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"dkj-cluster-worker"}, v1.NodeAddress{Type:"InternalIP", Address:"172.18.0.2"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::2"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I0117 02:08:18.441863   33952 instances.go:75] instance metadata for dkj-cluster-worker2: &cloudprovider.InstanceMetadata{ProviderID:"kind://dkj-cluster/kind/dkj-cluster-worker2", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"dkj-cluster-worker2"}, v1.NodeAddress{Type:"InternalIP", Address:"172.18.0.3"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::3"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I0117 02:08:18.449086   33952 node_controller.go:271] Update 3 nodes status took 64.348701ms.
I0117 02:08:48.448790   33952 instances.go:47] Check instance metadata for dkj-cluster-worker
I0117 02:08:48.448810   33952 instances.go:47] Check instance metadata for dkj-cluster-worker2
I0117 02:08:48.448838   33952 instances.go:47] Check instance metadata for dkj-cluster-control-plane
I0117 02:08:48.502731   33952 instances.go:75] instance metadata for dkj-cluster-control-plane: &cloudprovider.InstanceMetadata{ProviderID:"kind://dkj-cluster/kind/dkj-cluster-control-plane", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"dkj-cluster-control-plane"}, v1.NodeAddress{Type:"InternalIP", Address:"172.18.0.4"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::4"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I0117 02:08:48.503326   33952 instances.go:75] instance metadata for dkj-cluster-worker2: &cloudprovider.InstanceMetadata{ProviderID:"kind://dkj-cluster/kind/dkj-cluster-worker2", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"dkj-cluster-worker2"}, v1.NodeAddress{Type:"InternalIP", Address:"172.18.0.3"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::3"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I0117 02:08:48.504637   33952 instances.go:75] instance metadata for dkj-cluster-worker: &cloudprovider.InstanceMetadata{ProviderID:"kind://dkj-cluster/kind/dkj-cluster-worker", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"dkj-cluster-worker"}, v1.NodeAddress{Type:"InternalIP", Address:"172.18.0.2"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::2"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I0117 02:08:48.512145   33952 node_controller.go:271] Update 3 nodes status took 63.396359ms.
I0117 02:09:18.511538   33952 instances.go:47] Check instance metadata for dkj-cluster-worker
I0117 02:09:18.511579   33952 instances.go:47] Check instance metadata for dkj-cluster-worker2
I0117 02:09:18.511544   33952 instances.go:47] Check instance metadata for dkj-cluster-control-plane
I0117 02:09:18.568555   33952 instances.go:75] instance metadata for dkj-cluster-worker2: &cloudprovider.InstanceMetadata{ProviderID:"kind://dkj-cluster/kind/dkj-cluster-worker2", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"dkj-cluster-worker2"}, v1.NodeAddress{Type:"InternalIP", Address:"172.18.0.3"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::3"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I0117 02:09:18.573519   33952 instances.go:75] instance metadata for dkj-cluster-worker: &cloudprovider.InstanceMetadata{ProviderID:"kind://dkj-cluster/kind/dkj-cluster-worker", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"dkj-cluster-worker"}, v1.NodeAddress{Type:"InternalIP", Address:"172.18.0.2"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::2"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I0117 02:09:18.575028   33952 instances.go:75] instance metadata for dkj-cluster-control-plane: &cloudprovider.InstanceMetadata{ProviderID:"kind://dkj-cluster/kind/dkj-cluster-control-plane", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"dkj-cluster-control-plane"}, v1.NodeAddress{Type:"InternalIP", Address:"172.18.0.4"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::4"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I0117 02:09:18.584082   33952 node_controller.go:271] Update 3 nodes status took 72.621714ms.
I0117 02:09:25.075336   33952 controller.go:401] Ensuring load balancer for service default/nginx
I0117 02:09:25.075420   33952 loadbalancer.go:28] Ensure LoadBalancer cluster: dkj-cluster service: nginx
I0117 02:09:25.075915   33952 event.go:389] "Event occurred" object="default/nginx" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"
I0117 02:09:25.130290   33952 server.go:113] updating loadbalancer
I0117 02:09:25.130353   33952 proxy.go:236] address type Hostname, only InternalIP supported
I0117 02:09:25.130377   33952 proxy.go:236] address type Hostname, only InternalIP supported
I0117 02:09:25.130398   33952 proxy.go:271] envoy config info: &{HealthCheckPort:10256 ServicePorts:map[IPv4_80_TCP:{Listener:{Address:0.0.0.0 Port:80 Protocol:TCP} Cluster:[{Address:172.18.0.2 Port:30080 Protocol:TCP} {Address:172.18.0.3 Port:30080 Protocol:TCP}]}] SessionAffinity:None SourceRanges:[]}
I0117 02:09:25.130591   33952 proxy.go:289] updating loadbalancer with config 
resources:
- "@type": type.googleapis.com/envoy.config.listener.v3.Listener
  name: listener_IPv4_80_TCP
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 80
      protocol: TCP
  filter_chains:
  - filters:
    - name: envoy.filters.network.tcp_proxy
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
        access_log:
        - name: envoy.file_access_log
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
        stat_prefix: tcp_proxy
        cluster: cluster_IPv4_80_TCP
I0117 02:09:25.185797   33952 proxy.go:300] updating loadbalancer with config 
resources:
- "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster
  name: cluster_IPv4_80_TCP
  connect_timeout: 5s
  type: STATIC
  lb_policy: RANDOM
  health_checks:
  - timeout: 5s
    interval: 3s
    unhealthy_threshold: 2
    healthy_threshold: 1
    no_traffic_interval: 5s
    always_log_health_check_failures: true
    always_log_health_check_success: true
    event_log_path: /dev/stdout
    http_health_check:
      path: /healthz
  load_assignment:
    cluster_name: cluster_IPv4_80_TCP
    endpoints:
      - lb_endpoints:
        - endpoint:
            health_check_config:
              port_value: 10256
            address:
              socket_address:
                address: 172.18.0.2
                port_value: 30080
                protocol: TCP
      - lb_endpoints:
        - endpoint:
            health_check_config:
              port_value: 10256
            address:
              socket_address:
                address: 172.18.0.3
                port_value: 30080
                protocol: TCP
I0117 02:09:25.297034   33952 server.go:121] updating loadbalancer tunnels on userspace
I0117 02:09:25.317199   33952 tunnel.go:34] found port maps map[10000:60733 80:60805] associated to container kindccm-RZTQCN5REF5VESVEBSRFF5NKA2S54JVJFWGT3EOM
I0117 02:09:25.336578   33952 tunnel.go:41] setting IPv4 address 172.18.0.5 associated to container kindccm-RZTQCN5REF5VESVEBSRFF5NKA2S54JVJFWGT3EOM
E0117 02:09:25.348259   33952 controller.go:301] "Unhandled Error" err="error processing service default/nginx (retrying with exponential backoff): failed to ensure load balancer: exit status 2" logger="UnhandledError"
I0117 02:09:25.348310   33952 event.go:389] "Event occurred" object="default/nginx" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message="Error syncing load balancer: failed to ensure load balancer: exit status 2"
I0117 02:09:48.583325   33952 instances.go:47] Check instance metadata for dkj-cluster-worker
I0117 02:09:48.583346   33952 instances.go:47] Check instance metadata for dkj-cluster-worker2
I0117 02:09:48.583332   33952 instances.go:47] Check instance metadata for dkj-cluster-control-plane
I0117 02:09:48.638371   33952 instances.go:75] instance metadata for dkj-cluster-worker: &cloudprovider.InstanceMetadata{ProviderID:"kind://dkj-cluster/kind/dkj-cluster-worker", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"dkj-cluster-worker"}, v1.NodeAddress{Type:"InternalIP", Address:"172.18.0.2"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::2"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I0117 02:09:48.639902   33952 instances.go:75] instance metadata for dkj-cluster-control-plane: &cloudprovider.InstanceMetadata{ProviderID:"kind://dkj-cluster/kind/dkj-cluster-control-plane", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"dkj-cluster-control-plane"}, v1.NodeAddress{Type:"InternalIP", Address:"172.18.0.4"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::4"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I0117 02:09:48.640624   33952 instances.go:75] instance metadata for dkj-cluster-worker2: &cloudprovider.InstanceMetadata{ProviderID:"kind://dkj-cluster/kind/dkj-cluster-worker2", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"dkj-cluster-worker2"}, v1.NodeAddress{Type:"InternalIP", Address:"172.18.0.3"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::3"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I0117 02:09:48.648178   33952 node_controller.go:271] Update 3 nodes status took 64.895761ms.
I0117 02:10:18.648418   33952 instances.go:47] Check instance metadata for dkj-cluster-worker2
I0117 02:10:18.648549   33952 instances.go:47] Check instance metadata for dkj-cluster-control-plane
I0117 02:10:18.648560   33952 instances.go:47] Check instance metadata for dkj-cluster-worker
I0117 02:10:18.707063   33952 instances.go:75] instance metadata for dkj-cluster-control-plane: &cloudprovider.InstanceMetadata{ProviderID:"kind://dkj-cluster/kind/dkj-cluster-control-plane", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"dkj-cluster-control-plane"}, v1.NodeAddress{Type:"InternalIP", Address:"172.18.0.4"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::4"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I0117 02:10:18.708104   33952 instances.go:75] instance metadata for dkj-cluster-worker2: &cloudprovider.InstanceMetadata{ProviderID:"kind://dkj-cluster/kind/dkj-cluster-worker2", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"dkj-cluster-worker2"}, v1.NodeAddress{Type:"InternalIP", Address:"172.18.0.3"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::3"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I0117 02:10:18.710417   33952 instances.go:75] instance metadata for dkj-cluster-worker: &cloudprovider.InstanceMetadata{ProviderID:"kind://dkj-cluster/kind/dkj-cluster-worker", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"dkj-cluster-worker"}, v1.NodeAddress{Type:"InternalIP", Address:"172.18.0.2"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::2"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I0117 02:10:18.720142   33952 node_controller.go:271] Update 3 nodes status took 71.791589ms.
I0117 02:10:48.720142   33952 instances.go:47] Check instance metadata for dkj-cluster-control-plane
I0117 02:10:48.720142   33952 instances.go:47] Check instance metadata for dkj-cluster-worker2
I0117 02:10:48.720142   33952 instances.go:47] Check instance metadata for dkj-cluster-worker
I0117 02:10:48.774506   33952 instances.go:75] instance metadata for dkj-cluster-worker: &cloudprovider.InstanceMetadata{ProviderID:"kind://dkj-cluster/kind/dkj-cluster-worker", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"dkj-cluster-worker"}, v1.NodeAddress{Type:"InternalIP", Address:"172.18.0.2"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::2"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I0117 02:10:48.774806   33952 instances.go:75] instance metadata for dkj-cluster-worker2: &cloudprovider.InstanceMetadata{ProviderID:"kind://dkj-cluster/kind/dkj-cluster-worker2", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"dkj-cluster-worker2"}, v1.NodeAddress{Type:"InternalIP", Address:"172.18.0.3"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::3"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I0117 02:10:48.775963   33952 instances.go:75] instance metadata for dkj-cluster-control-plane: &cloudprovider.InstanceMetadata{ProviderID:"kind://dkj-cluster/kind/dkj-cluster-control-plane", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"dkj-cluster-control-plane"}, v1.NodeAddress{Type:"InternalIP", Address:"172.18.0.4"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::4"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I0117 02:10:48.782478   33952 node_controller.go:271] Update 3 nodes status took 62.426091ms.
I0117 02:11:18.782296   33952 instances.go:47] Check instance metadata for dkj-cluster-control-plane
I0117 02:11:18.782581   33952 instances.go:47] Check instance metadata for dkj-cluster-worker2
I0117 02:11:18.782974   33952 instances.go:47] Check instance metadata for dkj-cluster-worker
I0117 02:11:18.832230   33952 instances.go:75] instance metadata for dkj-cluster-worker2: &cloudprovider.InstanceMetadata{ProviderID:"kind://dkj-cluster/kind/dkj-cluster-worker2", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"dkj-cluster-worker2"}, v1.NodeAddress{Type:"InternalIP", Address:"172.18.0.3"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::3"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I0117 02:11:18.838546   33952 instances.go:75] instance metadata for dkj-cluster-worker: &cloudprovider.InstanceMetadata{ProviderID:"kind://dkj-cluster/kind/dkj-cluster-worker", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"dkj-cluster-worker"}, v1.NodeAddress{Type:"InternalIP", Address:"172.18.0.2"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::2"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I0117 02:11:18.840172   33952 instances.go:75] instance metadata for dkj-cluster-control-plane: &cloudprovider.InstanceMetadata{ProviderID:"kind://dkj-cluster/kind/dkj-cluster-control-plane", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"dkj-cluster-control-plane"}, v1.NodeAddress{Type:"InternalIP", Address:"172.18.0.4"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::4"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I0117 02:11:18.847461   33952 node_controller.go:271] Update 3 nodes status took 65.453578ms.
I0117 02:11:48.847407   33952 instances.go:47] Check instance metadata for dkj-cluster-worker2
I0117 02:11:48.847407   33952 instances.go:47] Check instance metadata for dkj-cluster-worker
I0117 02:11:48.848327   33952 instances.go:47] Check instance metadata for dkj-cluster-control-plane
I0117 02:11:48.899004   33952 instances.go:75] instance metadata for dkj-cluster-worker2: &cloudprovider.InstanceMetadata{ProviderID:"kind://dkj-cluster/kind/dkj-cluster-worker2", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"dkj-cluster-worker2"}, v1.NodeAddress{Type:"InternalIP", Address:"172.18.0.3"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::3"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I0117 02:11:48.901465   33952 instances.go:75] instance metadata for dkj-cluster-control-plane: &cloudprovider.InstanceMetadata{ProviderID:"kind://dkj-cluster/kind/dkj-cluster-control-plane", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"dkj-cluster-control-plane"}, v1.NodeAddress{Type:"InternalIP", Address:"172.18.0.4"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::4"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I0117 02:11:48.906711   33952 instances.go:75] instance metadata for dkj-cluster-worker: &cloudprovider.InstanceMetadata{ProviderID:"kind://dkj-cluster/kind/dkj-cluster-worker", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"dkj-cluster-worker"}, v1.NodeAddress{Type:"InternalIP", Address:"172.18.0.2"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::2"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I0117 02:11:48.914329   33952 node_controller.go:271] Update 3 nodes status took 67.051448ms.

and the status of the service is
Image

the log of container kindccm is

2025-01-17 02:08:04 [2025-01-16 18:08:04.994][11][info][upstream] [source/common/upstream/cds_api_helper.cc:32] cds: add 1 cluster(s), remove 0 cluster(s)
2025-01-17 02:08:04 [2025-01-16 18:08:04.995][11][info][upstream] [source/common/upstream/cds_api_helper.cc:71] cds: added/updated 0 cluster(s), skipped 1 unmodified cluster(s)
2025-01-17 02:08:08 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-16T18:07:59.202Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 02:08:08 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.3","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-16T18:07:59.202Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 02:08:08 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-16T18:08:04.200Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 02:08:08 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.3","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-16T18:08:04.200Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 02:08:08 [2025-01-16T18:08:05.021Z] "GET /ready HTTP/1.1" 200 - 0 5 0 - "172.18.0.1" "Go-http-client/1.1" "-" "127.0.0.1:60733" "-"
2025-01-17 02:08:18 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.3","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-16T18:08:09.198Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 02:08:18 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-16T18:08:09.198Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 02:08:18 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.3","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-16T18:08:14.207Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 02:08:18 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-16T18:08:14.207Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 02:08:28 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.3","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-16T18:08:19.216Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 02:08:28 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-16T18:08:19.216Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 02:08:28 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.3","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-16T18:08:24.214Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 02:08:28 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-16T18:08:24.214Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 02:08:38 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.3","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-16T18:08:29.213Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 02:08:38 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-16T18:08:29.213Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 02:08:38 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.3","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-16T18:08:34.217Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 02:08:38 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-16T18:08:34.217Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 02:08:48 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.3","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-16T18:08:39.218Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 02:08:48 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-16T18:08:39.218Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 02:08:48 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.3","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-16T18:08:44.217Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 02:08:48 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-16T18:08:44.217Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 02:08:58 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.3","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-16T18:08:49.226Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 02:08:58 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-16T18:08:49.226Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 02:08:58 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.3","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-16T18:08:54.235Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 02:08:58 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-16T18:08:54.235Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 02:09:08 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.3","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-16T18:08:59.235Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 02:09:08 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-16T18:08:59.235Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 02:09:08 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.3","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-16T18:09:04.233Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 02:09:08 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-16T18:09:04.233Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 02:09:18 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.3","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-16T18:09:09.232Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 02:09:18 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-16T18:09:09.232Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 02:09:18 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-16T18:09:14.230Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 02:09:18 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.3","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-16T18:09:14.230Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 02:09:25 [2025-01-16 18:09:25.271][11][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.core.v3.HealthCheck Using deprecated option 'envoy.config.core.v3.HealthCheck.event_log_path' from file health_check.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
@aojea
Copy link
Contributor

aojea commented Jan 16, 2025

this is weird

E0117 02:09:25.348259 33952 controller.go:301] "Unhandled Error" err="error processing service default/nginx (retrying with exponential backoff): failed to ensure load balancer: exit status 2" logger="UnhandledError"

can you see if the loadbalancer containerd (kindccm-...) has restarted or something?

@d2461795341
Copy link
Author

The kindccm container has not restarted

Image

Its log is only continuously outputting

2025-01-17 11:30:53 [2025-01-17 03:30:53.021][11][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.core.v3.HealthCheck Using deprecated option 'envoy.config.core.v3.HealthCheck.event_log_path' from file health_check.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
2025-01-17 11:30:53 [2025-01-17 03:30:53.021][11][info][upstream] [source/common/upstream/cds_api_helper.cc:32] cds: add 1 cluster(s), remove 0 cluster(s)
2025-01-17 11:30:53 [2025-01-17 03:30:53.021][11][info][upstream] [source/common/upstream/cds_api_helper.cc:71] cds: added/updated 0 cluster(s), skipped 1 unmodified cluster(s)
2025-01-17 11:30:55 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:30:47.546Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:30:55 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:30:47.546Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:30:55 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:30:52.545Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:30:55 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:30:52.545Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:30:55 [2025-01-17T03:30:53.046Z] "GET /ready HTTP/1.1" 200 - 0 5 0 - "172.18.0.1" "Go-http-client/1.1" "-" "127.0.0.1:60701" "-"
2025-01-17 11:31:05 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:30:57.545Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:31:05 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:30:57.545Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:31:05 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:31:02.539Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:31:05 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:31:02.539Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:31:15 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:31:07.548Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:31:15 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:31:07.548Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:31:15 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:31:12.557Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:31:15 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:31:12.557Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:31:25 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:31:17.555Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:31:25 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:31:17.556Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:31:25 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:31:22.554Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:31:25 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:31:22.554Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:31:35 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:31:27.553Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:31:35 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:31:27.553Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:31:35 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:31:32.552Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:31:35 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:31:32.552Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:31:45 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:31:37.558Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:31:45 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:31:37.558Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:31:45 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:31:42.566Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:31:45 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:31:42.566Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:31:55 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:31:47.564Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:31:55 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:31:47.564Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:31:55 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:31:52.562Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:31:55 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:31:52.562Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:32:05 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:31:57.558Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:32:05 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:31:57.559Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:32:05 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:32:02.559Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:32:05 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:32:02.559Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:32:13 [2025-01-17 03:32:13.276][11][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.core.v3.HealthCheck Using deprecated option 'envoy.config.core.v3.HealthCheck.event_log_path' from file health_check.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
2025-01-17 11:32:13 [2025-01-17 03:32:13.276][11][info][upstream] [source/common/upstream/cds_api_helper.cc:32] cds: add 1 cluster(s), remove 0 cluster(s)
2025-01-17 11:32:13 [2025-01-17 03:32:13.277][11][info][upstream] [source/common/upstream/cds_api_helper.cc:71] cds: added/updated 1 cluster(s), skipped 0 unmodified cluster(s)
2025-01-17 11:32:15 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:32:07.557Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:32:15 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:32:07.557Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:32:15 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:32:12.560Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:32:15 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:32:12.560Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:32:15 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","add_healthy_event":{"first_check":true},"timestamp":"2025-01-17T03:32:13.278Z","locality":{"region":"","zone":"","sub_zone":""}}
2025-01-17 11:32:15 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","add_healthy_event":{"first_check":true},"timestamp":"2025-01-17T03:32:13.278Z","locality":{"region":"","zone":"","sub_zone":""}}
2025-01-17 11:32:15 [2025-01-17T03:32:13.301Z] "GET /ready HTTP/1.1" 200 - 0 5 0 - "172.18.0.1" "Go-http-client/1.1" "-" "127.0.0.1:60701" "-"
2025-01-17 11:32:25 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:32:18.280Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:32:25 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:32:18.280Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:32:25 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:32:23.278Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:32:25 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:32:23.279Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:32:35 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:32:28.277Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:32:35 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:32:28.277Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:32:35 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:32:33.287Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:32:35 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:32:33.287Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:32:45 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:32:38.293Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:32:45 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:32:38.293Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:32:45 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:32:43.292Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:32:45 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:32:43.292Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:32:55 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:32:48.290Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:32:55 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:32:48.290Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:32:55 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:32:53.288Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:32:55 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:32:53.288Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:33:05 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:32:58.298Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:33:05 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:32:58.298Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:33:05 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:33:03.297Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:33:05 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:33:03.297Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:33:15 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:33:08.296Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:33:15 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:33:08.296Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:33:15 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:33:13.304Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:33:15 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:33:13.304Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:33:25 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:33:18.304Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:33:25 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:33:18.304Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:33:25 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:33:23.313Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:33:25 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:33:23.313Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:33:35 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:33:28.311Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:33:35 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:33:28.311Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:33:35 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:33:33.309Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:33:35 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:33:33.309Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:33:45 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:33:38.307Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:33:45 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:33:38.307Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:33:45 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:33:43.313Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:33:45 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:33:43.313Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:33:55 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:33:48.311Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:33:55 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:33:48.311Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:33:55 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:33:53.310Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:33:55 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:33:53.310Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:34:05 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:33:58.309Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:34:05 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:33:58.309Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:34:05 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:34:03.308Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:34:05 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:34:03.308Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:34:15 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:34:08.315Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:34:15 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:34:08.315Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:34:15 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:34:13.323Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:34:15 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:34:13.324Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:34:25 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:34:18.322Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:34:25 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:34:18.323Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:34:25 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:34:23.321Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:34:25 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:34:23.321Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:34:35 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:34:28.320Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:34:35 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:34:28.320Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:34:35 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:34:33.318Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:34:35 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:34:33.318Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:34:45 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:34:38.316Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:34:45 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:34:38.316Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:34:45 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:34:43.323Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 11:34:45 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T03:34:43.324Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}

@d2461795341
Copy link
Author

and I try to run cloud-provider-kind in windows rather than wsl2
the result is similar

the ouput of cloud-provider-kind is

I0117 11:57:35.675525   27616 server.go:105] creating container for loadbalancer
I0117 11:57:35.675525   27616 server.go:251] creating loadbalancer with parameters: [--detach --tty --label io.x-k8s.cloud-provider-kind.cluster=dkj-cluster --label io.x-k8s.cloud-provider-kind.loadbalancer.name=dkj-cluster/default/nginx --net kind --init=false --hostname kindccm-RZTQCN5REF5VESVEBSRFF5NKA2S54JVJFWGT3EOM --privileged --restart=on-failure --sysctl=net.ipv4.ip_forward=1 --sysctl=net.ipv4.conf.all.rp_filter=0 --sysctl=net.ipv4.ip_unprivileged_port_start=1 --publish=80/TCP --publish=10000/TCP --publish-all docker.io/envoyproxy/envoy:v1.30.1 bash -c echo -en 'node:
  cluster: cloud-provider-kind
  id: cloud-provider-kind-id

dynamic_resources:
  cds_config:
    resource_api_version: V3
    path: /home/envoy/cds.yaml
  lds_config:
    resource_api_version: V3
    path: /home/envoy/lds.yaml

admin:
  access_log_path: /dev/stdout
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 10000
' > /home/envoy/envoy.yaml && touch /home/envoy/cds.yaml && touch /home/envoy/lds.yaml && while true; do envoy -c /home/envoy/envoy.yaml && break; sleep 1; done]
I0117 11:57:36.169092   27616 server.go:113] updating loadbalancer
I0117 11:57:36.169615   27616 proxy.go:236] address type Hostname, only InternalIP supported
I0117 11:57:36.169615   27616 proxy.go:236] address type Hostname, only InternalIP supported
I0117 11:57:36.169615   27616 proxy.go:271] envoy config info: &{HealthCheckPort:10256 ServicePorts:map[IPv4_80_TCP:{Listener:{Address:0.0.0.0 Port:80 Protocol:TCP} Cluster:[{Address:172.18.0.2 Port:30080 Protocol:TCP} {Address:172.18.0.4 Port:30080 Protocol:TCP}]}] SessionAffinity:None SourceRanges:[]}
I0117 11:57:36.169615   27616 proxy.go:289] updating loadbalancer with config
resources:
- "@type": type.googleapis.com/envoy.config.listener.v3.Listener
  name: listener_IPv4_80_TCP
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 80
      protocol: TCP
  filter_chains:
  - filters:
    - name: envoy.filters.network.tcp_proxy
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
        access_log:
        - name: envoy.file_access_log
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
        stat_prefix: tcp_proxy
        cluster: cluster_IPv4_80_TCP
I0117 11:57:36.306974   27616 proxy.go:300] updating loadbalancer with config
resources:
- "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster
  name: cluster_IPv4_80_TCP
  connect_timeout: 5s
  type: STATIC
  lb_policy: RANDOM
  health_checks:
  - timeout: 5s
    interval: 3s
    unhealthy_threshold: 2
    healthy_threshold: 1
    no_traffic_interval: 5s
    always_log_health_check_failures: true
    always_log_health_check_success: true
    event_log_path: /dev/stdout
    http_health_check:
      path: /healthz
  load_assignment:
    cluster_name: cluster_IPv4_80_TCP
    endpoints:
      - lb_endpoints:
        - endpoint:
            health_check_config:
              port_value: 10256
            address:
              socket_address:
                address: 172.18.0.2
                port_value: 30080
                protocol: TCP
      - lb_endpoints:
        - endpoint:
            health_check_config:
              port_value: 10256
            address:
              socket_address:
                address: 172.18.0.4
                port_value: 30080
                protocol: TCP
I0117 11:57:36.707818   27616 server.go:121] updating loadbalancer tunnels on userspace
I0117 11:57:36.813764   27616 tunnel.go:34] found port maps map[10000:60731 80:60807] associated to container kindccm-RZTQCN5REF5VESVEBSRFF5NKA2S54JVJFWGT3EOM
I0117 11:57:36.922123   27616 tunnel.go:41] setting IPv4 address 172.18.0.5 associated to container kindccm-RZTQCN5REF5VESVEBSRFF5NKA2S54JVJFWGT3EOM
E0117 11:57:36.996281   27616 controller.go:301] "Unhandled Error" err="error processing service default/nginx (retrying with exponential backoff): failed to ensure load balancer: exit status 1" logger="UnhandledError"
I0117 11:57:36.996281   27616 event.go:389] "Event occurred" object="default/nginx" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message="Error syncing load balancer: failed to ensure load balancer: exit status 1"
I0117 11:57:41.998212   27616 controller.go:401] Ensuring load balancer for service default/nginx
I0117 11:57:41.998212   27616 loadbalancer.go:28] Ensure LoadBalancer cluster: dkj-cluster service: nginx
I0117 11:57:41.998212   27616 event.go:389] "Event occurred" object="default/nginx" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"
I0117 11:57:42.219233   27616 server.go:113] updating loadbalancer
I0117 11:57:42.219233   27616 proxy.go:236] address type Hostname, only InternalIP supported
I0117 11:57:42.219233   27616 proxy.go:236] address type Hostname, only InternalIP supported
I0117 11:57:42.219233   27616 proxy.go:271] envoy config info: &{HealthCheckPort:10256 ServicePorts:map[IPv4_80_TCP:{Listener:{Address:0.0.0.0 Port:80 Protocol:TCP} Cluster:[{Address:172.18.0.2 Port:30080 Protocol:TCP} {Address:172.18.0.4 Port:30080 Protocol:TCP}]}] SessionAffinity:None SourceRanges:[]}
I0117 11:57:42.219233   27616 proxy.go:289] updating loadbalancer with config
resources:
- "@type": type.googleapis.com/envoy.config.listener.v3.Listener
  name: listener_IPv4_80_TCP
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 80
      protocol: TCP
  filter_chains:
  - filters:
    - name: envoy.filters.network.tcp_proxy
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
        access_log:
        - name: envoy.file_access_log
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
        stat_prefix: tcp_proxy
        cluster: cluster_IPv4_80_TCP
I0117 11:57:42.373978   27616 proxy.go:300] updating loadbalancer with config
resources:
- "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster
  name: cluster_IPv4_80_TCP
  connect_timeout: 5s
  type: STATIC
  lb_policy: RANDOM
  health_checks:
  - timeout: 5s
    interval: 3s
    unhealthy_threshold: 2
    healthy_threshold: 1
    no_traffic_interval: 5s
    always_log_health_check_failures: true
    always_log_health_check_success: true
    event_log_path: /dev/stdout
    http_health_check:
      path: /healthz
  load_assignment:
    cluster_name: cluster_IPv4_80_TCP
    endpoints:
      - lb_endpoints:
        - endpoint:
            health_check_config:
              port_value: 10256
            address:
              socket_address:
                address: 172.18.0.2
                port_value: 30080
                protocol: TCP
      - lb_endpoints:
        - endpoint:
            health_check_config:
              port_value: 10256
            address:
              socket_address:
                address: 172.18.0.4
                port_value: 30080
                protocol: TCP
I0117 11:57:42.787588   27616 server.go:121] updating loadbalancer tunnels on userspace
I0117 11:57:42.894663   27616 tunnel.go:34] found port maps map[10000:60731 80:60807] associated to container kindccm-RZTQCN5REF5VESVEBSRFF5NKA2S54JVJFWGT3EOM
I0117 11:57:43.002697   27616 tunnel.go:41] setting IPv4 address 172.18.0.5 associated to container kindccm-RZTQCN5REF5VESVEBSRFF5NKA2S54JVJFWGT3EOM
E0117 11:57:43.076988   27616 controller.go:301] "Unhandled Error" err="error processing service default/nginx (retrying with exponential backoff): failed to ensure load balancer: exit status 1" logger="UnhandledError"
I0117 11:57:43.076988   27616 event.go:389] "Event occurred" object="default/nginx" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message="Error syncing load balancer: failed to ensure load balancer: exit status 1"
I0117 11:57:53.079030   27616 controller.go:401] Ensuring load balancer for service default/nginx
I0117 11:57:53.079030   27616 loadbalancer.go:28] Ensure LoadBalancer cluster: dkj-cluster service: nginx
I0117 11:57:53.079030   27616 event.go:389] "Event occurred" object="default/nginx" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"
I0117 11:57:53.306121   27616 server.go:113] updating loadbalancer
I0117 11:57:53.306121   27616 proxy.go:236] address type Hostname, only InternalIP supported
I0117 11:57:53.306638   27616 proxy.go:236] address type Hostname, only InternalIP supported
I0117 11:57:53.306638   27616 proxy.go:271] envoy config info: &{HealthCheckPort:10256 ServicePorts:map[IPv4_80_TCP:{Listener:{Address:0.0.0.0 Port:80 Protocol:TCP} Cluster:[{Address:172.18.0.4 Port:30080 Protocol:TCP} {Address:172.18.0.2 Port:30080 Protocol:TCP}]}] SessionAffinity:None SourceRanges:[]}
I0117 11:57:53.306638   27616 proxy.go:289] updating loadbalancer with config
resources:
- "@type": type.googleapis.com/envoy.config.listener.v3.Listener
  name: listener_IPv4_80_TCP
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 80
      protocol: TCP
  filter_chains:
  - filters:
    - name: envoy.filters.network.tcp_proxy
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
        access_log:
        - name: envoy.file_access_log
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
        stat_prefix: tcp_proxy
        cluster: cluster_IPv4_80_TCP
I0117 11:57:53.443313   27616 proxy.go:300] updating loadbalancer with config
resources:
- "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster
  name: cluster_IPv4_80_TCP
  connect_timeout: 5s
  type: STATIC
  lb_policy: RANDOM
  health_checks:
  - timeout: 5s
    interval: 3s
    unhealthy_threshold: 2
    healthy_threshold: 1
    no_traffic_interval: 5s
    always_log_health_check_failures: true
    always_log_health_check_success: true
    event_log_path: /dev/stdout
    http_health_check:
      path: /healthz
  load_assignment:
    cluster_name: cluster_IPv4_80_TCP
    endpoints:
      - lb_endpoints:
        - endpoint:
            health_check_config:
              port_value: 10256
            address:
              socket_address:
                address: 172.18.0.4
                port_value: 30080
                protocol: TCP
      - lb_endpoints:
        - endpoint:
            health_check_config:
              port_value: 10256
            address:
              socket_address:
                address: 172.18.0.2
                port_value: 30080
                protocol: TCP
I0117 11:57:53.829877   27616 server.go:121] updating loadbalancer tunnels on userspace
I0117 11:57:53.935793   27616 tunnel.go:34] found port maps map[10000:60731 80:60807] associated to container kindccm-RZTQCN5REF5VESVEBSRFF5NKA2S54JVJFWGT3EOM
I0117 11:57:54.043799   27616 tunnel.go:41] setting IPv4 address 172.18.0.5 associated to container kindccm-RZTQCN5REF5VESVEBSRFF5NKA2S54JVJFWGT3EOM
E0117 11:57:54.118744   27616 controller.go:301] "Unhandled Error" err="error processing service default/nginx (retrying with exponential backoff): failed to ensure load balancer: exit status 1" logger="UnhandledError"
I0117 11:57:54.118744   27616 event.go:389] "Event occurred" object="default/nginx" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message="Error syncing load balancer: failed to ensure load balancer: exit status 1"
I0117 11:58:05.577074   27616 instances.go:47] Check instance metadata for dkj-cluster-control-plane
I0117 11:58:05.577074   27616 instances.go:47] Check instance metadata for dkj-cluster-worker
I0117 11:58:05.577074   27616 instances.go:47] Check instance metadata for dkj-cluster-worker2
I0117 11:58:05.842669   27616 instances.go:75] instance metadata for dkj-cluster-worker: &cloudprovider.InstanceMetadata{ProviderID:"kind://dkj-cluster/kind/dkj-cluster-worker", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"dkj-cluster-worker"}, v1.NodeAddress{Type:"InternalIP", Address:"172.18.0.2"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::2"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I0117 11:58:05.842669   27616 instances.go:75] instance metadata for dkj-cluster-control-plane: &cloudprovider.InstanceMetadata{ProviderID:"kind://dkj-cluster/kind/dkj-cluster-control-plane", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"dkj-cluster-control-plane"}, v1.NodeAddress{Type:"InternalIP", Address:"172.18.0.3"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::3"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I0117 11:58:05.859045   27616 instances.go:75] instance metadata for dkj-cluster-worker2: &cloudprovider.InstanceMetadata{ProviderID:"kind://dkj-cluster/kind/dkj-cluster-worker2", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"dkj-cluster-worker2"}, v1.NodeAddress{Type:"InternalIP", Address:"172.18.0.4"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::4"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I0117 11:58:05.866112   27616 node_controller.go:271] Update 3 nodes status took 289.0375ms.
I0117 11:58:14.120645   27616 controller.go:401] Ensuring load balancer for service default/nginx
I0117 11:58:14.120645   27616 loadbalancer.go:28] Ensure LoadBalancer cluster: dkj-cluster service: nginx
I0117 11:58:14.120645   27616 event.go:389] "Event occurred" object="default/nginx" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"
I0117 11:58:14.365976   27616 server.go:113] updating loadbalancer
I0117 11:58:14.365976   27616 proxy.go:236] address type Hostname, only InternalIP supported
I0117 11:58:14.366520   27616 proxy.go:236] address type Hostname, only InternalIP supported
I0117 11:58:14.366520   27616 proxy.go:271] envoy config info: &{HealthCheckPort:10256 ServicePorts:map[IPv4_80_TCP:{Listener:{Address:0.0.0.0 Port:80 Protocol:TCP} Cluster:[{Address:172.18.0.2 Port:30080 Protocol:TCP} {Address:172.18.0.4 Port:30080 Protocol:TCP}]}] SessionAffinity:None SourceRanges:[]}
I0117 11:58:14.366520   27616 proxy.go:289] updating loadbalancer with config
resources:
- "@type": type.googleapis.com/envoy.config.listener.v3.Listener
  name: listener_IPv4_80_TCP
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 80
      protocol: TCP
  filter_chains:
  - filters:
    - name: envoy.filters.network.tcp_proxy
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
        access_log:
        - name: envoy.file_access_log
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
        stat_prefix: tcp_proxy
        cluster: cluster_IPv4_80_TCP
I0117 11:58:14.518995   27616 proxy.go:300] updating loadbalancer with config
resources:
- "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster
  name: cluster_IPv4_80_TCP
  connect_timeout: 5s
  type: STATIC
  lb_policy: RANDOM
  health_checks:
  - timeout: 5s
    interval: 3s
    unhealthy_threshold: 2
    healthy_threshold: 1
    no_traffic_interval: 5s
    always_log_health_check_failures: true
    always_log_health_check_success: true
    event_log_path: /dev/stdout
    http_health_check:
      path: /healthz
  load_assignment:
    cluster_name: cluster_IPv4_80_TCP
    endpoints:
      - lb_endpoints:
        - endpoint:
            health_check_config:
              port_value: 10256
            address:
              socket_address:
                address: 172.18.0.2
                port_value: 30080
                protocol: TCP
      - lb_endpoints:
        - endpoint:
            health_check_config:
              port_value: 10256
            address:
              socket_address:
                address: 172.18.0.4
                port_value: 30080
                protocol: TCP
I0117 11:58:14.932241   27616 server.go:121] updating loadbalancer tunnels on userspace
I0117 11:58:15.055090   27616 tunnel.go:34] found port maps map[10000:60731 80:60807] associated to container kindccm-RZTQCN5REF5VESVEBSRFF5NKA2S54JVJFWGT3EOM
I0117 11:58:15.162064   27616 tunnel.go:41] setting IPv4 address 172.18.0.5 associated to container kindccm-RZTQCN5REF5VESVEBSRFF5NKA2S54JVJFWGT3EOM
E0117 11:58:15.235060   27616 controller.go:301] "Unhandled Error" err="error processing service default/nginx (retrying with exponential backoff): failed to ensure load balancer: exit status 1" logger="UnhandledError"
I0117 11:58:15.235060   27616 event.go:389] "Event occurred" object="default/nginx" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message="Error syncing load balancer: failed to ensure load balancer: exit status 1"
I0117 11:58:35.866293   27616 instances.go:47] Check instance metadata for dkj-cluster-worker
I0117 11:58:35.866293   27616 instances.go:47] Check instance metadata for dkj-cluster-control-plane
I0117 11:58:35.866293   27616 instances.go:47] Check instance metadata for dkj-cluster-worker2
I0117 11:58:36.138623   27616 instances.go:75] instance metadata for dkj-cluster-worker: &cloudprovider.InstanceMetadata{ProviderID:"kind://dkj-cluster/kind/dkj-cluster-worker", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"dkj-cluster-worker"}, v1.NodeAddress{Type:"InternalIP", Address:"172.18.0.2"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::2"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I0117 11:58:36.152466   27616 instances.go:75] instance metadata for dkj-cluster-control-plane: &cloudprovider.InstanceMetadata{ProviderID:"kind://dkj-cluster/kind/dkj-cluster-control-plane", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"dkj-cluster-control-plane"}, v1.NodeAddress{Type:"InternalIP", Address:"172.18.0.3"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::3"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I0117 11:58:36.169250   27616 instances.go:75] instance metadata for dkj-cluster-worker2: &cloudprovider.InstanceMetadata{ProviderID:"kind://dkj-cluster/kind/dkj-cluster-worker2", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"dkj-cluster-worker2"}, v1.NodeAddress{Type:"InternalIP", Address:"172.18.0.4"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::4"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I0117 11:58:36.175667   27616 node_controller.go:271] Update 3 nodes status took 309.3734ms.

the log of kindccm is only continuously outputting

2025-01-17 12:00:16 [2025-01-17 04:00:16.871][11][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.core.v3.HealthCheck Using deprecated option 'envoy.config.core.v3.HealthCheck.event_log_path' from file health_check.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
2025-01-17 12:00:16 [2025-01-17 04:00:16.871][11][info][upstream] [source/common/upstream/cds_api_helper.cc:32] cds: add 1 cluster(s), remove 0 cluster(s)
2025-01-17 12:00:16 [2025-01-17 04:00:16.872][11][info][upstream] [source/common/upstream/cds_api_helper.cc:71] cds: added/updated 1 cluster(s), skipped 0 unmodified cluster(s)
2025-01-17 12:00:26 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","add_healthy_event":{"first_check":true},"timestamp":"2025-01-17T04:00:16.873Z","locality":{"region":"","zone":"","sub_zone":""}}
2025-01-17 12:00:26 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","add_healthy_event":{"first_check":true},"timestamp":"2025-01-17T04:00:16.873Z","locality":{"region":"","zone":"","sub_zone":""}}
2025-01-17 12:00:26 [2025-01-17T04:00:17.003Z] "GET /ready HTTP/1.1" 200 - 0 5 0 - "172.18.0.1" "Go-http-client/1.1" "-" "127.0.0.1:60731" "-"
2025-01-17 12:00:26 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:00:21.871Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:00:26 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:00:21.871Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:00:36 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:00:26.870Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:00:36 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:00:26.870Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:00:36 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:00:31.869Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:00:36 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:00:31.869Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:00:46 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:00:36.869Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:00:46 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:00:36.869Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:00:46 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:00:41.867Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:00:46 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:00:41.867Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:00:56 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:00:46.866Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:00:56 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:00:46.866Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:00:56 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:00:51.864Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:00:56 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:00:51.864Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:01:06 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:00:56.873Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:01:06 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:00:56.873Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:01:06 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:01:01.872Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:01:06 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:01:01.872Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:01:16 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:01:06.874Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:01:16 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:01:06.874Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:01:16 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:01:11.883Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:01:16 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:01:11.883Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:01:26 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:01:16.882Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:01:26 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:01:16.882Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:01:26 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:01:21.880Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:01:26 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:01:21.880Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:01:36 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:01:26.879Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:01:36 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:01:26.879Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:01:36 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:01:31.877Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:01:36 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:01:31.877Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:01:46 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:01:36.883Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:01:46 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:01:36.883Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:01:46 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:01:41.892Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:01:46 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:01:41.892Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:01:56 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:01:46.901Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:01:56 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:01:46.901Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:01:56 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:01:51.894Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:01:56 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:01:51.894Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:02:06 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:01:56.893Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:02:06 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:01:56.893Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:02:06 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:02:01.902Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:02:06 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:02:01.902Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:02:16 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:02:06.900Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:02:16 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:02:06.911Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:02:16 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:02:11.900Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:02:16 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:02:11.921Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:02:26 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:02:16.900Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:02:26 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:02:16.920Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:02:26 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:02:21.900Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:02:26 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:02:21.920Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:02:36 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:02:26.899Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:02:36 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:02:26.920Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:02:36 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:02:31.898Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:02:36 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:02:31.919Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:02:46 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:02:36.901Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:02:46 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:02:36.920Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:02:46 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:02:41.909Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:02:46 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:02:41.930Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:02:56 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:02:46.909Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:02:56 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:02:46.940Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:02:56 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.2","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:02:51.909Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}
2025-01-17 12:02:56 {"health_checker_type":"HTTP","host":{"socket_address":{"protocol":"TCP","address":"172.18.0.4","port_value":30080,"resolver_name":"","ipv4_compat":false}},"cluster_name":"cluster_IPv4_80_TCP","timestamp":"2025-01-17T04:02:51.939Z","locality":{"region":"","zone":"","sub_zone":""},"successful_health_check_event":{}}

@d2461795341
Copy link
Author

d2461795341 commented Jan 17, 2025 •
edited
Loading

At first, I suspected that the issue was related to the mirrored network configuration in WSL. However, I completely retried the process on Windows, creating the cluster with Kind, setting up the service resource, and starting cloud-provider-kind directly on Windows. Surprisingly, I encountered the same problem. This is quite weird.

@aojea
Copy link
Contributor

aojea commented Jan 17, 2025 •
edited
Loading

the key is in understanding what this error means

E0117 11:58:15.235060 27616 controller.go:301] "Unhandled Error" err="error processing service default/nginx (retrying with exponential backoff): failed to ensure load balancer: exit status 1" logger="UnhandledError"
I0117 11:58:15.235060 27616 event.go:389] "Event occurred" object="default/nginx" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message="Error syncing load balancer: failed to ensure load balancer: exit status 1"

are you running cloud provider kind with verbosity 7?

if not, try with that and paste the logs around that log error I pasted above

@d2461795341
Copy link
Author

thank you for the tip, I try to run cloud-provider-kind.exe -v 7 in windows, and I get

I0117 14:45:46.343022   32108 server.go:121] updating loadbalancer tunnels on userspace
I0117 14:45:46.449636   32108 tunnel.go:34] found port maps map[10000:60825 80:60749] associated to container kindccm-RZTQCN5REF5VESVEBSRFF5NKA2S54JVJFWGT3EOM
I0117 14:45:46.556796   32108 tunnel.go:41] setting IPv4 address 172.18.0.5 associated to container kindccm-RZTQCN5REF5VESVEBSRFF5NKA2S54JVJFWGT3EOM
I0117 14:45:46.625413   32108 controller.go:876] Finished syncing service "default/nginx" (1.0437726s)
I0117 14:45:46.625413   32108 event.go:389] "Event occurred" object="default/nginx" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message="Error syncing load balancer: failed to ensure load balancer: exit status 1"
E0117 14:45:46.625413   32108 controller.go:301] "Unhandled Error" err="error processing service default/nginx (retrying with exponential backoff): failed to ensure load balancer: exit status 1" logger="UnhandledError"
I0117 14:45:46.625413   32108 round_trippers.go:470] PATCH https://127.0.0.1:13806/api/v1/namespaces/default/events/nginx.181b67e39306955c
I0117 14:45:46.625413   32108 round_trippers.go:476] Request Headers:
I0117 14:45:46.625413   32108 round_trippers.go:480]     Content-Type: application/strategic-merge-patch+json
I0117 14:45:46.625413   32108 round_trippers.go:480]     Accept: application/json, */*
I0117 14:45:46.625413   32108 round_trippers.go:480]     User-Agent: cloud-provider-kind.exe/v0.0.0 (windows/amd64) kubernetes/$Format
I0117 14:45:46.629810   32108 round_trippers.go:581] Response Status: 200 OK in 4 milliseconds
I0117 14:46:06.537343   32108 reflector.go:397] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:251: forcing resync
I0117 14:46:06.537343   32108 reflector.go:397] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:251: forcing resync
I0117 14:46:06.765665   32108 controller.go:70] processing cluster dkj-cluster
I0117 14:46:06.765665   32108 controller.go:73] cluster dkj-cluster already exist
I0117 14:46:07.262806   32108 instances.go:47] Check instance metadata for dkj-cluster-worker
I0117 14:46:07.262806   32108 instances.go:47] Check instance metadata for dkj-cluster-control-plane
I0117 14:46:07.262806   32108 instances.go:47] Check instance metadata for dkj-cluster-worker2
I0117 14:46:07.508279   32108 instances.go:75] instance metadata for dkj-cluster-worker2: &cloudprovider.InstanceMetadata{ProviderID:"kind://dkj-cluster/kind/dkj-cluster-worker2", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"dkj-cluster-worker2"}, v1.NodeAddress{Type:"InternalIP", Address:"172.18.0.3"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::3"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I0117 14:46:07.508828   32108 round_trippers.go:470] PATCH https://127.0.0.1:13806/api/v1/nodes/dkj-cluster-worker2/status
I0117 14:46:07.508828   32108 round_trippers.go:476] Request Headers:
I0117 14:46:07.508828   32108 round_trippers.go:480]     Accept: application/vnd.kubernetes.protobuf,application/json
I0117 14:46:07.508828   32108 round_trippers.go:480]     Content-Type: application/strategic-merge-patch+json
I0117 14:46:07.508828   32108 round_trippers.go:480]     User-Agent: cloud-provider-kind.exe/v0.0.0 (windows/amd64) kubernetes/$Format
I0117 14:46:07.514646   32108 round_trippers.go:581] Response Status: 200 OK in 5 milliseconds
I0117 14:46:07.523849   32108 instances.go:75] instance metadata for dkj-cluster-worker: &cloudprovider.InstanceMetadata{ProviderID:"kind://dkj-cluster/kind/dkj-cluster-worker", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"dkj-cluster-worker"}, v1.NodeAddress{Type:"InternalIP", Address:"172.18.0.2"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::2"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I0117 14:46:07.524354   32108 round_trippers.go:470] PATCH https://127.0.0.1:13806/api/v1/nodes/dkj-cluster-worker/status
I0117 14:46:07.524354   32108 round_trippers.go:476] Request Headers:
I0117 14:46:07.524354   32108 round_trippers.go:480]     Accept: application/vnd.kubernetes.protobuf,application/json
I0117 14:46:07.524354   32108 round_trippers.go:480]     Content-Type: application/strategic-merge-patch+json
I0117 14:46:07.524354   32108 round_trippers.go:480]     User-Agent: cloud-provider-kind.exe/v0.0.0 (windows/amd64) kubernetes/$Format
I0117 14:46:07.530707   32108 round_trippers.go:581] Response Status: 200 OK in 6 milliseconds
I0117 14:46:07.538902   32108 instances.go:75] instance metadata for dkj-cluster-control-plane: &cloudprovider.InstanceMetadata{ProviderID:"kind://dkj-cluster/kind/dkj-cluster-control-plane", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"dkj-cluster-control-plane"}, v1.NodeAddress{Type:"InternalIP", Address:"172.18.0.4"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::4"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I0117 14:46:07.539416   32108 round_trippers.go:470] PATCH https://127.0.0.1:13806/api/v1/nodes/dkj-cluster-control-plane/status
I0117 14:46:07.539416   32108 round_trippers.go:476] Request Headers:
I0117 14:46:07.539943   32108 round_trippers.go:480]     Accept: application/vnd.kubernetes.protobuf,application/json
I0117 14:46:07.539943   32108 round_trippers.go:480]     Content-Type: application/strategic-merge-patch+json
I0117 14:46:07.539943   32108 round_trippers.go:480]     User-Agent: cloud-provider-kind.exe/v0.0.0 (windows/amd64) kubernetes/$Format
I0117 14:46:07.545741   32108 round_trippers.go:581] Response Status: 200 OK in 5 milliseconds
I0117 14:46:07.546787   32108 node_controller.go:271] Update 3 nodes status took 283.9802ms.
I0117 14:46:26.627123   32108 controller.go:401] Ensuring load balancer for service default/nginx
I0117 14:46:26.627123   32108 loadbalancer.go:28] Ensure LoadBalancer cluster: dkj-cluster service: nginx
I0117 14:46:26.627123   32108 round_trippers.go:470] PATCH https://127.0.0.1:13806/api/v1/namespaces/default/events/nginx.181b67e329a917b0
I0117 14:46:26.627123   32108 round_trippers.go:476] Request Headers:
I0117 14:46:26.627123   32108 round_trippers.go:480]     Accept: application/json, */*
I0117 14:46:26.627123   32108 round_trippers.go:480]     Content-Type: application/strategic-merge-patch+json
I0117 14:46:26.627123   32108 round_trippers.go:480]     User-Agent: cloud-provider-kind.exe/v0.0.0 (windows/amd64) kubernetes/$Format
I0117 14:46:26.627123   32108 event.go:389] "Event occurred" object="default/nginx" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"
I0117 14:46:26.634792   32108 round_trippers.go:581] Response Status: 200 OK in 7 milliseconds
I0117 14:46:26.850942   32108 server.go:113] updating loadbalancer
I0117 14:46:26.851473   32108 proxy.go:236] address type Hostname, only InternalIP supported
I0117 14:46:26.851473   32108 proxy.go:236] address type Hostname, only InternalIP supported
I0117 14:46:26.851473   32108 proxy.go:271] envoy config info: &{HealthCheckPort:10256 ServicePorts:map[IPv4_80_TCP:{Listener:{Address:0.0.0.0 Port:80 Protocol:TCP} Cluster:[{Address:172.18.0.2 Port:30080 Protocol:TCP} {Address:172.18.0.3 Port:30080 Protocol:TCP}]}] SessionAffinity:None SourceRanges:[]}
I0117 14:46:26.851473   32108 proxy.go:289] updating loadbalancer with config```

@aojea
Copy link
Contributor

aojea commented Jan 17, 2025

I0117 14:46:26.851473 32108 proxy.go:236] address type Hostname, only InternalIP supported

it seems is not able to obtain the ip of the container, try

docker inspect -f {{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}} kindccm-RZTQCN5REF5VESVEBSRFF5NKA2S54JVJFWGT3EOM , with kindccm-... the current container (we should replace such long string for something shorter)

@aojea aojea mentioned this issue Jan 17, 2025
Closed
@d2461795341
Copy link
Author

I get this content

docker inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" kindccm-RZTQCN5REF5VESVEBSRFF5NKA2S54JVJFWGT3EOM
172.18.0.5,fc00:f853:ccd:e793::5

@aojea
Copy link
Contributor

aojea commented Jan 17, 2025 •
edited
Loading

are you running cloud-provider-kind privileged?

try also runing with the flag --enable-lb-port-mapping=true

@d2461795341
Copy link
Author

d2461795341 commented Jan 17, 2025 •
edited
Loading

Thank you so much! I forgot to enable privileged mode in Windows and WSL2.

I am using Docker Desktop shared between WSL2 and Windows, and I’ve tried both NAT and mirrored WSL2 network configurations.

Notably:

  • In NAT mode, cloud-provider-kind needs to be started in privileged mode in WSL2. In this case, the external IP can only be accessed via curl from WSL2, and not from Windows. On Windows, the service can be accessed through localhost with the port kindccm exposing to host machine.
  • In mirrored mode: cloud-provider-kind must be started in privileged mode on either Windows or WSL2, along with the --enable-lb-port-mapping=true flag, for the LoadBalancer service to be assigned an external IP. However, the external IP at this point is not accessible—it cannot be accessed from either WSL2 or Windows. service can only be accessed through localhost with the port kindccm exposing to host machine.

Additionally, I also conducted tests on both macOS and Windows. In summary:

  • In WSL2, with NAT network configuration, the service can be accessed via the external IP of the LoadBalancer from WSL2, while on Windows, it can only be accessed via localhost using kindccm to expose the port to the host machine.
  • On macOS, simply starting cloud-provider-kind with privileged mode handles everything seamlessly.
  • The situation on Windows seems worse, as the service can only be accessed through localhost and the port exposed by kindccm to the host machine.

@d2461795341 d2461795341 mentioned this issue Jan 17, 2025
Open
@aojea
Copy link
Contributor

aojea commented Jan 20, 2025

It will be nice if you can update the README of the project with this "known issues" so others will benefit of this investigation.

@aojea aojea mentioned this issue Jan 20, 2025
Open
@aojea
Copy link
Contributor

aojea commented Jan 21, 2025

Is there anything left here? I'm not familiar with windows platforms so I can't say if this is WAI

@d2461795341
Copy link
Author

Sorry for the late reply. I was away for a few days due to some matters. I plan to try again in my Windows environment tonight. Also, I’d be happy to add the known issues to the README file.

@d2461795341 d2461795341 mentioned this issue Jan 24, 2025
Open
@dprotaso dprotaso mentioned this issue Jan 30, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@aojea @d2461795341