viz websocket close error

An error has occurred.
Websocket close error [1011: Internal Error] : HTTP error, status Code [503] (unexpected API response: service unavailable )

$ linkerd check
...
‼ tap API service is running
    FailedDiscoveryCheck: failing or missing response from https://<REDCATED>:8089/apis/tap.linkerd.io/v1alpha1: Get "https://<REDCATED>:8089/apis/tap.linkerd.io/v1alpha1": dial tcp <REDCATED>:8089: i/o timeout
    see https://linkerd.io/2.15/tasks/troubleshooting/#l5d-tap-api for hints

$ helm list -n linkerd-viz
NAME         NAMESPACE    REVISION  UPDATED                               STATUS    CHART                        APP VERSION
linkerd-viz  linkerd-viz  2         2024-05-21 11:55:07.180595 +0900 KST  deployed  linkerd-viz-0.0.0-undefined  edge-XX.X.X

# web deployment yaml
  - args:
    - -linkerd-metrics-api-addr=metrics-api.linkerd-viz.svc.cluster.local:8085
    - -cluster-domain=cluster.local
    - -controller-namespace=linkerd
    - -log-level=info
    - -log-format=plain
    - -enforced-host=^(localhost|127\.0\.0\.1|web\.linkerd-viz\.svc\.cluster\.local|web\.linkerd-viz\.svc|\[::1\])(:\d+)?$

# linkerd2/viz/charts/linkerd-viz/values.yaml
dashboard:
  # -- Host header validation regex for the dashboard. See the [Linkerd
  # documentation](https://linkerd.io/2/tasks/exposing-dashboard) for more
  # information
  enforcedHostRegexp: ""

module "eks" {
  # ... truncated ...
  node_security_group_additional_rules = {
    # Required for mutating webhook in vault-agent-injector, one of the components of vault.
    # https://github.com/hashicorp/vault-helm/issues/163
    ingress_vault_agent_injector_mutating_webhook = {
      description                   = "Allow ingress mutating webhook traffic from kube-apiserver to vault-agent-injector pod"
      protocol                      = "tcp"
      from_port                     = 8080
      to_port                       = 8080
      type                          = "ingress"
      source_cluster_security_group = true
    }
    ingress_linkerd_viz_tap_api = {
      description                   = "Allow ingress api calling traffic from kube-apiserver to linkerd-viz tap pod"
      protocol                      = "tcp"
      from_port                     = 8088
      to_port                       = 8089
      type                          = "ingress"
      source_cluster_security_group = true
    }
  }
  # ... truncated ...
}

# ingress-nginx/values.yaml
controller:
  # -- This configuration defines if Ingress Controller should allow users to set
  # their own *-snippet annotations, otherwise this is forbidden / dropped
  # when users add those annotations.
  # Global snippets in ConfigMap are still respected
  allowSnippetAnnotations: true

cat << EOF | kubectl apply -f -
---
# apiVersion: networking.k8s.io/v1beta1 # for k8s < v1.19
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web-ingress
  namespace: linkerd-viz
  annotations:
    nginx.ingress.kubernetes.io/upstream-vhost: \$service_name.\$namespace.svc.cluster.local:8084
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header Origin "";
      proxy_hide_header l5d-remote-ip;
      proxy_hide_header l5d-server-id;
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
spec:
  ingressClassName: nginx
  rules:
  - host: viz.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: web
            port:
              number: 8084
EOF

kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"

kubectl rollout restart deployment -n linkerd-viz

deployment.apps/metrics-api restarted
deployment.apps/prometheus restarted
deployment.apps/tap restarted
deployment.apps/tap-injector restarted
deployment.apps/web restarted

$ linkerd check
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ control plane pods are ready
√ cluster networks contains all pods
√ cluster networks contains all services

linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ proxy-init container runs as root user if docker container runtime is used

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust anchor

linkerd-webhooks-and-apisvc-tls
-------------------------------
√ proxy-injector webhook has valid cert
√ proxy-injector cert is valid for at least 60 days
√ sp-validator webhook has valid cert
√ sp-validator cert is valid for at least 60 days
√ policy-validator webhook has valid cert
√ policy-validator cert is valid for at least 60 days

linkerd-version
---------------
√ can determine the latest version
√ cli is up-to-date

control-plane-version
---------------------
√ can retrieve the control plane version
√ control plane is up-to-date
√ control plane and cli versions match

linkerd-control-plane-proxy
---------------------------
√ control plane proxies are healthy
√ control plane proxies are up-to-date
√ control plane proxies and cli versions match

linkerd-extension-checks
------------------------
√ namespace configuration for extensions

linkerd-viz
-----------
√ linkerd-viz Namespace exists
√ can initialize the client
√ linkerd-viz ClusterRoles exist
√ linkerd-viz ClusterRoleBindings exist
√ tap API server has valid cert
√ tap API server cert is valid for at least 60 days
√ tap API service is running
√ linkerd-viz pods are injected
√ viz extension pods are running
√ viz extension proxies are healthy
√ viz extension proxies are up-to-date
√ viz extension proxies and cli versions match
√ prometheus is installed and configured correctly
√ viz extension self-check

Status check results are √