Create OCNE Clusters Running on VMware vSphere
Configure Oracle Cloud Native Environment self-managed clusters to run on VMware vSphere
NOTE
This feature is experimental. It has not been thoroughly tested and is provided for informational purposes only. We make no guarantees about its safety or stability and do not recommend implementing this feature in production environments.The Cluster API project provides a standard set of Kubernetes-style APIs for cluster management. Officially, Verrazzano currently only supports using Cluster API to provision OCNE and OKE clusters on OCI.
However, you can also experiment with using the features of the Cluster API project directly to deploy OCNE clusters on VMware vSphere.
For more information on Cluster API or Cluster API with vSphere, see:
NOTE
Verrazzano and Cluster API use slightly different terminology for the same concepts:
- Admin cluster (Verrazzano) = management cluster (Cluster API)
- Managed cluster (Verrazzano) = workload cluster (Cluster API)
Before you begin
If you have an existing vSphere environment, you can ignore Set up a VMware Software-Defined Data Center and start from Prepare the VM environment. Confirm that your environment meets the requirements as specified at Cluster API Provider vSphere: Install Requirements.
Otherwise, create a vSphere environment. We recommend using the Oracle Cloud VMware Solution as described in Set up a VMware Software-Defined Data Center. It deploys a VMware software-defined data center (SDDC) on Oracle Cloud Infrastructure (OCI) and then integrates it with other Oracle services running on Oracle Cloud. This solution was developed in partnership with VMware to provide an environment that adheres to best practices recommended by VMware.
For more information on the Oracle Cloud VMware Solution, see Deploy a highly available VMware-based SDDC to the cloud in the Oracle Help Architecture Center.
Set up a VMware Software-Defined Data Center
NOTE
Skip this section if you have configured a vSphere environment or a VMware SDDC already.Use the Oracle Cloud VMware Solution to rapidly create a VMware SDDC.
-
Set up a virtual cloud network (VCN). You can choose to use an existing VCN or let the Oracle Cloud VMware Solution create its own VCN as part of the SDDC provisioning process. If you use an existing VCN, then make sure it meets the requirements defined in Prepare Your Deployment in the Oracle Help Architecture Center.
-
Deploy the SDDC. To request a new VMware SDDC on OCI, follow the instructions at Deploy the SDDC in the Oracle Help Architecture Center.
-
Ensure that the various components were created successfully. Follow the instructions at Monitor the SDDC Creation Process in the Oracle Help Architecture Center.
Prepare the VM environment
-
Download an Oracle Linux 8 ISO image from Oracle Linux Installation Media.
-
Upload the Oracle Linux 8 ISO image to vSphere. Use the steps at Upload ISO Image Installation Media for a Guest Operating System in the vSphere documentation.
-
Deploy a VM by following the instructions at Create a Virtual Machine with the New Virtual Machine Wizard in the vSphere documentation.
-
Install cloud-init on the VM.
$ sudo yum install -y cloud-init -
Initialize cloud-init.
When cloud-init is successfully configured, it returns a message similar to the following:$ cloud-init init --local$ cloud-init v. 20.1.0011 running 'init-local' at Fri, 01 Apr 2022 01:26:11 +0000. Up 38.70 seconds. -
Shut down the VM.
-
Convert the VM into a template and name it
OL8-Base-Template. Follow the instructions at Clone a Virtual Machine to a Template in the vSphere documentation.
Set up the admin cluster
The Cluster API requires an initial cluster as a starting point to deploy its resources.
-
Install kind. Follow the instructions at Installation in the kind documentation.
-
Create a Kubernetes cluster using kind. This cluster must be accessible by the VMware SDDC. Follow the instructions at Quick Start: Install and/or configure a Kubernetes cluster in The Cluster API Book.
-
Install the clusterctl CLI tool. clusterctl manages the lifecycle operations of a cluster API admin cluster. Follow instructions at Quick Start: Install clusterctl in the Cluster API Book.
-
Install the Verrazzano CLI tool using the instructions at CLI Setup.
-
Install Verrazzano on the cluster using either the
devorprodprofile. Follow the instructions at Install with CLI. -
On the cluster, set the following vSphere environment variables. Update the values to reflect your own environment.
For information on the values of the environment variables, see Configuring and installing Cluster API Provider vSphere in a management cluster in the Cluster API Provider vSphere documentation.$ export VSPHERE_PASSWORD="<vmware-password>" $ export VSPHERE_USERNAME="administrator@vsphere.local" $ export VSPHERE_SERVER="<IP address or FQDN>" $ export VSPHERE_DATACENTER="<SDDC-Datacenter>" $ export VSPHERE_DATASTORE="<vSAN-Datastore>" $ export VSPHERE_NETWORK="workload" $ export VSPHERE_RESOURCE_POOL="*/Resources/Workload" $ export VSPHERE_FOLDER="<folder-name>" $ export VSPHERE_TEMPLATE="OL8-Base-Template" $ export VSPHERE_SSH_AUTHORIZED_KEY="<Public-SSH-Authorized-Key>" $ export VSPHERE_TLS_THUMBPRINT="<SHA1 thumbprint of vCenter certificate>" $ export VSPHERE_STORAGE_POLICY="" $ export CONTROL_PLANE_ENDPOINT_IP="<IP address or FQDN>" -
Install the Cluster API Provider vSphere to initialize the admin cluster.
$ clusterctl init -n verrazzano-capi -i vsphereclusterctl will report when the admin cluster was successfully initialized.
Create a managed cluster
The Cluster API uses a cluster template to deploy a predefined set of Cluster API objects and create a managed cluster.
-
Copy the cluster template and save it locally as
vsphere-capi.yaml.Click here for the cluster template
apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster metadata: labels: cluster.x-k8s.io/cluster-name: ${CLUSTER_NAME} name: ${CLUSTER_NAME} namespace: ${NAMESPACE} spec: clusterNetwork: pods: cidrBlocks: - ${POD_CIDR=192.168.0.0/16} serviceDomain: cluster.local services: cidrBlocks: - ${CLUSTER_CIDR=10.128.0.0/12} controlPlaneRef: apiVersion: controlplane.cluster.x-k8s.io/v1alpha1 kind: OCNEControlPlane name: ${CLUSTER_NAME}-control-plane namespace: ${NAMESPACE} infrastructureRef: apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: VSphereCluster name: ${CLUSTER_NAME} namespace: ${NAMESPACE} --- apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: VSphereCluster metadata: name: ${CLUSTER_NAME} namespace: ${NAMESPACE} spec: controlPlaneEndpoint: host: ${CONTROL_PLANE_ENDPOINT_IP} port: 6443 identityRef: kind: Secret name: ${CLUSTER_NAME} server: ${VSPHERE_SERVER} thumbprint: '${VSPHERE_TLS_THUMBPRINT}' --- apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: VSphereMachineTemplate metadata: name: ${CLUSTER_NAME}-control-plane namespace: ${NAMESPACE} spec: template: spec: cloneMode: linkedClone datacenter: ${VSPHERE_DATACENTER=oci-w01dc} datastore: ${VSPHERE_DATASTORE=vsanDatastore} diskGiB: ${VSPHERE_DISK=200} folder: ${VSPHERE_FOLDER=CAPI} memoryMiB: ${VSPHERE_MEMORY=32384} network: devices: - dhcp4: true networkName: "${VSPHERE_NETWORK=workload}" numCPUs: ${VSPHERE_CPU=4} os: Linux resourcePool: '${VSPHERE_RESOURCE_POOL=*/Resources/Workload}' server: '${VSPHERE_SERVER=11.0.11.130}' storagePolicyName: ${VSPHERE_STORAGE_POLICY=""} template: ${VSPHERE_TEMPLATE=OL8-Base-Template} thumbprint: '${VSPHERE_TLS_THUMBPRINT}' --- apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: VSphereMachineTemplate metadata: name: ${CLUSTER_NAME}-md-0 namespace: ${NAMESPACE} spec: template: spec: cloneMode: linkedClone datacenter: ${VSPHERE_DATACENTER=oci-w01dc} datastore: ${VSPHERE_DATASTORE=vsanDatastore} diskGiB: ${VSPHERE_DISK=200} folder: ${VSPHERE_FOLDER=CAPI} memoryMiB: ${VSPHERE_MEMORY=32384} network: devices: - dhcp4: true networkName: "${VSPHERE_NETWORK=workload}" numCPUs: ${VSPHERE_CPU=4} os: Linux resourcePool: '${VSPHERE_RESOURCE_POOL=*/Resources/Workload}' server: '${VSPHERE_SERVER=11.0.11.130}' storagePolicyName: ${VSPHERE_STORAGE_POLICY=""} template: ${VSPHERE_TEMPLATE=OL8-Base-Template} thumbprint: '${VSPHERE_TLS_THUMBPRINT}' --- apiVersion: controlplane.cluster.x-k8s.io/v1alpha1 kind: OCNEControlPlane metadata: name: ${CLUSTER_NAME}-control-plane namespace: ${NAMESPACE} spec: moduleOperator: enabled: true verrazzanoPlatformOperator: enabled: true controlPlaneConfig: clusterConfiguration: apiServer: extraArgs: cloud-provider: external certSANs: - localhost - 127.0.0.1 dns: imageRepository: ${OCNE_IMAGE_REPOSITORY=container-registry.oracle.com}/${OCNE_IMAGE_PATH=olcne} imageTag: ${DNS_TAG=v1.9.3} etcd: local: imageRepository: ${OCNE_IMAGE_REPOSITORY=container-registry.oracle.com}/${OCNE_IMAGE_PATH=olcne} imageTag: ${ETCD_TAG=3.5.6} controllerManager: extraArgs: cloud-provider: external networking: {} scheduler: {} imageRepository: ${OCNE_IMAGE_REPOSITORY=container-registry.oracle.com}/${OCNE_IMAGE_PATH=olcne} files: - content: | apiVersion: v1 kind: Pod metadata: creationTimestamp: null name: kube-vip namespace: kube-system spec: containers: - args: - manager env: - name: cp_enable value: "true" - name: vip_interface value: "" - name: address value: ${CONTROL_PLANE_ENDPOINT_IP} - name: port value: "6443" - name: vip_arp value: "true" - name: vip_leaderelection value: "true" - name: vip_leaseduration value: "15" - name: vip_renewdeadline value: "10" - name: vip_retryperiod value: "2" image: ghcr.io/kube-vip/kube-vip:v0.5.11 imagePullPolicy: IfNotPresent name: kube-vip resources: {} securityContext: capabilities: add: - NET_ADMIN - NET_RAW volumeMounts: - mountPath: /etc/kubernetes/admin.conf name: kubeconfig hostAliases: - hostnames: - kubernetes ip: 127.0.0.1 hostNetwork: true volumes: - hostPath: path: /etc/kubernetes/admin.conf type: FileOrCreate name: kubeconfig status: {} owner: root:root path: /etc/kubernetes/manifests/kube-vip.yaml initConfiguration: nodeRegistration: criSocket: /var/run/crio/crio.sock kubeletExtraArgs: cloud-provider: external name: '{{ local_hostname }}' joinConfiguration: discovery: {} nodeRegistration: criSocket: /var/run/crio/crio.sock kubeletExtraArgs: cloud-provider: external name: '{{ local_hostname }}' verbosity: 9 preOCNECommands: - hostnamectl set-hostname "{{ ds.meta_data.hostname }}" - echo "::1 ipv6-localhost ipv6-loopback localhost6 localhost6.localdomain6" >/etc/hosts - echo "127.0.0.1 {{ ds.meta_data.hostname }} {{ local_hostname }} localhost localhost.localdomain localhost4 localhost4.localdomain4" >>/etc/hosts users: - name: opc sshAuthorizedKeys: - ${VSPHERE_SSH_AUTHORIZED_KEY} sudo: ALL=(ALL) NOPASSWD:ALL machineTemplate: infrastructureRef: apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: VSphereMachineTemplate name: ${CLUSTER_NAME}-control-plane namespace: ${NAMESPACE} replicas: ${CONTROL_PLANE_MACHINE_COUNT=1} version: ${KUBERNETES_VERSION=v1.26.6} --- apiVersion: bootstrap.cluster.x-k8s.io/v1alpha1 kind: OCNEConfigTemplate metadata: name: ${CLUSTER_NAME}-md-0 namespace: ${NAMESPACE} spec: template: spec: clusterConfiguration: imageRepository: ${OCNE_IMAGE_REPOSITORY=container-registry.oracle.com}/${OCNE_IMAGE_PATH=olcne} joinConfiguration: nodeRegistration: kubeletExtraArgs: cloud-provider: external name: '{{ local_hostname }}' verbosity: 9 preOCNECommands: - hostnamectl set-hostname "{{ ds.meta_data.hostname }}" - echo "::1 ipv6-localhost ipv6-loopback localhost6 localhost6.localdomain6" >/etc/hosts - echo "127.0.0.1 {{ ds.meta_data.hostname }} {{ local_hostname }} localhost localhost.localdomain localhost4 localhost4.localdomain4" >>/etc/hosts users: - name: opc sshAuthorizedKeys: - ${VSPHERE_SSH_AUTHORIZED_KEY} sudo: ALL=(ALL) NOPASSWD:ALL --- apiVersion: cluster.x-k8s.io/v1beta1 kind: MachineDeployment metadata: labels: cluster.x-k8s.io/cluster-name: ${CLUSTER_NAME} name: ${CLUSTER_NAME}-md-0 namespace: ${NAMESPACE} spec: clusterName: ${CLUSTER_NAME} replicas: ${NODE_MACHINE_COUNT=3} selector: matchLabels: {} template: metadata: labels: cluster.x-k8s.io/cluster-name: ${CLUSTER_NAME} spec: bootstrap: configRef: apiVersion: bootstrap.cluster.x-k8s.io/v1alpha1 kind: OCNEConfigTemplate name: ${CLUSTER_NAME}-md-0 clusterName: ${CLUSTER_NAME} infrastructureRef: apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: VSphereMachineTemplate name: ${CLUSTER_NAME}-md-0 version: ${KUBERNETES_VERSION=v1.26.6} --- apiVersion: addons.cluster.x-k8s.io/v1beta1 kind: ClusterResourceSet metadata: name: ${CLUSTER_NAME}-crs-0 namespace: ${NAMESPACE} spec: clusterSelector: matchLabels: cluster.x-k8s.io/cluster-name: ${CLUSTER_NAME} resources: - kind: Secret name: ${CLUSTER_NAME}-vsphere-csi-controller - kind: ConfigMap name: ${CLUSTER_NAME}-vsphere-csi-controller-role - kind: ConfigMap name: ${CLUSTER_NAME}-vsphere-csi-controller-binding - kind: Secret name: ${CLUSTER_NAME}-csi-vsphere-config - kind: ConfigMap name: csi.vsphere.vmware.com - kind: ConfigMap name: vsphere-csi-controller-sa - kind: ConfigMap name: vsphere-csi-node-sa - kind: ConfigMap name: ${CLUSTER_NAME}-vsphere-csi-node-cluster-role - kind: ConfigMap name: ${CLUSTER_NAME}-vsphere-csi-node-cluster-role-binding - kind: ConfigMap name: ${CLUSTER_NAME}-vsphere-csi-node-role - kind: ConfigMap name: ${CLUSTER_NAME}-vsphere-csi-node-binding - kind: ConfigMap name: ${CLUSTER_NAME}-internal-feature-states.csi.vsphere.vmware.com - kind: ConfigMap name: ${CLUSTER_NAME}-vsphere-csi-controller-service - kind: ConfigMap name: ${CLUSTER_NAME}-vsphere-csi-controller - kind: ConfigMap name: ${CLUSTER_NAME}-vsphere-csi-node - kind: ConfigMap name: ${CLUSTER_NAME}-vsphere-csi-node-windows - kind: Secret name: ${CLUSTER_NAME}-cloud-controller-manager - kind: Secret name: ${CLUSTER_NAME}-cloud-provider-vsphere-credentials - kind: ConfigMap name: ${CLUSTER_NAME}-cpi-manifests strategy: Reconcile --- apiVersion: v1 kind: Secret metadata: name: ${CLUSTER_NAME} namespace: ${NAMESPACE} stringData: password: ${VSPHERE_PASSWORD} username: ${VSPHERE_USERNAME} --- apiVersion: v1 kind: ConfigMap metadata: name: ${CLUSTER_NAME}-vsphere-csi-controller namespace: ${NAMESPACE} data: data: | apiVersion: v1 kind: ServiceAccount metadata: name: vsphere-csi-controller namespace: kube-system --- apiVersion: v1 data: data: | apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: vsphere-csi-controller-role rules: - apiGroups: [""] resources: ["nodes", "pods", "configmaps"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch", "update"] - apiGroups: [""] resources: ["persistentvolumeclaims/status"] verbs: ["patch"] - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "create", "update", "delete", "patch"] - apiGroups: [""] resources: ["events"] verbs: ["get", "list", "watch", "create", "update", "patch"] - apiGroups: ["coordination.k8s.io"] resources: ["leases"] verbs: ["get", "watch", "list", "delete", "update", "create"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses", "csinodes"] verbs: ["get", "list", "watch"] - apiGroups: ["storage.k8s.io"] resources: ["volumeattachments"] verbs: ["get", "list", "watch", "patch"] - apiGroups: ["cns.vmware.com"] resources: ["triggercsifullsyncs"] verbs: ["create", "get", "update", "watch", "list"] - apiGroups: ["cns.vmware.com"] resources: ["cnsvspherevolumemigrations"] verbs: ["create", "get", "list", "watch", "update", "delete"] - apiGroups: ["apiextensions.k8s.io"] resources: ["customresourcedefinitions"] verbs: ["get", "create", "update"] - apiGroups: ["storage.k8s.io"] resources: ["volumeattachments/status"] verbs: ["patch"] - apiGroups: ["cns.vmware.com"] resources: ["cnsvolumeoperationrequests"] verbs: ["create", "get", "list", "update", "delete"] - apiGroups: [ "snapshot.storage.k8s.io" ] resources: [ "volumesnapshots" ] verbs: [ "get", "list" ] - apiGroups: [ "snapshot.storage.k8s.io" ] resources: [ "volumesnapshotclasses" ] verbs: [ "watch", "get", "list" ] - apiGroups: [ "snapshot.storage.k8s.io" ] resources: [ "volumesnapshotcontents" ] verbs: [ "create", "get", "list", "watch", "update", "delete", "patch"] - apiGroups: [ "snapshot.storage.k8s.io" ] resources: [ "volumesnapshotcontents/status" ] verbs: [ "update", "patch" ] - apiGroups: [ "cns.vmware.com" ] resources: [ "csinodetopologies" ] verbs: ["get", "update", "watch", "list"] kind: ConfigMap metadata: name: ${CLUSTER_NAME}-vsphere-csi-controller-role namespace: ${NAMESPACE} --- apiVersion: v1 data: data: | apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: vsphere-csi-controller-binding roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: vsphere-csi-controller-role subjects: - kind: ServiceAccount name: vsphere-csi-controller namespace: kube-system kind: ConfigMap metadata: name: ${CLUSTER_NAME}-vsphere-csi-controller-binding namespace: ${NAMESPACE} --- apiVersion: v1 kind: Secret metadata: name: ${CLUSTER_NAME}-csi-vsphere-config namespace: ${NAMESPACE} stringData: data: | apiVersion: v1 kind: Secret metadata: name: csi-vsphere-config namespace: kube-system stringData: csi-vsphere.conf: |+ [Global] thumbprint = "${VSPHERE_TLS_THUMBPRINT}" cluster-id = "${NAMESPACE}/${CLUSTER_NAME}" [VirtualCenter "${VSPHERE_SERVER}"] insecure-flag = "true" user = "${VSPHERE_USERNAME}" password = "${VSPHERE_PASSWORD}" datacenters = "${VSPHERE_DATACENTER}" targetvSANFileShareDatastoreURLs = "${VSPHERE_DATASTORE_URL_SAN}" [Network] public-network = "${VSPHERE_NETWORK=workload}" type: Opaque type: addons.cluster.x-k8s.io/resource-set --- apiVersion: v1 data: data: | apiVersion: storage.k8s.io/v1 kind: CSIDriver metadata: name: csi.vsphere.vmware.com spec: attachRequired: true podInfoOnMount: false kind: ConfigMap metadata: name: csi.vsphere.vmware.com namespace: ${NAMESPACE} --- apiVersion: v1 data: data: | kind: ServiceAccount apiVersion: v1 metadata: name: vsphere-csi-controller namespace: kube-system kind: ConfigMap metadata: name: vsphere-csi-controller-sa namespace: ${NAMESPACE} --- apiVersion: v1 data: data: | kind: ServiceAccount apiVersion: v1 metadata: name: vsphere-csi-node namespace: kube-system kind: ConfigMap metadata: name: vsphere-csi-node-sa namespace: ${NAMESPACE} --- apiVersion: v1 data: data: | apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: vsphere-csi-node-cluster-role rules: - apiGroups: ["cns.vmware.com"] resources: ["csinodetopologies"] verbs: ["create", "watch", "get", "patch"] - apiGroups: [""] resources: ["nodes"] verbs: ["get"] kind: ConfigMap metadata: name: ${CLUSTER_NAME}-vsphere-csi-node-cluster-role namespace: ${NAMESPACE} --- apiVersion: v1 data: data: | kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: vsphere-csi-node-cluster-role-binding subjects: - kind: ServiceAccount name: vsphere-csi-node namespace: kube-system roleRef: kind: ClusterRole name: vsphere-csi-node-cluster-role apiGroup: rbac.authorization.k8s.io kind: ConfigMap metadata: name: ${CLUSTER_NAME}-vsphere-csi-node-cluster-role-binding namespace: ${NAMESPACE} --- apiVersion: v1 data: data: | kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: vsphere-csi-node-role namespace: kube-system rules: - apiGroups: [""] resources: ["configmaps"] verbs: ["get", "list", "watch"] kind: ConfigMap metadata: name: ${CLUSTER_NAME}-vsphere-csi-node-role namespace: ${NAMESPACE} --- apiVersion: v1 data: data: | kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: vsphere-csi-node-binding namespace: kube-system subjects: - kind: ServiceAccount name: vsphere-csi-node namespace: kube-system roleRef: kind: Role name: vsphere-csi-node-role apiGroup: rbac.authorization.k8s.io kind: ConfigMap metadata: name: ${CLUSTER_NAME}-vsphere-csi-node-binding namespace: ${NAMESPACE} --- apiVersion: v1 data: data: | apiVersion: v1 data: "csi-migration": "true" "csi-auth-check": "true" "online-volume-extend": "true" "trigger-csi-fullsync": "false" "async-query-volume": "true" "improved-csi-idempotency": "true" "improved-volume-topology": "true" "block-volume-snapshot": "true" "csi-windows-support": "false" "use-csinode-id": "true" "list-volumes": "false" "pv-to-backingdiskobjectid-mapping": "false" "cnsmgr-suspend-create-volume": "true" "topology-preferential-datastores": "true" "max-pvscsi-targets-per-vm": "true" kind: ConfigMap metadata: name: internal-feature-states.csi.vsphere.vmware.com namespace: kube-system kind: ConfigMap metadata: name: ${CLUSTER_NAME}-internal-feature-states.csi.vsphere.vmware.com namespace: ${NAMESPACE} --- apiVersion: v1 data: data: | apiVersion: v1 kind: Service metadata: name: vsphere-csi-controller namespace: kube-system labels: app: vsphere-csi-controller spec: ports: - name: ctlr port: 2112 targetPort: 2112 protocol: TCP - name: syncer port: 2113 targetPort: 2113 protocol: TCP selector: app: vsphere-csi-controller kind: ConfigMap metadata: name: ${CLUSTER_NAME}-vsphere-csi-controller-service namespace: ${NAMESPACE} --- apiVersion: v1 data: data: | kind: Deployment apiVersion: apps/v1 metadata: name: vsphere-csi-controller namespace: kube-system spec: replicas: 1 strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 maxSurge: 0 selector: matchLabels: app: vsphere-csi-controller template: metadata: labels: app: vsphere-csi-controller role: vsphere-csi spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: "app" operator: In values: - vsphere-csi-controller topologyKey: "kubernetes.io/hostname" serviceAccountName: vsphere-csi-controller nodeSelector: node-role.kubernetes.io/control-plane: "" tolerations: - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule - key: node-role.kubernetes.io/control-plane operator: Exists effect: NoSchedule # uncomment below toleration if you need an aggressive pod eviction in case when # node becomes not-ready or unreachable. Default is 300 seconds if not specified. #- key: node.kubernetes.io/not-ready # operator: Exists # effect: NoExecute # tolerationSeconds: 30 #- key: node.kubernetes.io/unreachable # operator: Exists # effect: NoExecute # tolerationSeconds: 30 dnsPolicy: "Default" containers: - name: csi-attacher image: k8s.gcr.io/sig-storage/csi-attacher:v3.5.0 args: - "--v=4" - "--timeout=300s" - "--csi-address=$(ADDRESS)" - "--leader-election" - "--kube-api-qps=100" - "--kube-api-burst=100" env: - name: ADDRESS value: /csi/csi.sock volumeMounts: - mountPath: /csi name: socket-dir - name: csi-resizer image: k8s.gcr.io/sig-storage/csi-resizer:v1.5.0 args: - "--v=4" - "--timeout=300s" - "--handle-volume-inuse-error=false" - "--csi-address=$(ADDRESS)" - "--kube-api-qps=100" - "--kube-api-burst=100" - "--leader-election" env: - name: ADDRESS value: /csi/csi.sock volumeMounts: - mountPath: /csi name: socket-dir - name: vsphere-csi-controller image: gcr.io/cloud-provider-vsphere/csi/release/driver:v2.7.0 args: - "--fss-name=internal-feature-states.csi.vsphere.vmware.com" - "--fss-namespace=$(CSI_NAMESPACE)" imagePullPolicy: "Always" env: - name: CSI_ENDPOINT value: unix:///csi/csi.sock - name: X_CSI_MODE value: "controller" - name: X_CSI_SPEC_DISABLE_LEN_CHECK value: "true" - name: X_CSI_SERIAL_VOL_ACCESS_TIMEOUT value: 3m - name: VSPHERE_CSI_CONFIG value: "/etc/cloud/csi-vsphere.conf" - name: LOGGER_LEVEL value: "PRODUCTION" # Options: DEVELOPMENT, PRODUCTION - name: INCLUSTER_CLIENT_QPS value: "100" - name: INCLUSTER_CLIENT_BURST value: "100" - name: CSI_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace volumeMounts: - mountPath: /etc/cloud name: vsphere-config-volume readOnly: true - mountPath: /csi name: socket-dir ports: - name: healthz containerPort: 9808 protocol: TCP - name: prometheus containerPort: 2112 protocol: TCP livenessProbe: httpGet: path: /healthz port: healthz initialDelaySeconds: 10 timeoutSeconds: 3 periodSeconds: 5 failureThreshold: 3 - name: liveness-probe image: k8s.gcr.io/sig-storage/livenessprobe:v2.7.0 args: - "--v=4" - "--csi-address=/csi/csi.sock" volumeMounts: - name: socket-dir mountPath: /csi - name: vsphere-syncer image: gcr.io/cloud-provider-vsphere/csi/release/syncer:v2.7.0 args: - "--leader-election" - "--fss-name=internal-feature-states.csi.vsphere.vmware.com" - "--fss-namespace=$(CSI_NAMESPACE)" imagePullPolicy: "Always" ports: - containerPort: 2113 name: prometheus protocol: TCP env: - name: FULL_SYNC_INTERVAL_MINUTES value: "30" - name: VSPHERE_CSI_CONFIG value: "/etc/cloud/csi-vsphere.conf" - name: LOGGER_LEVEL value: "PRODUCTION" # Options: DEVELOPMENT, PRODUCTION - name: INCLUSTER_CLIENT_QPS value: "100" - name: INCLUSTER_CLIENT_BURST value: "100" - name: GODEBUG value: x509sha1=1 - name: CSI_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace volumeMounts: - mountPath: /etc/cloud name: vsphere-config-volume readOnly: true - name: csi-provisioner image: k8s.gcr.io/sig-storage/csi-provisioner:v3.2.1 args: - "--v=4" - "--timeout=300s" - "--csi-address=$(ADDRESS)" - "--kube-api-qps=100" - "--kube-api-burst=100" - "--leader-election" - "--default-fstype=ext4" # needed only for topology aware setup #- "--feature-gates=Topology=true" #- "--strict-topology" env: - name: ADDRESS value: /csi/csi.sock volumeMounts: - mountPath: /csi name: socket-dir - name: csi-snapshotter image: k8s.gcr.io/sig-storage/csi-snapshotter:v6.0.1 args: - "--v=4" - "--kube-api-qps=100" - "--kube-api-burst=100" - "--timeout=300s" - "--csi-address=$(ADDRESS)" - "--leader-election" env: - name: ADDRESS value: /csi/csi.sock volumeMounts: - mountPath: /csi name: socket-dir volumes: - name: vsphere-config-volume secret: secretName: csi-vsphere-config - name: socket-dir emptyDir: {} kind: ConfigMap metadata: name: ${CLUSTER_NAME}-vsphere-csi-controller namespace: ${NAMESPACE} --- apiVersion: v1 data: data: | kind: DaemonSet apiVersion: apps/v1 metadata: name: vsphere-csi-node namespace: kube-system spec: selector: matchLabels: app: vsphere-csi-node updateStrategy: type: "RollingUpdate" rollingUpdate: maxUnavailable: 1 template: metadata: labels: app: vsphere-csi-node role: vsphere-csi spec: nodeSelector: kubernetes.io/os: linux serviceAccountName: vsphere-csi-node hostNetwork: true dnsPolicy: "ClusterFirstWithHostNet" containers: - name: node-driver-registrar image: k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.1 args: - "--v=5" - "--csi-address=$(ADDRESS)" - "--kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)" env: - name: ADDRESS value: /csi/csi.sock - name: DRIVER_REG_SOCK_PATH value: /var/lib/kubelet/plugins/csi.vsphere.vmware.com/csi.sock volumeMounts: - name: plugin-dir mountPath: /csi - name: registration-dir mountPath: /registration livenessProbe: exec: command: - /csi-node-driver-registrar - --kubelet-registration-path=/var/lib/kubelet/plugins/csi.vsphere.vmware.com/csi.sock - --mode=kubelet-registration-probe initialDelaySeconds: 3 - name: vsphere-csi-node image: gcr.io/cloud-provider-vsphere/csi/release/driver:v2.7.0 args: - "--fss-name=internal-feature-states.csi.vsphere.vmware.com" - "--fss-namespace=$(CSI_NAMESPACE)" imagePullPolicy: "Always" env: - name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName - name: CSI_ENDPOINT value: unix:///csi/csi.sock - name: MAX_VOLUMES_PER_NODE value: "59" # Maximum number of volumes that controller can publish to the node. If value is not set or zero Kubernetes decide how many volumes can be published by the controller to the node. - name: X_CSI_MODE value: "node" - name: X_CSI_SPEC_REQ_VALIDATION value: "false" - name: X_CSI_SPEC_DISABLE_LEN_CHECK value: "true" - name: LOGGER_LEVEL value: "PRODUCTION" # Options: DEVELOPMENT, PRODUCTION - name: GODEBUG value: x509sha1=1 - name: CSI_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: NODEGETINFO_WATCH_TIMEOUT_MINUTES value: "1" securityContext: privileged: true capabilities: add: ["SYS_ADMIN"] allowPrivilegeEscalation: true volumeMounts: - name: plugin-dir mountPath: /csi - name: pods-mount-dir mountPath: /var/lib/kubelet # needed so that any mounts setup inside this container are # propagated back to the host machine. mountPropagation: "Bidirectional" - name: device-dir mountPath: /dev - name: blocks-dir mountPath: /sys/block - name: sys-devices-dir mountPath: /sys/devices ports: - name: healthz containerPort: 9808 protocol: TCP livenessProbe: httpGet: path: /healthz port: healthz initialDelaySeconds: 10 timeoutSeconds: 5 periodSeconds: 5 failureThreshold: 3 - name: liveness-probe image: k8s.gcr.io/sig-storage/livenessprobe:v2.7.0 args: - "--v=4" - "--csi-address=/csi/csi.sock" volumeMounts: - name: plugin-dir mountPath: /csi volumes: - name: registration-dir hostPath: path: /var/lib/kubelet/plugins_registry type: Directory - name: plugin-dir hostPath: path: /var/lib/kubelet/plugins/csi.vsphere.vmware.com type: DirectoryOrCreate - name: pods-mount-dir hostPath: path: /var/lib/kubelet type: Directory - name: device-dir hostPath: path: /dev - name: blocks-dir hostPath: path: /sys/block type: Directory - name: sys-devices-dir hostPath: path: /sys/devices type: Directory tolerations: - effect: NoExecute operator: Exists - effect: NoSchedule operator: Exists kind: ConfigMap metadata: name: ${CLUSTER_NAME}-vsphere-csi-node namespace: ${NAMESPACE} --- apiVersion: v1 data: data: | kind: DaemonSet apiVersion: apps/v1 metadata: name: vsphere-csi-node-windows namespace: kube-system spec: selector: matchLabels: app: vsphere-csi-node-windows updateStrategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 template: metadata: labels: app: vsphere-csi-node-windows role: vsphere-csi-windows spec: nodeSelector: kubernetes.io/os: windows serviceAccountName: vsphere-csi-node containers: - name: node-driver-registrar image: k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.1 args: - "--v=5" - "--csi-address=$(ADDRESS)" - "--kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)" env: - name: ADDRESS value: 'unix://C:\\csi\\csi.sock' - name: DRIVER_REG_SOCK_PATH value: 'C:\\var\\lib\\kubelet\\plugins\\csi.vsphere.vmware.com\\csi.sock' volumeMounts: - name: plugin-dir mountPath: /csi - name: registration-dir mountPath: /registration livenessProbe: exec: command: - /csi-node-driver-registrar.exe - --kubelet-registration-path=C:\\var\\lib\\kubelet\\plugins\\csi.vsphere.vmware.com\\csi.sock - --mode=kubelet-registration-probe initialDelaySeconds: 3 - name: vsphere-csi-node image: gcr.io/cloud-provider-vsphere/csi/release/driver:v2.7.0 args: - "--fss-name=internal-feature-states.csi.vsphere.vmware.com" - "--fss-namespace=$(CSI_NAMESPACE)" imagePullPolicy: "Always" env: - name: NODE_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName - name: CSI_ENDPOINT value: 'unix://C:\\csi\\csi.sock' - name: MAX_VOLUMES_PER_NODE value: "59" # Maximum number of volumes that controller can publish to the node. If value is not set or zero Kubernetes decide how many volumes can be published by the controller to the node. - name: X_CSI_MODE value: node - name: X_CSI_SPEC_REQ_VALIDATION value: 'false' - name: X_CSI_SPEC_DISABLE_LEN_CHECK value: "true" - name: LOGGER_LEVEL value: "PRODUCTION" # Options: DEVELOPMENT, PRODUCTION - name: X_CSI_LOG_LEVEL value: DEBUG - name: CSI_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: NODEGETINFO_WATCH_TIMEOUT_MINUTES value: "1" volumeMounts: - name: plugin-dir mountPath: 'C:\csi' - name: pods-mount-dir mountPath: 'C:\var\lib\kubelet' - name: csi-proxy-volume-v1 mountPath: \\.\pipe\csi-proxy-volume-v1 - name: csi-proxy-filesystem-v1 mountPath: \\.\pipe\csi-proxy-filesystem-v1 - name: csi-proxy-disk-v1 mountPath: \\.\pipe\csi-proxy-disk-v1 - name: csi-proxy-system-v1alpha1 mountPath: \\.\pipe\csi-proxy-system-v1alpha1 ports: - name: healthz containerPort: 9808 protocol: TCP livenessProbe: httpGet: path: /healthz port: healthz initialDelaySeconds: 10 timeoutSeconds: 5 periodSeconds: 5 failureThreshold: 3 - name: liveness-probe image: k8s.gcr.io/sig-storage/livenessprobe:v2.7.0 args: - "--v=4" - "--csi-address=/csi/csi.sock" volumeMounts: - name: plugin-dir mountPath: /csi volumes: - name: registration-dir hostPath: path: 'C:\var\lib\kubelet\plugins_registry\' type: Directory - name: plugin-dir hostPath: path: 'C:\var\lib\kubelet\plugins\csi.vsphere.vmware.com\' type: DirectoryOrCreate - name: pods-mount-dir hostPath: path: \var\lib\kubelet type: Directory - name: csi-proxy-disk-v1 hostPath: path: \\.\pipe\csi-proxy-disk-v1 type: '' - name: csi-proxy-volume-v1 hostPath: path: \\.\pipe\csi-proxy-volume-v1 type: '' - name: csi-proxy-filesystem-v1 hostPath: path: \\.\pipe\csi-proxy-filesystem-v1 type: '' - name: csi-proxy-system-v1alpha1 hostPath: path: \\.\pipe\csi-proxy-system-v1alpha1 type: '' tolerations: - effect: NoExecute operator: Exists - effect: NoSchedule operator: Exists kind: ConfigMap metadata: name: ${CLUSTER_NAME}-vsphere-csi-node-windows namespace: ${NAMESPACE} --- apiVersion: v1 kind: Secret metadata: name: ${CLUSTER_NAME}-cloud-controller-manager namespace: ${NAMESPACE} stringData: data: | apiVersion: v1 kind: ServiceAccount metadata: labels: component: cloud-controller-manager vsphere-cpi-infra: service-account name: cloud-controller-manager namespace: kube-system type: addons.cluster.x-k8s.io/resource-set --- apiVersion: v1 kind: Secret metadata: name: ${CLUSTER_NAME}-cloud-provider-vsphere-credentials namespace: ${NAMESPACE} stringData: data: | apiVersion: v1 kind: Secret metadata: labels: component: cloud-controller-manager vsphere-cpi-infra: secret name: cloud-provider-vsphere-credentials namespace: kube-system stringData: ${VSPHERE_SERVER}.password: ${VSPHERE_PASSWORD} ${VSPHERE_SERVER}.username: ${VSPHERE_USERNAME} type: Opaque type: addons.cluster.x-k8s.io/resource-set --- apiVersion: v1 data: data: | --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: component: cloud-controller-manager vsphere-cpi-infra: role name: system:cloud-controller-manager rules: - apiGroups: - "" resources: - events verbs: - create - patch - update - apiGroups: - "" resources: - nodes verbs: - '*' - apiGroups: - "" resources: - nodes/status verbs: - patch - apiGroups: - "" resources: - services verbs: - list - patch - update - watch - apiGroups: - "" resources: - services/status verbs: - patch - apiGroups: - "" resources: - serviceaccounts verbs: - create - get - list - watch - update - apiGroups: - "" resources: - persistentvolumes verbs: - get - list - watch - update - apiGroups: - "" resources: - endpoints verbs: - create - get - list - watch - update - apiGroups: - "" resources: - secrets verbs: - get - list - watch - apiGroups: - coordination.k8s.io resources: - leases verbs: - get - watch - list - update - create --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: labels: component: cloud-controller-manager vsphere-cpi-infra: cluster-role-binding name: system:cloud-controller-manager roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:cloud-controller-manager subjects: - kind: ServiceAccount name: cloud-controller-manager namespace: kube-system - kind: User name: cloud-controller-manager --- apiVersion: v1 data: vsphere.conf: | global: port: 443 secretName: cloud-provider-vsphere-credentials secretNamespace: kube-system thumbprint: '${VSPHERE_TLS_THUMBPRINT}' vcenter: ${VSPHERE_SERVER}: datacenters: - '${VSPHERE_DATACENTER}' server: '${VSPHERE_SERVER}' kind: ConfigMap metadata: name: vsphere-cloud-config namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: labels: component: cloud-controller-manager vsphere-cpi-infra: role-binding name: servicecatalog.k8s.io:apiserver-authentication-reader namespace: kube-system roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: extension-apiserver-authentication-reader subjects: - kind: ServiceAccount name: cloud-controller-manager namespace: kube-system - kind: User name: cloud-controller-manager --- apiVersion: apps/v1 kind: DaemonSet metadata: labels: component: cloud-controller-manager tier: control-plane name: vsphere-cloud-controller-manager namespace: kube-system spec: selector: matchLabels: name: vsphere-cloud-controller-manager template: metadata: labels: component: cloud-controller-manager name: vsphere-cloud-controller-manager tier: control-plane spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: node-role.kubernetes.io/control-plane operator: Exists - matchExpressions: - key: node-role.kubernetes.io/master operator: Exists containers: - args: - --v=2 - --cloud-provider=vsphere - --cloud-config=/etc/cloud/vsphere.conf image: gcr.io/cloud-provider-vsphere/cpi/release/manager:v1.25.3 name: vsphere-cloud-controller-manager resources: requests: cpu: 200m volumeMounts: - mountPath: /etc/cloud name: vsphere-config-volume readOnly: true hostNetwork: true priorityClassName: system-node-critical securityContext: runAsUser: 1001 serviceAccountName: cloud-controller-manager tolerations: - effect: NoSchedule key: node.cloudprovider.kubernetes.io/uninitialized value: "true" - effect: NoSchedule key: node-role.kubernetes.io/master operator: Exists - effect: NoSchedule key: node-role.kubernetes.io/control-plane operator: Exists - effect: NoSchedule key: node.kubernetes.io/not-ready operator: Exists volumes: - configMap: name: vsphere-cloud-config name: vsphere-config-volume updateStrategy: type: RollingUpdate kind: ConfigMap metadata: name: ${CLUSTER_NAME}-cpi-manifests namespace: ${NAMESPACE} --- apiVersion: addons.cluster.x-k8s.io/v1beta1 kind: ClusterResourceSet metadata: name: ${CLUSTER_NAME}-calico-module-resource namespace: ${NAMESPACE} spec: clusterSelector: matchLabels: cluster.x-k8s.io/cluster-name: ${CLUSTER_NAME} resources: - kind: ConfigMap name: ${CLUSTER_NAME}-calico-module-cr strategy: Reconcile --- apiVersion: v1 data: calico.yaml: | apiVersion: platform.verrazzano.io/v1alpha1 kind: Module metadata: name: calico namespace: default spec: moduleName: calico targetNamespace: default values: tigeraOperator: version: ${TIGERA_TAG=v1.29.0} installation: cni: type: Calico calicoNetwork: bgp: Disabled ipPools: - cidr: ${POD_CIDR=192.168.0.0/16} encapsulation: VXLAN registry: ${OCNE_IMAGE_REPOSITORY=container-registry.oracle.com} imagePath: ${OCNE_IMAGE_PATH=olcne} kind: ConfigMap metadata: annotations: note: generated labels: type: generated name: ${CLUSTER_NAME}-calico-module-cr namespace: ${NAMESPACE} -
Generate and apply the template by running the following command:
$ clusterctl generate yaml --from vsphere-capi.yaml | kubectl apply -f -
To get the kubeconfig file, run:
$ clusterctl get kubeconfig kluster1 -n kluster1 > kluster1
Finish cluster configuration
After the cluster resources are created, you must perform some additional steps to finish the configuration of the cluster.
-
If vSphere does not have a load balancer, then you can deploy MetalLB.
$ export KUBECONFIG=kluster1 ADDRESS_RANGE=${1:-"subnet-from-vSphere-network"}; $ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.7/config/manifests/metallb-native.yaml --wait=true; $ kubectl rollout status deployment -n metallb-system controller -w; $ kubectl apply -f - <<EOF1 apiVersion: metallb.io/v1beta1 kind: IPAddressPool metadata: name: vzlocalpool namespace: metallb-system spec: addresses: - ${ADDRESS_RANGE} EOF1 $ kubectl apply -f - <<-EOF2 apiVersion: metallb.io/v1beta1 kind: L2Advertisement metadata: name: vzmetallb namespace: metallb-system spec: ipAddressPools: - vzlocalpool EOF2 $ sleep 10; $ kubectl wait --namespace metallb-system --for=condition=ready pod --all --timeout=300s -
Create a default storage class on the cluster.
$ export KUBECONFIG=kluster1 $ kubectl apply -f - <<-EOF kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: vmware-sc annotations: storageclass.kubernetes.io/is-default-class: "true" provisioner: csi.vsphere.vmware.com volumeBindingMode: WaitForFirstConsumer EOF -
Install Verrazzano on the managed cluster.
$ export KUBECONFIG=kluster1 $ vz install -f - <<EOF apiVersion: install.verrazzano.io/v1beta1 kind: Verrazzano metadata: name: example-verrazzano spec: profile: dev defaultVolumeSource: persistentVolumeClaim: claimName: verrazzano-storage volumeClaimSpecTemplates: - metadata: name: verrazzano-storage spec: resources: requests: storage: 2Gi EOF
Your admin cluster and first managed cluster are now up and running and ready to deploy applications. You can also add more managed clusters.
For more information, refer to the documentation for Cluster API and Cluster API vSphere:
Troubleshoot the deployment
If the deployment of the vSphere resources fails, then you can check the log files to diagnose the issue.
The vSphere cluster controller provider logs:
$ kubectl logs -n verrazzano-capi -l cluster.x-k8s.io/provider=infrastructure-vsphere
The OCNE control plane provider logs:
$ kubectl logs -n verrazzano-capi -l cluster.x-k8s.io/provider=control-plane-ocne
NOTE: If the CSI pod deploys before Calico, then the pod may enter a CrashLoop state. Restart the pod to fix the issue.
$ kubectl --kubeconfig kluster1 scale deploy -n kube-system vsphere-csi-controller --replicas=0
$ kubectl --kubeconfig kluster1 scale deploy -n kube-system vsphere-csi-controller --replicas=1
Delete the clusters
- Delete the managed clusters.
$ kubectl delete cluster $CLUSTER_NAME - Delete the admin cluster.
$ kind delete cluster
Do not use kubectl delete -f capi-quickstart.yaml to delete the entire cluster template at once because it might leave behind pending resources that you need to clean up manually.
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.