Local cluster setup

EKS Anywhere docker provider deployments

EKS Anywhere supports a Docker provider for development and testing use cases only. This allows you to try EKS Anywhere on your local system before deploying to a supported provider to create either:

  • A single, standalone cluster or
  • Multiple management/workload clusters on the same provider, as described in Cluster topologies

    . The management/workload topology is recommended for production clusters and can be tried out here using both eksctl and GitOps tools.

Create a standalone cluster

Prerequisite Checklist

To install the EKS Anywhere binaries and see system requirements please follow the installation guide

.

Steps

  1. Generate a cluster config

    CLUSTER_NAME=mgmt
eksctl anywhere generate clusterconfig $CLUSTER_NAME \
   --provider docker > $CLUSTER_NAME.yaml

    The command above creates a file named eksa-cluster.yaml with the contents below in the path where it is executed. The configuration specification is divided into two sections:

    • Cluster
    • DockerDatacenterConfig
    apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: Cluster
metadata:
   name: mgmt
spec:
   clusterNetwork:
      cniConfig:
         cilium: {}
      pods:
         cidrBlocks:
            - 192.168.0.0/16
      services:
         cidrBlocks:
            - 10.96.0.0/12
   controlPlaneConfiguration:
      count: 1
   datacenterRef:
      kind: DockerDatacenterConfig
      name: mgmt
   externalEtcdConfiguration:
      count: 1
   kubernetesVersion: "1.25"
   managementCluster:
      name: mgmt
   workerNodeGroupConfigurations:
      - count: 1
         name: md-0
---
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: DockerDatacenterConfig
metadata:
   name: mgmt
spec: {}
    • Apart from the base configuration, you can add additional optional configuration to enable supported features:

  2. Configure Curated Packages

    The Amazon EKS Anywhere Curated Packages are only available to customers with the Amazon EKS Anywhere Enterprise Subscription. To request a free trial, talk to your Amazon representative or connect with one here

    . Cluster creation will succeed if authentication is not set up, but some warnings may be generated. Detailed package configurations can be found here

    .

    If you are going to use packages, set up authentication. These credentials should have limited capabilities

    :

    export EKSA_AWS_ACCESS_KEY_ID="your*access*id"
export EKSA_AWS_SECRET_ACCESS_KEY="your*secret*key"
export EKSA_AWS_REGION="us-west-2"

    NOTE: The Amazon EKS Anywhere Curated Packages are only available to customers with the Amazon EKS Anywhere Enterprise Subscription. Due to this there might be some warnings in the CLI if proper authentication is not set up.

  3. Create Cluster:

    For a regular cluster create (with internet access), type the following:

    eksctl anywhere create cluster \
   # --install-packages packages.yaml \ # uncomment to install curated packages at cluster creation
   -f $CLUSTER_NAME.yaml

    For an airgapped cluster create, follow Preparation for airgapped deployments

    instructions, then type the following:

    eksctl anywhere create cluster 
   # --install-packages packages.yaml \ # uncomment to install curated packages at cluster creation
   -f $CLUSTER_NAME.yaml \
   --bundles-override ./eks-anywhere-downloads/bundle-release.yaml

    Example command output

    Performing setup and validations
✅ validation succeeded {"validation": "docker Provider setup is valid"}
Creating new bootstrap cluster
Installing cluster-api providers on bootstrap cluster
Provider specific setup
Creating new workload cluster
Installing networking on workload cluster
Installing cluster-api providers on workload cluster
Moving cluster management from bootstrap to workload cluster
Installing EKS-A custom components (CRD and controller) on workload cluster
Creating EKS-A CRDs instances on workload cluster
Installing GitOps Toolkit on workload cluster
GitOps field not specified, bootstrap flux skipped
Deleting bootstrap cluster
🎉 Cluster created!
----------------------------------------------------------------------------------
The Amazon EKS Anywhere Curated Packages are only available to customers with the
Amazon EKS Anywhere Enterprise Subscription
----------------------------------------------------------------------------------
Installing curated packages controller on management cluster
secret/aws-secret created
job.batch/eksa-auth-refresher created

    NOTE: to install curated packages during cluster creation, use --install-packages packages.yaml flag

  4. Use the cluster

    Once the cluster is created you can use it with the generated KUBECONFIG file in your local directory

    export KUBECONFIG=${PWD}/${CLUSTER_NAME}/${CLUSTER_NAME}-eks-a-cluster.kubeconfig
kubectl get ns

    Example command output

    NAME                                STATUS   AGE
capd-system                         Active   21m
capi-kubeadm-bootstrap-system       Active   21m
capi-kubeadm-control-plane-system   Active   21m
capi-system                         Active   21m
capi-webhook-system                 Active   21m
cert-manager                        Active   22m
default                             Active   23m
eksa-packages                       Active   23m
eksa-system                         Active   20m
kube-node-lease                     Active   23m
kube-public                         Active   23m
kube-system                         Active   23m

    You can now use the cluster like you would any Kubernetes cluster. Deploy the test application with:

    kubectl apply -f "https://anywhere.eks.amazonaws.com/manifests/hello-eks-a.yaml"

    Verify the test application in the deploy test application section

    .

Create management/workload clusters

To try the recommended EKS Anywhere topology,

you can create a management cluster and one or more workload clusters on the same Docker provider.

Prerequisite Checklist

To install the EKS Anywhere binaries and see system requirements please follow the installation guide

.

Create a management cluster

  1. Generate a management cluster config (named mgmt for this example):

    CLUSTER_NAME=mgmt
eksctl anywhere generate clusterconfig $CLUSTER_NAME \
   --provider docker > eksa-mgmt-cluster.yaml

  2. Modify the management cluster config (eksa-mgmt-cluster.yaml) you could use the same one described earlier or modify it to use GitOps, as shown below:

    apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: Cluster
metadata:
  name: mgmt
  namespace: default
spec:
  bundlesRef:
    apiVersion: anywhere.eks.amazonaws.com/v1alpha1
    name: bundles-1
    namespace: eksa-system
  clusterNetwork:
    cniConfig:
      cilium: {}
    pods:
      cidrBlocks:
      - 192.168.0.0/16
    services:
      cidrBlocks:
      - 10.96.0.0/12
  controlPlaneConfiguration:
    count: 1
  datacenterRef:
    kind: DockerDatacenterConfig
    name: mgmt
  externalEtcdConfiguration:
    count: 1
  gitOpsRef:
    kind: FluxConfig
    name: mgmt
  kubernetesVersion: "1.25"
  managementCluster:
    name: mgmt
  workerNodeGroupConfigurations:
  - count: 1
    name: md-1

    

    apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: DockerDatacenterConfig
metadata:
name: mgmt
namespace: default
spec: {}
    

    

    apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: FluxConfig
metadata:
name: mgmt
namespace: default
spec:
branch: main
clusterConfigPath: clusters/mgmt
github:
owner: <your github account, such as example for https://github.com/example>
personal: true
repository: <your github repo, such as test for https://github.com/example/test>
systemNamespace: flux-system

  3. Configure Curated Packages

    The Amazon EKS Anywhere Curated Packages are only available to customers with the Amazon EKS Anywhere Enterprise Subscription. To request a free trial, talk to your Amazon representative or connect with one here

    . Cluster creation will succeed if authentication is not set up, but some warnings may be generated. Detailed package configurations can be found here

    .

    If you are going to use packages, set up authentication. These credentials should have limited capabilities

    :

    export EKSA_AWS_ACCESS_KEY_ID="your*access*id"
export EKSA_AWS_SECRET_ACCESS_KEY="your*secret*key"

  4. Create cluster:

    For a regular cluster create (with internet access), type the following:

    eksctl anywhere create cluster \ 
   # --install-packages packages.yaml \ # uncomment to install curated packages at cluster creation
   -f $CLUSTER_NAME.yaml

    For an airgapped cluster create, follow Preparation for airgapped deployments

    instructions, then type the following:

    eksctl anywhere create cluster \
   # --install-packages packages.yaml \ # uncomment to install curated packages at cluster creation
   -f $CLUSTER_NAME.yaml \
   --bundles-override ./eks-anywhere-downloads/bundle-release.yaml

  5. Once the cluster is created you can use it with the generated KUBECONFIG file in your local directory:

    export KUBECONFIG=${PWD}/${CLUSTER_NAME}/${CLUSTER_NAME}-eks-a-cluster.kubeconfig

  6. Check the initial cluster’s CRD:

    To ensure you are looking at the initial cluster, list the CRD to see that the name of its management cluster is itself:

    kubectl get clusters mgmt -o yaml

    Example command output

    ...
kubernetesVersion: "1.25"
managementCluster:
  name: mgmt
workerNodeGroupConfigurations:
...

Create separate workload clusters

Follow these steps to have your management cluster create and manage separate workload clusters.

  1. Generate a workload cluster config:

    CLUSTER_NAME=w01
eksctl anywhere generate clusterconfig $CLUSTER_NAME \
   --provider docker > eksa-w01-cluster.yaml

    Refer to the initial config described earlier for the required and optional settings.

    NOTE: Ensure workload cluster object names (Cluster, DockerDatacenterConfig, etc.) are distinct from management cluster object names. Be sure to set the managementCluster field to identify the name of the management cluster.

  2. Create a workload cluster in one of the following ways:

    • GitOps: See Manage separate workload clusters with GitOps

    • Terraform: See Manage separate workload clusters with Terraform

    • Kubernetes CLI: The cluster lifecycle feature lets you use kubectl to manage a workload cluster. For example:

      kubectl apply -f eksa-w01-cluster.yaml

    • eksctl CLI: Useful for temporary cluster configurations. To create a workload cluster with eksctl, do one of the following. For a regular cluster create (with internet access), type the following:

      eksctl anywhere create cluster \
    -f eksa-w01-cluster.yaml  \
   # --install-packages packages.yaml \ # uncomment to install curated packages at cluster creation
    --kubeconfig mgmt/mgmt-eks-a-cluster.kubeconfig

      For an airgapped cluster create, follow Preparation for airgapped deployments

      instructions, then type the following:

      eksctl create cluster \
   # --install-packages packages.yaml \ # uncomment to install curated packages at cluster creation
   -f $CLUSTER_NAME.yaml \
   --bundles-override ./eks-anywhere-downloads/bundle-release.yaml \
    --kubeconfig mgmt/mgmt-eks-a-cluster.kubeconfig

      As noted earlier, adding the --kubeconfig option tells eksctl to use the management cluster identified by that kubeconfig file to create a different workload cluster.

  3. To check the workload cluster, get the workload cluster credentials and run a test workload:

    • If your workload cluster was created with eksctl, change your credentials to point to the new workload cluster (for example, w01), then run the test application with:

      export CLUSTER_NAME=w01
export KUBECONFIG=${PWD}/${CLUSTER_NAME}/${CLUSTER_NAME}-eks-a-cluster.kubeconfig
kubectl apply -f "https://anywhere.eks.amazonaws.com/manifests/hello-eks-a.yaml"

    • If your workload cluster was created with GitOps or Terraform, you can get credentials and run the test application as follows:

      kubectl get secret -n eksa-system w01-kubeconfig -o jsonpath=‘{.data.value}' | base64 —decode > w01.kubeconfig
export KUBECONFIG=w01.kubeconfig
kubectl apply -f "https://anywhere.eks.amazonaws.com/manifests/hello-eks-a.yaml"

      NOTE: For Docker, you must modify the server field of the kubeconfig file by replacing the IP with 127.0.0.1 and the port with its value. The port’s value can be found by running docker ps and checking the workload cluster’s load balancer.

  4. Add more workload clusters:

    To add more workload clusters, go through the same steps for creating the initial workload, copying the config file to a new name (such as eksa-w02-cluster.yaml), modifying resource names, and running the create cluster command again.

Next steps:

  • See the Cluster management

    section for more information on common operational tasks like scaling and deleting the cluster.

  • See the Package management

    section for more information on post-creation curated packages installation.

To verify that a cluster control plane is up and running, use the kubectl command to show that the control plane pods are all running.

kubectl get po -A -l control-plane=controller-manager
NAMESPACE                           NAME                                                             READY   STATUS    RESTARTS   AGE
capi-kubeadm-bootstrap-system       capi-kubeadm-bootstrap-controller-manager-57b99f579f-sd85g       2/2     Running   0          47m
capi-kubeadm-control-plane-system   capi-kubeadm-control-plane-controller-manager-79cdf98fb8-ll498   2/2     Running   0          47m
capi-system                         capi-controller-manager-59f4547955-2ks8t                         2/2     Running   0          47m
capi-webhook-system                 capi-controller-manager-bb4dc9878-2j8mg                          2/2     Running   0          47m
capi-webhook-system                 capi-kubeadm-bootstrap-controller-manager-6b4cb6f656-qfppd       2/2     Running   0          47m
capi-webhook-system                 capi-kubeadm-control-plane-controller-manager-bf7878ffc-rgsm8    2/2     Running   0          47m
capi-webhook-system                 capv-controller-manager-5668dbcd5-v5szb                          2/2     Running   0          47m
capv-system                         capv-controller-manager-584886b7bd-f66hs                         2/2     Running   0          47m

You may also check the status of the cluster control plane resource directly. This can be especially useful to verify clusters with multiple control plane nodes after an upgrade.

kubectl get kubeadmcontrolplanes.controlplane.cluster.x-k8s.io
NAME                       INITIALIZED   API SERVER AVAILABLE   VERSION              REPLICAS   READY   UPDATED   UNAVAILABLE
supportbundletestcluster   true          true                   v1.20.7-eks-1-20-6   1          1       1

To verify that the expected number of cluster worker nodes are up and running, use the kubectl command to show that nodes are Ready.

This will confirm that the expected number of worker nodes are present. Worker nodes are named using the cluster name followed by the worker node group name (example: my-cluster-md-0)

kubectl get nodes
NAME                                           STATUS   ROLES                  AGE    VERSION
supportbundletestcluster-md-0-55bb5ccd-mrcf9   Ready    <none>                 4m   v1.20.7-eks-1-20-6
supportbundletestcluster-md-0-55bb5ccd-zrh97   Ready    <none>                 4m   v1.20.7-eks-1-20-6
supportbundletestcluster-mdrwf                 Ready    control-plane,master   5m   v1.20.7-eks-1-20-6

To test a workload in your cluster you can try deploying the hello-eks-anywhere

.