Amazon EKS automatic scaling based on cluster metrics

You can download this article in PDF format via the link below to support us.Download the guide in PDF formatClose

Auto-scaling is one of the many advantages that Kubernetes must provide. This can automatically configure resources based on traffic/demand. As the traffic demand increases, more resources will be provided, and vice versa. Kubernetes provides three types of extensions:

  • Vertical Pod automatic scaler: Scaling ratio, increase/decrease the CPU or memory allocated to the Pod.
  • Horizontal pod auto scaler: Increase/decrease the number of Pods per service according to traffic/demand.
  • Automatic cluster scaling: Expand cluster nodes according to the number of pending Pods.

Running Kubernetes on AWS (EKS) provides the option of scaling cluster nodes based on cluster resource utilization indicators. This article guides readers to configure automatic expansion based on cluster indicators. We will use the “cluster node memory utilization” metric to scale the nodes.

N/B: This is only possible on EKS clusters with Provisioned Worker Nodes, not on EKS Fargate clusters. With Fargate, there is no need to configure instances for your cluster because it provides computing power for your containers. It can also automatically expand resources for you based on traffic demand.

Requirements/prerequisites

  • An AWS account.
  • A user is created on this account, and the user has permissions to configure account resources and access CloudWatch.
  • An AWS EKS cluster equipped with a managed node group.
  • By deploying the CloudWatch Agent on the cluster, container insights are enabled for the EKS cluster.

Part 1: Create AWS EKS SNS Topic and Subscription

We will create two SNS topics, namely “scale up” and “scale down”, and add email subscriptions to this topic. The following CloudFormation templates can be used for this purpose.

Reduce topics and subscription templates.

AWSTemplateFormatVersion: "2010-09-09"
Description: "Template to create SNS Topic and Subscriptions"

Parameters:
    Name:
        Type: String
        Description: The Topic Name
        Default: EKS_CLUSTER_MEMORY_LESS_THAN_50

Resources:
    Email1Notification:
        Type: AWS::SNS::Subscription
        Properties:
           Endpoint: [email protected]
           Protocol: email
           TopicArn: !Ref ScaleDownTopic

    Email2Notification:
        Type: AWS::SNS::Subscription
        Properties:
           Endpoint: [email protected]
           Protocol: email
           TopicArn: !Ref ScaleDownTopic

    ScaleDownTopic:   
        Type: AWS::SNS::Topic
        Properties: 
           Tags: 
              - Key: Name
                Value: !Ref Name
              - Key: createdBy
                Value: Maureen Barasa
              - Key: Project
                Value: test-blog
           TopicName: !Ref Name

Outputs:    
    ScaleDown:
        Description: The scale down topic
        Value: !Ref ScaleDownTopic
        Export:
          Name: "EKSScaleDown"
          Value: !Ref ScaleDownTopic

N/B: Users should customize the template. The names of resources, email endpoints and tags should meet the specific requirements of the user.

Expand topics and subscription templates

AWSTemplateFormatVersion: "2010-09-09"
Description: "Template to create SNS Topic and Subscriptions"
Parameters:
    Name:
        Type: String
        Description: The Topic Name
        Default: EKS_CLUSTER_MEMORY_MORE_THAN_70

Resources:
    Email1Notification:
        Type: AWS::SNS::Subscription
        Properties:
           Endpoint: <your-email-address>
           Protocol: email
           TopicArn: !Ref ScaleUpTopic

    Email2Notification:
        Type: AWS::SNS::Subscription
        Properties:
           Endpoint: <your-email-2-addresS>
           Protocol: email
           TopicArn: !Ref ScaleUpTopic

    ScaleUpTopic:   
        Type: AWS::SNS::Topic
        Properties: 
           Tags: 
              - Key: Name
                Value: !Ref Name
              - Key: createdBy
                Value: <your-name>
              - Key: Project
                Value: <project-name>
           TopicName: !Ref Name

Outputs:
    ScaleUp:
        Description: The scale up topic
        Value: !Ref ScaleUpTopic
        Export:
          Name: "EKSScaleUp"
          Value: !Ref ScaleUpTopic

N/B: Users should customize the template. The names of resources, email endpoints and tags should meet the specific requirements of the user.

  • <您的电子邮件地址> Is your first email address, for example [email protected], [email protected]
  • <您的电子邮件-2-地址> Is the second email address, for example [email protected]
  • <项目名称> Is the name of the item to which the theme applies

Under Output, we allow to export our values, so we can use them in the template that will create CloudWatch Alarms.

After creating the SNS topic and subscription, go to the given email endpoint and confirm the subscription. Now, we are ready to create extended policies and CloudWatch alarms.

Part 2: Create an extension strategy and attach it to your AWS EKS NodeGroup ASG

Next, we will create two policies and attach them to the EKS node group auto-scaling group. Use the cloud formation template below.

AWSTemplateFormatVersion: "2010-09-09"
Description: "create a scaling policy and apply it to EKS ASG Group"

Resources:
  ScaleDownPolicy: 
    Type: AWS::AutoScaling::ScalingPolicy
    Properties: 
      AdjustmentType: "ChangeInCapacity"
      PolicyType: "SimpleScaling"
      Cooldown: "300"
      AutoScalingGroupName: "eks-ASG-GroupName"
      ScalingAdjustment: -2

  ScaleUpPolicy: 
    Type: AWS::AutoScaling::ScalingPolicy
    Properties: 
      AdjustmentType: "ChangeInCapacity"
      PolicyType: "SimpleScaling"
      Cooldown: "300"
      AutoScalingGroupName: "eks-ASG-GroupName"
      ScalingAdjustment: 2

Outputs:
    ScaleUp:
        Description: The scale up policy
        Value: !Ref ScaleUpPolicy
        Export:
          Name: "EKSScaleUpPolicy"
          Value: !Ref ScaleUpPolicy

    ScaleDown:
        Description: The scale down policy
        Value: !Ref ScaleDownPolicy
        Export:
          Name: "EKSScaleDownPolicy"
          Value: !Ref ScaleDownPolicy

Part 3: Create AWS EKS CloudWatch Alarm

Finally, we will create CloudWatch alarms. Here, we will use the values ​​created from the above template. Namely SNS theme and expansion strategy.

On the CloudWatch console, select the alarm. Then click Create Alert.

Create alert

Under “Select Metrics”, select “Container insights“Namespace, and then press “Cluster Name” to select. Under your EKS cluster name, select node_memory_utilization Metrics, and then click Select Metrics. Next, specify the index conditions. For the period, you can choose a time between one minute and one day. This is the time it takes to analyze the metric or expression to create a single data point of the alert.

Amazon EKS automatic scaling based on cluster metricsSpecify index conditions

Next, we define the conditions of the alert. For the first case, we will create an alarm where the node memory utilization is greater than 70. Click Next.

Amazon EKS automatic scaling based on cluster metricsDefine alarm conditions

Then, we configure the actions to be taken when the alarm is triggered. For us, our first action is to send a notification to the corresponding SNS topic created in Part 1 above. For example, when the average memory utilization is greater than 70, it should send a notification to the EKS_CLUSTER_MEMORY_MORE_THAN_70 SNS topic.

Amazon EKS automatic scaling based on cluster metricsConfigure alert actions

The second action to be done is to expand according to the expansion strategy we defined in Part 2 of this article. Make sure to select the EKS cluster node, scaling group. For the high memory utilization alarm, select the expansion policy you created, and then click Next.

Amazon EKS automatic scaling based on cluster metricsSelect zoom action

Finally, create a name and description for the alert as shown below, and click Next. After carefully evaluating the configuration, you can now click “Create Alert”.

Amazon EKS automatic scaling based on cluster metricsCreate alarm name and description

It is also important to create a low memory alert, follow the same steps as above. This time to ensure that for the alert, under certain conditions, we should choose the relevant SNS topic and expansion strategy. For example, use the EKS_CLUSTER_MEMORY_LESS_THAN_50 SNS topic and the scaling strategy for scaling operations.

Now you have successfully configured the EKS cluster auto-scaling using CloudWatch Alarms and SNS topics.

Amazon EKS automatic scaling based on cluster metricsCloudWatch alarm created

Other Kubernetes guides:

  • How to configure Kubernetes dynamic volume configuration using Heketi and GlusterFS
  • Install Kubernetes Metrics Server on Amazon EKS cluster
  • Grant developers access to the EKS Kubernetes cluster
  • Kubectl cheat sheet for Kubernetes administrators and CKA exam preparation

Happy Building! ! !

You can download this article in PDF format via the link below to support us.Download the guide in PDF formatClose

Sidebar