AKS Case Study

Kubernetes is by far the most popular container orchestration tool, yet the complexities of managing the tool have led to the rise of fully-managed Kubernetes services over the past few years.

Although Azure supports multiple container tools, it’s now going all-in on Kubernetes and will deprecate its original offerings this year. The great part about cloud-based managed Kubernetes services like Azure Kubernetes Service (AKS) is that it integrates natively with other Azure services, and you don’t have to worry about managing the availability of your underlying clusters, auto scaling, or patching your underlying VMs.

In this blog post, we’ll be reviewing the basics of Kubernetes and AKS, before diving into a real-life use case with AKS.

1. What is Kubernetes?
2. Why use Kubernetes?
3. Azure Kubernetes Service (AKS) Deep Dive
4. Real-Life AKS Deployment

Basics of Kubernetes

Kubernetes is a portable, extensible, open source platform for container orchestration. It allows developers and engineers to manage containerized workloads and services through both declarative configuration and automation.

Basic benefits of Kubernetes include:

• Run distributed systems resiliently
• Automatically mount a storage system
• Automated rollouts and rollbacks
• Self-healing
• Secret and configuration management

Key Terms

API Server: Exposes the underlying Kubernetes API. This is how various management tools interact with the Kubernetes cluster

Controller Manager: Watches the state of the cluster through API server and when necessary makes changes attempting to move the current state towards the desired state.

Etcd: Highly available key value store which maintains the Kubernetes cluster state.

Scheduler: Schedules unassigned pods to nodes. Determines the most optimal node to run your pod

Node: A physical or virtual machine which is where Kubernetes runs your containers.

Kube-proxy: A network proxy that proxies requests to Kubernetes services and their backend pods

Pods: One or more containers logically grouped together. Usually they need to share the same resources.

Kublet: Agent that processes orchestration requests and handles starting pods that have been assigned to its node by the scheduler

Why Use Kubernetes?

When running containers in a production environment, containers need to be managed to ensure they are operating as expected in an effort to ensure there is no downtime.

• Container Orchestration: Without container orchestration, If a container was to go down and stop working, an engineer would need to know the container has failed and manually start a new one. Wouldn’t it be better if this was handled automatically by its own system? Kubernetes provides a robust declarative framework to run your containerized applications and services resiliently.
• Cloud Agnostic: Kubernetes has been designed and built to be used anywhere (public/private/hybrid clouds)
• Prevents Vendor Lock-In: Your containerized application and Kubernetes manifests will run the same way on any platform with minimal changes
• Increase Developer Agility and Faster Time-to-Market: Spend less time scripting deployment workflows and focus on developing. Kubernetes provides a declarative configuration which allows engineers to define how their service is to be ran by Kubernetes, Kubernetes will then ensure the state of the application is maintained
• Cloud Aware: Kubernetes understands and supports a number of various clouds such as Google Cloud, Azure, AWS. This allows Kubernetes to instantiate various public cloud based resources, such as instances, VMs, load balancers, public IPs, storage..etc.

Basics of Azure Kubernetes Services

Azure Kubernetes Service (AKS) is a fully-managed service that allows you to run Kubernetes in Azure without having to manage your own Kubernetes clusters. Azure manages all the complex parts of running Kubernetes, and you can focus on your containers. Basic features include:

• Pay only for the nodes (VMs)
• Easier cluster upgrades
• Integrated with various Azure and OSS tools and services
• Kubernetes RBAC and Azure Active Directory Integration
• Enforce rules defined in Azure Policy across multiple clusters
• Kubernetes can scale your Nodes using cluster autoscaler
• Expand your scale even greater by scheduling your containers on Azure Container Instances

Azure Kubernetes Best Practices

Cluster Multi-Tenancy

• Logically isolate clusters to separate teams and projects in an effort to try to minimize the number of physical AKS clusters you deploy
• Namespace allows you to isolate inside of a Kubernetes cluster
• Same best practices with hub-spoke but you do it within the Kubernetes cluster itself

Scheduling and Resource Quotas

• Enforce resource quotas — Plan out and apply resource quotas at the namespace level
• Plan for availability
• Define pod disruption budgets
• Limit resource intensive applications — Apply taints and tolerations to constrain resource intensive applications to specific nodes

Cluster Security

Azure AD and Kubernetes RBAC integration

• Bind your Kubernetes RBAC roles with Azure AD Users/Groups
• Grant your Azure AD users or groups access to Kubernetes resources within a namespace or across a cluster

Kubernetes Cluster Updates

• Kubernetes releases updates at a quicker pace than more traditional infrastructure platforms. These updates usually include new features, and bug or security fixes.
• AKS supports four minor versions of Kubernetes
• Upgrading AKS clusters are as simple as executing a Azure CLI command. AKS handles a graceful upgrade by safely cordon and draining old nodes in order to minimize disruption to running applications. Once new nodes are up and containers are running, old nodes are deleted by AKS.

Node Patching

Linux

AKS automatically checks for kernel and security updates on a nightly basis and if available AKS will install them on Linux nodes. If a reboot is required, AKS will not automatically reboot the node, a best practice for patching Linux nodes is to leverage the kured (Kubernetes Reboot Daemon) which looks for the existence of /var/run/reboot-required file (created when a reboot is required) and will automatically reboot during a predefined scheduled time.

Windows

The process for patching Windows nodes is slightly different. Patches aren’t applied on a daily basis like Linux nodes. Windows nodes must be updated by performing an AKS upgrade which creates new nodes on the latest base Windows Server image and patches.

Pod Identities

If your containers require access to the ARM API, there is no need to provide fixed credentials that must be rotated periodically. Azure’s pod identities solution can be deployed to your cluster which allows your containers to dynamically acquire access to Azure API and services through the use of Managed Identities (marked Azure MSI in the diagram below).

Limit container access

Avoid creating applications and containers that require escalated privileges or root access.

Monitoring

As AKS is already integrated with other Azure services, you can use Azure Monitor to monitor containers in AKS.

• Toggled based implementation, can be enabled after the fact or enforced via Azure Policy
• Multi and Cluster specific views
• Integrates with Log Analytics
• Ability to query historic data
• Analyze your Cluster, Nodes, Controllers, and Containers
• Alert on Cluster & Container performance by writing customizable Log Analytics search queries
• Integrate Application logging and exception handling with Application Insights

Real Life Example

Logicworks is a Microsoft Azure Gold Partner that helps companies migrate their applications to Azure. In the example below, one of our customers was looking to deploy and scale their public-facing web application on AKS in order to solve for the following business use case:

• Achieve portability across on-prem and public clouds
• Accelerate containerized application development
• Unify development and operational teams on a single platform
• Take advantage of native integration into the Azure ecosystem to easily achieve:
• Enterprise-Grade Security
• Azure Active Directory integration
• Track, validate, and enforce compliance across Azure estate and AKS clusters
• Hardened OS images for nodes
• Operational Excellence
• Achieve high availability and fault tolerance through the use of availability zones
• Elastically provision compute capacity without needing to automate and manage underlying infrastructure.
• Gain insight into and visibility into your AKS environment through automatically configured control plane telemetry, log aggregation, and container health.

The customer’s architecture includes a lot of the common best practices to ensure we can meet the customers business and operational requirements:

Cluster Multi-Tenancy

SDLC environments are split across two clusters isolating Production from lower level SDLC environments such as dev/stage. The use of namespaces provides the same operation benefits while saving cost and operational complexity by not deploying an AKS cluster per SDLC environment.

Scheduling and Resource Quotas

Since multiple SDLC environments and other applications share the same cluster, it’s imperative that scheduling and resource quotas are established to ensure applications and the services they depend on get the resources required for operation. When combined with cluster autoscaler we can ensure that our applications get the resources they need and that compute infrastructure is scaled in when they need it.

Azure AD integration

Leverages Azure AD to authenticate/authorize users to access and initiate CRUD (create, update, and delete) operations against AKS clusters. AAD integration makes it convenient and easy to unify layers of authentication (Azure and Kubernetes) and provide the right personnel with the level of access they require to meet their responsibilities while adhering to principle of least privilege

Pod Identities

Instead of hardcoding static credentials within our containers, Pod Identity is deployed into the default namespace and dynamically assigns Managed Identities to the appropriate pods determined by label. This provides our example application the ability to write to Cosmos DB and our CI/CD pipelines the ability to deploy containers to production and stage clusters.

Ingress Controller

Ingress controllers bring traffic into the AKS cluster by creating ingress rules and routes, providing application services with reverse proxying, traffic routing/load balancing, and TLS termination. This allows us to evenly distribute traffic across our application services to ensure scalability and meet reliability requirements.

Monitoring

Naturally, monitoring the day-to-day performance and operations of our AKS clusters is key to maintaining uptime and proactively solving potential issues. Using AKS’ toggle-based implementation, application services hosted on the AKS cluster can easily be monitored and debugged using Azure Monitor.

WhiteSource streamlines application delivery and development with Microsoft Azure and Azure Kubernetes Service

WhiteSource is a Microsoft partner and global leader in open-source component security and compliance management, also referred to as Software Composition Analysis. The company has more than 700 customers worldwide, including many of the Fortune 100. To increase the portability of its solution, WhiteSource moved to containerized development using Kubernetes, and it chose Microsoft Azure and Azure Kubernetes Service (AKS) as its development and delivery platform. Using AKS, WhiteSource benefits from easy deployment, stellar support, and efficient management of Kubernetes clusters.

Azure Kubernetes Service provides all the functionality to give our customers exactly what they need … you can have a cluster up and running in a few minutes due to the intuitiveness of the product.

Uzi Yossef: Senior DevOps Engineer

WhiteSource

Using open-source software has become a necessity — you simply can’t develop at today’s pace without it. WhiteSource research shows that 60 to 80 percent of the code base in applications developed in 2018 is open source. The problem is that security information about open-source components is scattered across many sources with varied levels of credibility. According to a report from WhiteSource, vulnerabilities are developers’ top concerns when working with these components. WhiteSource helps mitigate those concerns about security with a comprehensive, end-to-end open-source management solution running in Microsoft Azure.

The right technology for product improvements

The high quality of WhiteSource’s cloud-based solution has led to rapid growth, and the company saw the need to change its application delivery mechanism to better support its customers’ needs. Containerized development with Kubernetes provided the answer. “With Kubernetes, we can easily distribute containers for faster deployments and rollbacks,” says Uzi Yossef, Senior DevOps Engineer at WhiteSource. “I can have my entire environment set up as a manifest file, which makes it straightforward to keep everything in one place and under source control. For our customers, using Kubernetes makes our product cloud-agnostic, so it can work with whatever cloud they have.”

After initially trying a different Kubernetes product, WhiteSource chose to deploy its solution in Azure using Microsoft Azure Kubernetes Service (AKS), which offered a wide range of advantages. “AKS is much easier to use than other Kubernetes offerings,” says Yossef. “We don’t have to do the manual integrations that are necessary with other products. AKS has the fast scalability we need and the ability to connect to the serverless container framework, so we’re moving more and more workloads to it.”

By using serverless computing within Azure, WhiteSource gains the advantages of running Kubernetes as a service without the overhead of building and maintaining its own managed cluster — Azure handles the provisioning, scaling, and management of resources. This is a key feature for WhiteSource, due to the sensitive nature of the information its solution handles.

“We’re working with our customers’ application data, so it’s important that we can quickly scale out the necessary resources and then immediately delete them when execution is complete,” explains Yossef. “Azure Kubernetes Service provides all the functionality to give our customers exactly what they need, and more importantly, you can have a cluster up and running in a few minutes due to the intuitiveness of the product.”

A comprehensive cloud stack and a strong partnership

WhiteSource has found the cohesive, end-to-end nature of the Azure stack to be another advantage. The company is using Azure DevOps, Azure Application Gateway, Azure Blob storage, Azure Container Registry, and Azure Database for MySQL. “We see Azure as a very mature, integrated environment that helps accelerate delivery of our solution and provide seamless, rolling updates,” says Yossef. “It’s great that we can get all this functionality from a single vendor.”

Throughout its AKS deployment and adoption, WhiteSource has been able to rely on support from Microsoft to remain focused on its main goal of pleasing its customers. “We trust the direction Microsoft is going with Azure and AKS, and that helps us focus on our own innovation,” says Yossef. “We’ve gotten amazing technical help from the Microsoft global black belt team, and there are lots of examples and walkthroughs that made it easy to get up and running with AKS. Everything is well documented, and that’s not the case with other services.”

WhiteSource is a member of the Microsoft Partner Network — and Microsoft is a WhiteSource customer. The companies’ relationship has been a classic win-win scenario. WhiteSource has found some pleasant surprises in working with Microsoft. “Microsoft is a huge enterprise, and you often think of such companies as being heavy and slow moving,” says Maya Rotenberg, Vice President of Marketing at WhiteSource. “But we’ve found Microsoft to be a very open and flexible company that invests in its partners like no other company we’ve worked with. We get amazing support on the technical side and the go-to-market side — we’re on this journey together.”

Summary

Azure Kubernetes Service is a powerful service for running containers in the cloud. Best of all, you only pay for the VMs and other resources consumed, not for AKS itself, so it’s easy to try out. With the best practices described in this post and the AKS Quickstart, you should be able to launch a test cluster in under an hour and see the benefits of AKS for yourself.

Thankyou! keep learning! keep growing! keep sharing!

Krithika Sharma
If you enjoyed this, follow me on Medium for more
Let’s connect on LinkedIn