Kubernetes

What is Kubernetes?

Here are the key ideas someone should know about Kubernetes:

• Kubernetes is an open-source container orchestration system for automating application deployment, scaling, and management. It allows you to deploy and manage containerized applications easily.

• Kubernetes coordinates clusters of nodes (physical or virtual machines) and deploys containers across them. It handles scheduling onto nodes in the cluster and actively manages workloads to ensure the desired state of the applications is maintained.

• Kubernetes uses pods to deploy containers. A pod is a group of one or more containers that are guaranteed to run together on the same node. Pods are the smallest deployable units in Kubernetes.

• Kubernetes has a collection of resources you create to specify what you want to deploy and how to deploy it:

  • Pods - As mentioned, pods represent a single instance of an application in Kubernetes.

  • Deployments - A deployment controls pods and provides rollout and rollback functionality for pods.

  • Services - A service exposes a set of pods to external traffic and load balances traffic between them.

  • Volumes - A volume persists data beyond the lifecycle of a pod.

  • Namespaces - Namespaces partition resources into logically named groups.

• Kubernetes has a control plane that makes global decisions about what to deploy where. The Kubernetes control plane includes the API server, scheduler, and controllers.

• Kubernetes supports application portability across infrastructure providers. You can deploy Kubernetes anywhere from on-premise environments to all major cloud platforms.

• Kubernetes has a strong ecosystem of supporting tools like Helm for package management, Prometheus for monitoring, Flow for network management, etc.

• Kubernetes continues to gain popularity and is becoming the de facto standard for container orchestration. Many major tech companies are using Kubernetes in production.

What are additional details about a pod?

A pod in Kubernetes can contain one or more containers that are guaranteed to be co-located on the same node and share some resources.

Some key things to know about pods:

• Pods are the smallest deployable units of computing that can be created and managed in Kubernetes. They represent a single instance of an application in Kubernetes.

• The containers within a pod share an IP address and port space, can discover each other and communicate using localhost, and can share files mounted into the pod.

• Pods inherit the security context, storage, and network configurations of their nodes.

• Pods are meant to be "ephemeral" - they can be created, assigned a unique ID (UID), and then destroyed - and are not durable entities. Pods should not be used to maintain state or share data between containers.

• Best practices are to have only one container per pod (???), with some exceptions:

• Container initializing helper containers: Containers that monitor or enable another main container.

• Tightly coupled containers: Containers that function as a single cohesive application (e.g. MySQL server + MySQL client).

• Ambassador containers: Proxy containers that manage outbound network connectivity for main application containers.

• When there are multiple containers in a pod, Kubernetes manages them as a single entity and all containers are placed on the same worker node. However, each container has a separate startup sequence and individual health checks.

• Containers in a pod have some shared resources like network interfaces and storage volumes. A pod serves as a higher-level wrapper around one or more containers that need to function tightly together.

What are details about namespaces?

Namespaces are a way to partition resources in Kubernetes. They allow you to split a physical cluster into multiple virtual clusters. Some reasons to use namespaces are:

• Isolation - Separate environments for development, testing, and production. Each environment gets its own namespace.

• Encryption - Require encryption for one namespace but not another.

• Resource quotas - Limit resources like CPU and memory for a namespace.

• Access control - Use Kubernetes role-based access control to grant users access to only the necessary namespaces.

Some properties of namespaces:

• They are logical divisions - Namespaces do not have hard separation and are more for convenience. Namespace resources still compete for and share underlying physical resources.

• Each resource in Kubernetes belongs to exactly one namespace.

• You can specify a namespace when creating a resource using --namespace <name>. If you omit the namespace, the resource is created in the default namespace.

• Each namespace has quotas, limits, and access control policies.

• You can list namespaces with kubectl get namespaces.

• There are a few reserved namespaces:

  -  default - Resources without a specified namespace go here.

  -  kube-system - For resources created by the Kubernetes system.

  -  kube-public - For public resources.

• You can create custom namespaces as needed. For example, you can have:

  -  dev, test, prod for different environments.

  -  team-a, team-b for different teams.

  -  app1, app2 for different applications.

• You can delete namespaces when no longer needed. Deleting a namespace deletes all resources within that namespace!

How is state maintained in Kubernetes?

State maintenance in Kubernetes can be challenging since pods are ephemeral (can be destroyed and recreated).

There are a few main ways to manage state in Kubernetes:

• Volumes - Volumes are the recommended way to persist data in Kubernetes pods. There are many volume types, including:

  -  PersistentVolumeClaims (PVCs) - Claim storage space from a PersistentVolume. Usually supports ReadWriteOnce access, where the volume can be mounted by a single pod.

  • Storage Classes - Provide a way for administrators to describe types of storage to users. The user can then request storage with a PersistentVolumeClaim, and get a persistent volume with matching storage without knowing the details.

  • Persistent Volumes (PVs) - The actual storage mechanism, such as NFS, iSCSI, or a cloud disk. Backed by physical storage options like GCE disks, Azure disks, etc.

• StatefulSets - Provide stable storage and networking for pods. StatefulSets launch pods with a predictable name and ordinal index. The pods also have sticky identities (DNS hostnames), stable storage (using PVCs), and stable networking (each pod gets a single IP address).

• Secrets - Can be used to store sensitive state data and expose to pods as needed. The secrets can be simple key-value pairs or files/docker config.

• ConfigMaps - Also hold configuration data that can be consumed by pods, but are less sensitive than secrets. ConfigMaps are ideal for storing non-confidential configuration information like environment variables.

• Service for Pods - By creating a Service that targets pods in a StatefulSet, pods get a stable DNS name using the pattern <servicename>-<ordinalID>.pod. This can be used to maintain stable identities for pods.

• Init Containers - Can be used to set up volumes before the main pod container runs. Good for scenarios like migrating data from an external source into a Kubernetes volume.

• Databases and Storage Services - For more complicated state needs, you can deploy dedicated database systems or storage services on Kubernetes using PersistentVolumes and StatefulSets. Popular options are PostgreSQL, MySQL, MongoDB, Redis, etc.

What are resources in Kubernetes?

Kubernetes resources are objects you create to define the infrastructure and workloads for your applications.

There are many types of Kubernetes resources, including:

• Pods - The smallest deployable units in Kubernetes. Pods encapsulate container(s), storage resources, unique network IP, and options that govern how containers should run.

• Deployments - Define and monitor pods and also help with scaling and rolling updates of pods.

• StatefulSets - Similar to Deployments but maintain pod identity and state, useful for stateful applications like databases.

• DaemonSets - Ensure that all (or some) nodes run a copy of a pod. Used for operators and logging apps, among others.

• Secrets - Securely store sensitive data and expose as environment variables or files to pods that need access.

• ConfigMaps - Store non-confidential configuration data and expose to pods as environment variables, command-line arguments, or files.

• PersistentVolumes (PVs) - Represent storage resources (e.g. GCE disks) and how they are shared between pods.

• PersistentVolumeClaims (PVCs) - Request for storage and how much/how it should be accessed. Usually bound to a PV.

• StorageClasses - Define different storage "classes" that can be requested in PVCs. Usually backed by a particular PersistentVolume plugin/provisioner.

• Services - Acts as an ambassador for pods by enabling them to be accessed via a DNS name and IP address. Also enables load balancing across pods.

• Ingresses - Exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. Uses a load balancer and routing rules.

• Roles & Role Bindings - Define RBAC (Role-Based Access Control) to control resource access in Kubernetes for users/groups.

• Namespaces - Support multiple virtual clusters backed by the same physical cluster. Each namespace has its own set of resources.

Kubernetes resources allow you to describe your application's infrastructure and deployment in declarative configuration files which can be reviewed, edited, and version controlled. Kubernetes then manages the resource to match the desired state you have specified.

What does the scheduler do?

The Kubernetes scheduler is responsible for assigning pods to nodes in the cluster. It monitors the cluster state and scheduling requirements to determine the best node for each pod.

Some of the specific responsibilities of the Kubernetes scheduler include:

• Filtering nodes that do not meet the scheduling requirements for a pod (e.g. not enough resources, incorrect operating system, etc.)

• Prioritizing nodes that best match the scheduling requirements and then ranking the nodes. Scheduling priorities include resource requests, resource quality of service, affinity and anti-affinity requirements, data locality, spread/packing constraints, etc.

• Selecting a node for the pod from the prioritized list of candidates based on resource availability and other constraints.

• Optimizing workload spreading and balancing based on requirements for a pod (if any are specified). This aims to maximize resource utilization while avoiding overload.

• Monitoring pod scheduling metrics to help make ongoing scheduling decisions. Things like pod pending time, resource availability, etc.

• Allowing you to configure scheduling policies based on your workload needs. You can specify node selectors, affinity rules, resource requirements, etc. for pods to influence scheduling.

• Supporting scheduling frameworks like Kubernetes native Quality of Service, Kubernetes Scheduler Profiles, and kube-arbitrator. These allow more advanced resource allocation and workload orchestration.

• Maintaining fault tolerance in edge cases. For example, if a node dies or resources are released while the scheduler is making a decision, it will re-evaluate to find a safe placement for the pod.

• Integrating with scheduling plugins and kubelets on each node to gather resource availability and metrics.

• Constantly re-evaluating cluster state to look for pods that need to be moved or re-scheduled due to changes in node health, capacity, policies, etc.

How does Kubernetes handle node degradation?

Kubernetes has several mechanisms to handle performance degradation or failures of nodes:

1. Node Conditions - Each node has conditions that track different aspects of node health like ready status, out of disk, network unavailable, etc. The scheduler uses node conditions to filter out unhealthy nodes when scheduling pods.

2. Node Taints - Kubernetes administrators can taint nodes to prevent new pods from scheduling onto them. This is useful when a node is having issues so you can stop feeding it more workload. Existing pods will continue running unless evicted.

3. Pod Eviction - As a last resort, Kubernetes may evict pods from a node to preserve cluster stability. This usually only happens if a node has severe performance issues or failures and cannot support its current pods. Static pods are evicted last since they are required for cluster control plane operations.

4. Pod Disruption Budgets - To limit how many pods can be evicted from a node at once, administrators set up pod disruption budgets. This helps avoid overwhelming other nodes by rapidly rescheduling too many pods from a degraded node.

5. Node Draining - Administrators can manually "drain" a node to reschedule all pods from it. Node draining uses pod disruption budgets to do this gradually. Draining nodes is useful when decommissioning a node or performing maintenance on it.

6. Replica Scaling - For workloads like Deployments that maintain a set replica count, Kubernetes will scale up additional pods on other available nodes if pods on a degraded node cannot run properly or go down. This maintains the desired state of the workload.

7. Pod Re-scheduling - The Kubernetes scheduler is constantly monitoring for pods that may need to be re-scheduled onto new nodes for performance or availability reasons. If it recognizes a pod is having issues due to node performance, it will find a more stable node to place the pod on.

8. Node Auto-Repair - Some Kubernetes environments use node auto-repair tools to automatically roll nodes that have gone down or are exhibiting performance issues. The tool drains the node, walks it through any necessary repairs or reprovisioning, and then puts it back into rotation - helping maximize cluster resilience.