Kubecost is responsible for providing cost break down by Spark application. This is responsible for storing and scaling of EKS cluster and spark application metrics We use Prometheus deployment for scraping and ingesting metrics into Amazon Managed Prometheus and Kubecost Prometheus is responsible for monitoring EKS cluster including spark applications in your EKS cluster. Kubernetes metrics server is responsible for aggregating cpu, memory and other container resource usage within your cluster This is responsible for scaling CoreDNS pods in your Kubernetes clusterĬluster Proportional Autoscaler Documentation Kubernetes Cluster Autoscaler automatically adjusts the size of Kubernetes cluster and is available for scaling nodegroups (such as core-node-group) in the cluster Karpenter is nodegroup-less autoscaler that provides just-in-time compute capacity for spark applications on Kubernetes clusters The Amazon EFS Container Storage Interface (CSI) driver provides a CSI interface that allows Kubernetes clusters running on AWS to manage the lifecycle of Amazon EFS file systems. Kube-proxy is available as an EKS add-on and it maintains network rules on your nodes and enables network communication to your spark application podsĮBS CSI driver is available as an EKS add-on and it allows EKS clusters to manage the lifecycle of EBS volumes VPC CNI is available as an EKS add-on and is responsible for creating ENI's and IPv4 or IPv6 addresses for your spark application podsĬoreDNS is available as an EKS add-on and is responsible for resolving DNS queries for spark application and for Kubernetes cluster We don't recommend removing critical add-ons ( Amazon VPC CNI, CoreDNS, Kube-proxy). You can see the complete list of add-ons available below. ![]() ![]() Amazon Elastic File System (EFS), EFS mounts, Kubernetes Storage Class for EFS, and Kubernetes Persistent Volume Claim for mounting Airflow DAGs for Airflow pods.ĪWS for FluentBit is employed for logging, and a combination of Prometheus, Amazon Managed Prometheus, and open source Grafana are used for observability.Kubernetes service accounts and AWS IAM roles for service account (IRSA) for Airflow Webserver, Airflow Scheduler, and Airflow Worker.Amazon RDS PostgreSQL instance and security group for Airflow meta database.e.g., Cluster Autoscaler, CoreDNS, Observability, Logging etc.Īpache Airflow core components (with airflow-core.tf): Core Node group with 3 instances spanning multi-AZs for running Apache Airflow and other system critical pods.In terms of infrastructure, below are the resources that are created by this pattern:ĮKS Cluster Control plane with public endpoint (recommended for demo/poc environment) We recommend keeping the defaults and only customize if you have viable alternative option available for replacement. This pattern uses opinionated defaults to keep the deployment experience simple but also keeps it flexible so that you can pick and choose necessary add-ons during deployment. This blueprint deploys Airflow on Amazon EKS managed node groups and leverages Karpenter to run the workloads. This pattern deploys self-managed Apache Airflow deployment on EKS. Self-managed Apache Airflow deployment on Amazon EKS Introduction
0 Comments
Leave a Reply. |