🚀 Spark on Kubernetes 部署指南

本指南演示如何在 Kubernetes 上安装并运行 Spark，包括：

安装 Spark Operator
配置 ServiceAccount 和权限
创建持久化存储卷
编写并部署 Spark Application

1️⃣ 安装 Spark Operator

推荐使用 Helm 安装：

# 添加 Helm 仓库
helm repo add spark-operator https://kubeflow.github.io/spark-operator
helm repo update

# 安装 Spark Operator
helm install spark-operator spark-operator/spark-operator \
  --namespace spark-operator \
  --create-namespace \
  --set spark.jobNamespaces="{spark-operator}" \
  --set controller.nodeSelector."kubernetes\.io/hostname"="xxt" \
  --set webhook.nodeSelector."kubernetes\.io/hostname"="xxt"

📦 网络不佳时，可下载源码包安装：

wget https://github.com/kubeflow/spark-operator/releases/download/spark-operator-chart-1.4.6/spark-operator-1.4.6.tgz

helm install spark-operator ./spark-operator-1.4.6.tgz \
  --namespace spark-operator \
  --create-namespace

安装验证：

kubectl get pods -n spark-operator

示例输出：

NAME                              READY   STATUS    RESTARTS   AGE
spark-operator-7dcb44748b-cbjfr   1/1     Running   0          2d18h

确认监听的命名空间：

kubectl get deployment spark-operator-controller -n spark-operator -o yaml | grep namespaces

2️⃣ 配置 ServiceAccount 和权限

Spark Driver 和 Executor 需要具备集群访问权限。

2.1 创建 ServiceAccount

创建 spark-account.yaml：

apiVersion: v1
kind: ServiceAccount
metadata:
  name: spark-account
  namespace: spark-operator

2.2 创建 ClusterRoleBinding

创建 spark-role-binding.yaml：

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: spark-cluster-role-binding
subjects:
  - kind: ServiceAccount
    name: spark-account
    namespace: spark-operator
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

⚠️ 提示： 出于安全考虑，生产环境请使用最小权限原则，避免直接使用 cluster-admin。

2.3 应用配置

kubectl apply -f spark-account.yaml
kubectl apply -f spark-role-binding.yaml

验证：

kubectl get serviceaccount spark-account -n spark-operator
kubectl get clusterrolebinding spark-cluster-role-binding

✅ 成功后，spark-account 已具备权限。

3️⃣ 创建存储卷

如需挂载共享数据，可配置 PV 和 PVC。

3.1 创建 PersistentVolume (PV)

创建 pv.yaml：

apiVersion: v1
kind: PersistentVolume
metadata:
  name: spark-pv
spec:
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteMany
  hostPath:
    path: /Users/xjw/software/spark-operator/data/spark
    type: DirectoryOrCreate
  storageClassName: local-storage

📢 注意： hostPath 仅用于单节点测试环境。

3.2 创建 PersistentVolumeClaim (PVC)

创建 pvc.yaml：

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: spark-pvc
  namespace: spark-operator
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 2Gi
  storageClassName: local-storage

3.3 部署 PV 和 PVC

kubectl apply -f pv.yaml
kubectl apply -f pvc.yaml

验证绑定状态：

kubectl get pv
kubectl get pvc -n spark-operator

4️⃣ 部署 Spark Application

以下示例运行 Spark Pi：

创建 spark-custom.yaml：

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pi-custom-resource
  namespace: spark-operator
spec:
  type: Scala
  mode: cluster
  image: spark:3.5.5
  imagePullPolicy: IfNotPresent
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples_2.12-3.5.5.jar"
  sparkVersion: 3.5.5
  restartPolicy:
    type: Never
  arguments:
    - "10000"
  sparkConf:
    "spark.driver.extraJavaOptions": "-Dlog4j.configurationFile=file:/mnt/data/log4j2.properties"
    "spark.executor.extraJavaOptions": "-Dlog4j.configurationFile=file:/mnt/data/log4j2.properties"
  volumes:
    - name: spark-storage
      persistentVolumeClaim:
        claimName: spark-pvc
    - name: host-time
      hostPath:
        path: /etc/localtime
  driver:
    cores: 1
    memory: 512m
    serviceAccount: spark-account
    nodeSelector:
      kubernetes.io/hostname: xjw
    volumeMounts:
      - name: spark-storage
        mountPath: /mnt/data
      - name: host-time
        mountPath: /etc/localtime
        readOnly: true
    env:
      - name: TZ
        value: Asia/Shanghai
  executor:
    cores: 1
    instances: 2
    memory: 512m
    nodeSelector:
      kubernetes.io/hostname: xjw
    volumeMounts:
      - name: spark-storage
        mountPath: /mnt/data
      - name: host-time
        mountPath: /etc/localtime
        readOnly: true
    env:
      - name: TZ
        value: Asia/Shanghai

部署：

kubectl apply -f spark-custom.yaml

5️⃣ 验证运行与挂载

查看 Application 状态：

kubectl describe sparkapplication spark-pi-custom-resource -n spark-operator

确认挂载：

Mounts:
  /mnt/data from spark-storage (rw)

查看日志：

kubectl logs spark-pi-custom-resource-driver -n spark-operator

若运行成功，日志末尾应包含：

Pi is roughly 3.1460957304786525

6️⃣ 清理资源

删除 Pod 和 Spark Application：

kubectl delete pod spark-pi-custom-resource-driver -n spark-operator
kubectl delete sparkapplication spark-pi-custom-resource -n spark-operator

✨ 小贴士

✅ 版本兼容 spec.sparkVersion 与镜像版本需保持一致。

✅ 访问 Spark UI 可使用端口转发：

kubectl port-forward svc/spark-pi-custom-resource-ui-svc 4040:4040 -n spark-operator

✅ 多租户安全 生产环境建议自定义 Role 而非 cluster-admin。

📚 参考资料

文档信息

本文作者：Xuxiaotuan
本文链接：https://xuyinyin.cn/2025/07/08/spark-on-k8s/
版权声明：自由转载-非商用-非衍生-保持署名（创意共享3.0许可证）

梦洛河

Spark On Kubernetes 部署指南