K8S牛刀小试CI之Jenkins on K8S篇(三)

虚拟化
, ,

上一篇讲的主要是docker上jenkins相关的CI/CD.这一篇主要讲如何在k8上完成之前的功能,并利用k8的特点为jenkins提供强大的调度效率.

0x00. Jenkins on K8

Jenkins一般采用主-从集群模式. 类似k8一样,jenkins-master是不负载任务的. 只负责调度,jenkins-slave负责构建.那么这个架构在k8上如何快速构建呢? 这里我们最好使用jenkins提供的 kubernetes plugin . 安装重启. (注意k8插件安装时如果网络不好会很长.建议开代理?) 插件github文档参考

首先看看我们插件的主要配置,进入k8插件 (路径: 系统管理-系统设置-云-Kubernetes:)

  • k8的api-server的url
  • k8的https安全验证
  • k8的命名空间
  • jenkins的master-url
  • 镜像- k8的pod模板(这个其实就是pod的yaml图形化模板,里面可以添加许多参数)

那么大致流程就是:

  1. 先在harbor仓库放入最新的master(blueocean) 和 jnlp-slave镜像. (√)
  2. 使用kubectl create -f xx 创建jenkins主从相关的*.yaml 文件.
  3. 在jenkins-master 中创建一个pipeline(通过blue ocean更直观简单)
  4. 在jenkins-master 的控制面板的k8插件里面配置相关参数以实现动态分配资源构建.

我们从第二步开始走:

先想想把docker应用转到k8准备需要哪些yaml文件: Deploy/rc部署镜像 , pv/pvc(存储映射) , svc(服务绑定)等等. 这个还是参考github一下为好..

master的dp.yaml 文件.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
#A版本dp
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: jenkins
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 2
      maxUnavailable: 0
  template:
    metadata:
      labels:
        name: jenkins
    spec:
      serviceAccountName: jenkins
      imagePullSecrets:
      - name: harbor
      nodeSelector:
        jenkins: master
      containers:
      - name: jenkins
        image: 172.31.21.226/ideal/jenkins:lts
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
          name: web
          protocol: TCP
        - containerPort: 50000
          name: agent
          protocol: TCP
        volumeMounts:
        - name: jenkinshome
          mountPath: /var/jenkins_home
        env:
        - name: JAVA_OPTS  #这个可以不要么,意义是同步时差?
          value: "-Duser.timezone=Asia/Shanghai"
      volumes:
      - name: jenkinshome
        persistentVolumeClaim:
          claimName: jenkins-pvc


#这是另一个B版本的,使用cepth作为存储,可以对比参考. (下面都会放两份)
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: jenkins
  namespace: cicd
  labels:
    app: jenkins
spec:
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: jenkins
        tier: jenkins
    spec:
      containers:
      - image: chadmoon/jenkins-docker-kubectl:latest
        name: jenkins
        securityContext:
          privileged: true
        ports:
        - containerPort: 8080
          name: jenkins
        - containerPort: 50000
          name: agent
          protocol: TCP
        volumeMounts:
        - name: docker
          mountPath: /var/run/docker.sock
        - name: jenkins-persistent-storage
          mountPath: /root/.jenkins
        - name: kube-config
          mountPath: /root/.kube/config
        - name: image-registry
          mountPath: /root/.docker
      volumes:
      - name: docker
        hostPath:
          path: /var/run/docker.sock
      - name: jenkins-persistent-storage
        persistentVolumeClaim:
          claimName: jenkins-pvc
      - name: kube-config
        hostPath:
          path: /root/.kube/config
      - name: image-registry
        configMap:
          name: image-registry-auth

jenkins.pv.yaml 和jenkins.pvc.yaml 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
#A版pv
apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkins-pv
  labels:
    type: jenkins
spec:
  capacity:
    storage: 50Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /data/jenkins

#A版pvc
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
  selector:
    matchLabels:
      type: jenkins

#B版pv


#B版pvc
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
 name: jenkins-pvc
 namespace: cicd
spec:
 accessModes:
    - ReadWriteOnce
 resources:
   requests:
     storage: 20Gi
 storageClassName: ceph-web

jenkins.svc.yaml 和jenkins.ingress.yaml (可选,以后肯定要替换)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
#A版svc
kind: Service
apiVersion: v1
metadata:
  name: jenkins
spec:
  ports:
  - port: 8080
    targetPort: 8080
    name: web
  - port: 50000
    targetPort: 50000
    name: agent
  selector:
    name: jenkins

#A版ingress
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: jenkins-web
spec:
  rules:
  - host: jenkins.isoftone.com
    http:
      paths:
      - path: /
        backend:
          serviceName: jenkins
          servicePort: 8080


#B版svc与ingress
apiVersion: v1
kind: Service
metadata:
  name: jenkins-web-ui
  namespace: cicd
  labels:
    app: jenkins
spec:
  ports:
    - port: 80
      targetPort: 8080
      name: web-ui
    - port: 50000
      targetPort: 50000
      name: agent
  selector:
    app: jenkins
    tier: jenkins
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: jenkins-web-ui
  namespace: cicd
spec:
  rules:
  - host: jenkins.com
    http:
      paths:
      - backend:
          serviceName: jenkins-web-ui
          servicePort: 80

A版还有一个service-account.yaml ,这个比较陌生,先猜测一下是不是做权限设置的. 类比一下之前heapster中的heapster-rbac.yaml .(发现类型是ClusterRoleBinding ,说明并不一样)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
# In GKE need to get RBAC permissions first with
# kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin [--user=<user-name>|--group=<group-name>]

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins

---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: jenkins
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get","list","watch"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: jenkins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: jenkins
subjects:
- kind: ServiceAccount
  name: jenkins

0x02.参考文章

推荐先看一下时间早一点的,讲了一下集群原理跟演变过程. 不建议直接上手操作.