helm部署jenkins到k8s并创建pipeline构建项目

时间： 2023-08-22 admin IT培训

helm部署jenkins到k8s并创建pipeline构建项目推荐度：
相关推荐

安装部署nfs

1. nfs服务器创建目录

[yeqiang@harbor ~]$ sudo mkdir /home/nfs/jenkins -p

2. 设置其他用户可以写入该目录

[yeqiang@harbor ~]$ sudo chmod o+w /home/nfs/jenkins/ -R

3. 配置/etc/exports

/home/nfs/jenkins 192.168.1.167(rw,sync,no_subtree_check,no_root_squash)
/home/nfs/jenkins/agent 192.168.1.167(rw,sync,no_subtree_check,no_root_squash)

4. 启动nfs服务，配置开启自启动

[yeqiang@harbor ~]$ sudo systemctl enable nfs-server --now
Created symlink /etc/systemd/system/multi-user.target.wants/nfs-server.service → /usr/lib/systemd/system/nfs-server.service.

5. 检查服务及端口(111/tcp)状态

[yeqiang@harbor ~]$ netstat -lnetp | grep 111
(Not all processes could be identified, non-owned process infowill not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      0          19646      -                   
tcp6       0      0 :::111                  :::*                    LISTEN      0          19650      -                   
[yeqiang@harbor ~]$ sudo exportfs -v
/home/nfs/jenkins192.168.1.167(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
/home/nfs/jenkins/agent192.168.1.167(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)

创建k8s存储

1. 创建StorageClass，配置文件nfs-storege-class.yml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:name: nfs-storageannotations:storageclass.kubernetes.io/is-default-class: 'false'
provisioner: kubernetes.io/no-provisioner
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

执行创建

[yeqiang@harbor k8s]$ kubectl apply -f nfs-storege-class.yml 
storageclass.storage.k8s.io/nfs-storage created
[yeqiang@harbor k8s]$ kubectl get storageclass
NAME                      PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
nfs-storage               kubernetes.io/no-provisioner   Delete          WaitForFirstConsumer   false                  6s

2. 创建PersistantVolume，配置文件jenkins-pv.yml

apiVersion: v1
kind: PersistentVolume
metadata:name: jenkins-pv
spec:capacity:storage: 10GiaccessModes:- ReadWriteOncevolumeMode: FilesystempersistentVolumeReclaimPolicy: RecyclestorageClassName: "nfs-storage"nfs:# 根据实际共享目录修改path: /home/nfs/jenkins# 根据实际 nfs服务器地址修改server: 192.168.1.167

执行创建

[yeqiang@harbor k8s]$ kubectl apply -f jenkins-pv.yml 
persistentvolume/jenkins-pv created
[yeqiang@harbor k8s]$ kubectl get pv 
NAME                          CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                               STORAGECLASS    REASON   AGE
jenkins-pv                    10Gi       RWO            Recycle          Available                                       nfs-storage              6s

helm部署jenkins

1. helm查看jenkins信息

[yeqiang@harbor k8s]$ helm repo add jenkins 
"jenkins" has been added to your repositories
[yeqiang@harbor k8s]$ helm search repo jenkins
NAME           	CHART VERSION	APP VERSION	DESCRIPTION                                       
jenkins/jenkins	2.15.1       	lts        	Open source continuous integration server. It s...
stable/jenkins 	2.5.4        	lts        	DEPRECATED - Open source continuous integration...

注意stable/jenkins已经标注过时了，我们选择jenkins/jenkins

2. 创建namespace

[yeqiang@harbor k8s]$ kubectl create ns jenkins
namespace/jenkins created

3. helm安装

[yeqiang@harbor k8s]$ helm install jenkins jenkins/jenkins -n jenkins --set persistence.storageClass="nfs-storage"
NAME: jenkins
LAST DEPLOYED: Thu Oct 29 10:25:54 2020
NAMESPACE: jenkins
STATUS: deployed
REVISION: 1
NOTES:
1. Get your 'admin' user password by running:printf $(kubectl get secret --namespace jenkins jenkins -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode);echo
2. Get the Jenkins URL to visit by running these commands in the same shell:export POD_NAME=$(kubectl get pods --namespace jenkins -l "app.kubernetes.io/component=jenkins-master" -l "app.kubernetes.io/instance=jenkins" -o jsonpath="{.items[0].metadata.name}")echo http://127.0.0.1:8080kubectl --namespace jenkins port-forward $POD_NAME 8080:80803. Login with the password from step 1 and the username: admin4. Use Jenkins Configuration as Code by specifying configScripts in your values.yaml file, see documentation: http:///configuration-as-code and examples:  more information on running Jenkins on Kubernetes, visit:

For more information about Jenkins Configuration as Code, visit:
/

参数说明：

persistence.storageClass 赋值为刚才创建的StorageClass名称nfs-sotrage

4. 故障排除

[yeqiang@harbor k8s]$ kubectl get pods -n jenkins
NAME                       READY   STATUS    RESTARTS   AGE
jenkins-58978c6d6d-dszqz   1/2     Running   2          6m8s
[yeqiang@harbor k8s]$ kubectl describe pod jenkins-58978c6d6d-dszqz -n jenkins
Name:         jenkins-58978c6d6d-dszqz
Namespace:    jenkins
Priority:     0
Node:         192.168.1.167/192.168.1.167
Start Time:   Thu, 29 Oct 2020 10:25:55 +0800
Labels:       app.kubernetes.io/component=jenkins-masterapp.kubernetes.io/instance=jenkinsapp.kubernetes.io/managed-by=Helmapp.kubernetes.io/name=jenkinspod-template-hash=58978c6d6d
Annotations:  checksum/config: 9823a20535eb4e992b6741805b204c7c82d33080079887578e27c24328df7b29
Status:       Running
IP:           172.20.0.108
IPs:IP:           172.20.0.108
Controlled By:  ReplicaSet/jenkins-58978c6d6d
Init Containers:copy-default-config:Container ID:  docker://e29d8d6285ddebaf565a206213da47aac723ae79a883ff6108aafd7108a5b1fdImage:         jenkins/jenkins:ltsImage ID:      docker-pullable://jenkins/jenkins@sha256:aa1ec2a6a106608a286dc06c334053fced1df87a7cb307853ca1cf8fa48ba09bPort:          <none>Host Port:     <none>Command:sh/var/jenkins_config/apply_config.shState:          TerminatedReason:       CompletedExit Code:    0Started:      Thu, 29 Oct 2020 10:25:58 +0800Finished:     Thu, 29 Oct 2020 10:26:40 +0800Ready:          TrueRestart Count:  0Limits:cpu:     2memory:  4GiRequests:cpu:     50mmemory:  256MiEnvironment:ADMIN_PASSWORD:  <set to the key 'jenkins-admin-password' in secret 'jenkins'>  Optional: falseADMIN_USER:      <set to the key 'jenkins-admin-user' in secret 'jenkins'>      Optional: falseMounts:/tmp from tmp (rw)/usr/share/jenkins/ref/plugins from plugins (rw)/var/jenkins_config from jenkins-config (rw)/var/jenkins_home from jenkins-home (rw)/var/jenkins_plugins from plugin-dir (rw)/var/run/secrets/kubernetes.io/serviceaccount from jenkins-token-krt5l (ro)
Containers:jenkins:Container ID:  docker://13cb74bdc25782b5bf7258fc4c369544571866c3a029dbc26f9ae489fb7b742bImage:         jenkins/jenkins:ltsImage ID:      docker-pullable://jenkins/jenkins@sha256:aa1ec2a6a106608a286dc06c334053fced1df87a7cb307853ca1cf8fa48ba09bPorts:         8080/TCP, 50000/TCPHost Ports:    0/TCP, 0/TCPArgs:--argumentsRealm.passwd.$(ADMIN_USER)=$(ADMIN_PASSWORD)--argumentsRealm.roles.$(ADMIN_USER)=admin--httpPort=8080State:          RunningStarted:      Thu, 29 Oct 2020 10:31:24 +0800Last State:     TerminatedReason:       ErrorExit Code:    143Started:      Thu, 29 Oct 2020 10:29:01 +0800Finished:     Thu, 29 Oct 2020 10:31:18 +0800Ready:          FalseRestart Count:  2Limits:cpu:     2memory:  4GiRequests:cpu:      50mmemory:   256MiLiveness:   http-get http://:http/login delay=90s timeout=5s period=10s #success=1 #failure=5Readiness:  http-get http://:http/login delay=60s timeout=5s period=10s #success=1 #failure=3Environment:POD_NAME:                  jenkins-58978c6d6d-dszqz (v1:metadata.name)JAVA_OPTS:                 -Dcasc.reload.token=$(POD_NAME)JENKINS_OPTS:              JENKINS_SLAVE_AGENT_PORT:  50000ADMIN_PASSWORD:            <set to the key 'jenkins-admin-password' in secret 'jenkins'>  Optional: falseADMIN_USER:                <set to the key 'jenkins-admin-user' in secret 'jenkins'>      Optional: falseCASC_JENKINS_CONFIG:       /var/jenkins_home/casc_configsMounts:/tmp from tmp (rw)/usr/share/jenkins/ref/plugins/ from plugin-dir (rw)/var/jenkins_config from jenkins-config (ro)/var/jenkins_home from jenkins-home (rw)/var/jenkins_home/casc_configs from sc-config-volume (rw)/var/run/secrets/kubernetes.io/serviceaccount from jenkins-token-krt5l (ro)jenkins-sc-config:Container ID:   docker://a33d84511d702b29e1d1b206793666bb360f071e0f7577a24f660c8a0d4eef1bImage:          kiwigrid/k8s-sidecar:0.1.193Image ID:       docker-pullable://kiwigrid/k8s-sidecar@sha256:170069ff097679f8fafcf01510d1fe05260af683b949fd2b05866e18ca4d9c74Port:           <none>Host Port:      <none>State:          RunningStarted:      Thu, 29 Oct 2020 10:26:48 +0800Ready:          TrueRestart Count:  0Environment:POD_NAME:           jenkins-58978c6d6d-dszqz (v1:metadata.name)LABEL:              jenkins-jenkins-configFOLDER:             /var/jenkins_home/casc_configsNAMESPACE:          jenkinsREQ_URL:            http://localhost:8080/reload-configuration-as-code/?casc-reload-token=$(POD_NAME)REQ_METHOD:         POSTREQ_RETRY_CONNECT:  10Mounts:/var/jenkins_home from jenkins-home (rw)/var/jenkins_home/casc_configs from sc-config-volume (rw)/var/run/secrets/kubernetes.io/serviceaccount from jenkins-token-krt5l (ro)
Conditions:Type              StatusInitialized       True Ready             False ContainersReady   False PodScheduled      True 
Volumes:plugins:Type:       EmptyDir (a temporary directory that shares a pod's lifetime)Medium:     SizeLimit:  <unset>tmp:Type:       EmptyDir (a temporary directory that shares a pod's lifetime)Medium:     SizeLimit:  <unset>jenkins-config:Type:      ConfigMap (a volume populated by a ConfigMap)Name:      jenkinsOptional:  falseplugin-dir:Type:       EmptyDir (a temporary directory that shares a pod's lifetime)Medium:     SizeLimit:  <unset>jenkins-home:Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)ClaimName:  jenkinsReadOnly:   falsesc-config-volume:Type:       EmptyDir (a temporary directory that shares a pod's lifetime)Medium:     SizeLimit:  <unset>jenkins-token-krt5l:Type:        Secret (a volume populated by a Secret)SecretName:  jenkins-token-krt5lOptional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300snode.kubernetes.io/unreachable:NoExecute for 300s
Events:Type     Reason     Age                    From                   Message----     ------     ----                   ----                   -------Normal   Scheduled  6m10s                  default-scheduler      Successfully assigned jenkins/jenkins-58978c6d6d-dszqz to 192.168.1.167Normal   Pulling    6m9s                   kubelet, 192.168.1.167  Pulling image "jenkins/jenkins:lts"Normal   Pulled     6m8s                   kubelet, 192.168.1.167  Successfully pulled image "jenkins/jenkins:lts"Normal   Created    6m7s                   kubelet, 192.168.1.167  Created container copy-default-configNormal   Started    6m7s                   kubelet, 192.168.1.167  Started container copy-default-configNormal   Pulled     5m18s                  kubelet, 192.168.1.167  Successfully pulled image "jenkins/jenkins:lts"Normal   Created    5m18s                  kubelet, 192.168.1.167  Created container jenkinsNormal   Started    5m17s                  kubelet, 192.168.1.167  Started container jenkinsNormal   Pulled     5m17s                  kubelet, 192.168.1.167  Container image "kiwigrid/k8s-sidecar:0.1.193" already present on machineNormal   Created    5m17s                  kubelet, 192.168.1.167  Created container jenkins-sc-configNormal   Started    5m17s                  kubelet, 192.168.1.167  Started container jenkins-sc-configWarning  Unhealthy  3m11s (x7 over 4m11s)  kubelet, 192.168.1.167  Readiness probe failed: Get http://172.20.0.108:8080/login: dial tcp 172.20.0.108:8080: connect: connection refusedNormal   Pulling    3m7s (x2 over 5m24s)   kubelet, 192.168.1.167  Pulling image "jenkins/jenkins:lts"Normal   Killing    3m7s                   kubelet, 192.168.1.167  Container jenkins failed liveness probe, will be restartedWarning  Unhealthy  67s (x8 over 3m47s)    kubelet, 192.168.1.167  Liveness probe failed: Get http://172.20.0.108:8080/login: dial tcp 172.20.0.108:8080: connect: connection refused

Liveness probe failed：

把deployments里面的jenkins配置项目failureThreshold改为10

原因：jenkins默认引用的国外更新源，初始化时间较长，pod还未就绪就被k8s杀死了。

可能应为网络问题，pod可能还会反创建。大约十几分钟后。可以正常访问

5. 访问Jenkins

查看访问地址

[yeqiang@harbor k8s]$ kubectl get svc -n jenkins
NAME            TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)     AGE
jenkins         ClusterIP   10.68.62.85   <none>        8080/TCP    26m
jenkins-agent   ClusterIP   10.68.46.25   <none>        50000/TCP   26m

查看登陆用户及密码

[yeqiang@harbor k8s]$ kubectl edit secret jenkins -n jenkins

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:jenkins-admin-password: clM5bXgxMUNvQw==jenkins-admin-user: YWRtaW4=
kind: Secret
metadata:annotations:meta.helm.sh/release-name: jenkinsmeta.helm.sh/release-namespace: jenkinscreationTimestamp: "2020-10-29T02:25:54Z"labels:app.kubernetes.io/component: jenkins-masterapp.kubernetes.io/instance: jenkinsapp.kubernetes.io/managed-by: Helmapp.kubernetes.io/name: jenkinshelm.sh/chart: jenkins-2.15.1managedFields:- apiVersion: v1fieldsType: FieldsV1fieldsV1:f:data:.: {}f:jenkins-admin-password: {}f:jenkins-admin-user: {}f:metadata:f:annotations:.: {}f:meta.helm.sh/release-name: {}f:meta.helm.sh/release-namespace: {}f:labels:.: {}f:app.kubernetes.io/component: {}f:app.kubernetes.io/instance: {}f:app.kubernetes.io/managed-by: {}f:app.kubernetes.io/name: {}f:helm.sh/chart: {}f:type: {}manager: Go-http-clientoperation: Updatetime: "2020-10-29T02:25:54Z"name: jenkinsnamespace: jenkinsresourceVersion: "2557784"selfLink: /api/v1/namespaces/jenkins/secrets/jenkinsuid: e8eb14b7-2e7c-42b4-83b5-39889042e72b
type: Opaque

[yeqiang@harbor k8s]$ echo 'YWRtaW4=' | base64 -d
admin
[yeqiang@harbor k8s]$ echo 'clM5bXgxMUNvQw==' | base64 -d
rS9mx11CoC

得到用户admin，密码rS9mx11CoC

登陆系统

创建jenkins PipeLine构建部署项目

1. 查看jenkins cloud 配置

2. 编写pipeline

def label = "jenkins-slave"
podTemplate(label: label, cloud: 'kubernetes', containers: [containerTemplate(alwaysPullImage: true, args: 'cat', command: '/bin/sh -c', image: 'maven:3.6.3-openjdk-8', livenessProbe: containerLivenessProbe(execArgs: '', failureThreshold: 0, initialDelaySeconds: 0, periodSeconds: 0, successThreshold: 0, timeoutSeconds: 0), name: 'maven', resourceLimitCpu: '', resourceLimitMemory: '', resourceRequestCpu: '', resourceRequestMemory: '', ttyEnabled: true, workingDir: '/home/jenkins/agent')],     workspaceVolume: nfsWorkspaceVolume(serverAddress: '192.168.1.167', serverPath: '/home/nfs/jenkins/agent', readOnly: 'false')
) {node(label) {stage('git') {git '.git'}stage('mvn'){container('maven'){	        sh '''
cat > /usr/share/maven/conf/settings.xml << EOF
<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns=".0.0"xmlns:xsi=""xsi:schemaLocation=".0.0 .0.0.xsd"><localRepository>/root/.m2</localRepository><mirrors><mirror><id>alimaven</id><name>aliyun maven</name><url>/</url><mirrorOf>central</mirrorOf>        </mirror></mirrors>
</settings>
EOF
'''		sh "mvn clean package -Dmaven.test.skip=true"}            }}
}

故障排除：

Could not resolve host

放弃使用镜像maven:3.3.9-jdk-8-alpine，采用maven:3.6.3-openjdk-8。原因：maven:3.3.9-jdk-8-alpine对/etc/resolv.conf解析有故障