本环境采用kubeadm安装(为了简单,非二进制环境安装),master节点未做高可用.经典的1master/2node模式.
dawdler-series 终究还是要是要容器化.大势所趋所以被迫搞一下子.
全球第一个关于buildkit+nexus私有库实现的部署文档.
操作系统 | cpu | 内存大小 | ip地址 | 描述 |
---|---|---|---|---|
Fedora-Server-dvd-x86_64-36-1.5 | 4u | 4G | 192.168.43.144 | k8s-master |
Fedora-Server-dvd-x86_64-36-1.5 | 4u | 4G | 192.168.43.146 | k8s-node1 |
Fedora-Server-dvd-x86_64-36-1.5 | 4u | 4G | 192.168.43.147 | k8s-node2 |
Fedora-Server-dvd-x86_64-36-1.5 | 4u | 8G | 192.168.43.137 | nexus |
三台k8s的主机分别做以下设置
vim /etc/hosts
···text 192.168.43.144 k8s-master1 192.168.43.146 k8s-node1 192.168.43.147 k8s-node2 192.168.43.137 my.nexus.org
也可以用 hostnamectl set-hostname k8s-master1
#### 1.2.2 关闭防火墙与selinux
```shell
systemctl stop firewalld
systemctl disable firewalld
setenforce 0 #表示临时关闭selinux防火墙
永久关闭selinux vim /etc/selinux/config
SELINUX=disabled
yum install -y ipvsadm ipset
yum install iproute-tc.x86_64 -y
yum install chrony -y
systemctl enable chronyd
systemctl start chronyd
chronyc sources
vim /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
vm.swappiness=0 #vm.swappiness是操作系统控制物理内存交换出去的策略.它允许的值是一个百分比的值,范围在0-100,该值默认为60.vm.swappiness设置为0表示尽量少swap,100表示尽量将inactive的内存页做交换.
sysctl -p /etc/sysctl.d/k8s.conf
修改/etc/fstab文件,注释掉 SWAP 的自动挂载.
swapoff -a
sed -i '/swap/s/^/#/' /etc/fstab
systemctl mask dev-zram0.swap
sysctl -p
使用free -m确认 swap 已经关闭.
mkdir /etc/sysconfig/modules/
vim /etc/sysconfig/modules/ipvs.modules
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
modprobe br_netfilter
使用lsmod | grep -e ip_vs -e nf_conntrack_命令查看是否已经正确加载所需的内核模块.
改变权限并加载ipvs模块
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://download.docker.com/linux/fedora/docker-ce.repo
yum install containerd.io.x86_64 -y
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
如果不能正常访问互联网(你懂得)的同学则需要替换配置文件
sed -i "s#k8s.gcr.io#registry.cn-hangzhou.aliyuncs.com/google_containers#g" /etc/containerd/config.toml
sed -i 's#SystemdCgroup = false#SystemdCgroup = true#g' /etc/containerd/config.toml
sed -i "s#https://registry-1.docker.io#https://registry.cn-hangzhou.aliyuncs.com#g" /etc/containerd/config.toml
启动Containerd 并设置自启动
systemctl daemon-reload
systemctl enable containerd
systemctl restart containerd
vim /etc/yum.repos.d/kubernetes.repo
不能正常访问互联网(你懂得)的同学用这个源即可
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
可以正常访问互联网(你懂得)的同学建议用下面的源
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-$basearch
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
yum install -y kubelet kubeadm kubectl
systemctl daemon-reload
systemctl enable kubelet && systemctl start kubelet
crictl config runtime-endpoint /run/containerd/containerd.sock
安装nexus之前可以跳过此步骤,安装之后再配置.
mkdir ~/.docker
通过base64输出nexus账号密码
echo -n "admin:123456" | base64
输出: YWRtaW46MTIzNDU2
vim ~/.docker/config.json
{
"auths": {
"https://my.nexus.org:8083": {
"auth": "YWRtaW46MTIzNDU2"
}
}
}
在master节点执行
kubeadm config images list
可以正常访问互联网(你懂得)的可以去除--image-repository
kubeadm config images pull --kubernetes-version=v1.24.2 --image-repository=registry.aliyuncs.com/google_containers
命令行初始化 可以正常访问互联网(你懂得)的可以去除--image-repository
kubeadm init \
--apiserver-advertise-address=192.168.43.144 \
--image-repository registry.aliyuncs.com/google_containers \
--pod-network-cidr=10.244.0.0/16 \
--service-cidr=10.1.0.0/16 \
--kubernetes-version=v1.24.2
配置文件初始化方式
kubeadm config print init-defaults > kubeadm.yaml
编辑 kubeadm.yaml
不可以正常访问互联网(你懂得)的同学请替换 registry.cn-hangzhou.aliyuncs.com/google_containers
kubeadm init --config=kubeadm.yaml
执行完成之后会有以下输出
kubeadm join 192.168.43.144:6443 --token ds62l9.scyt0ise8k9qakae \
--discovery-token-ca-cert-hash sha256:c1d0577681ad0c5dacd7814fc60afd71fe9adb33b3c6168bf93854645d33e33e
设置ConfigMap的kube-system/kube-proxy中的config.conf配置mode: "ipvs"
kubectl edit cm kube-proxy -n kube-system
删除pod(会重新启动新的)
kubectl get pod -n kube-system |grep kube-proxy |awk '{system("kubectl delete pod "$1" -n kube-system")}'
运行kubeadm init输出
在node节点中执行
kubeadm join 192.168.43.144:6443 --token ds62l9.scyt0ise8k9qakae \
--discovery-token-ca-cert-hash sha256:c1d0577681ad0c5dacd7814fc60afd71fe9adb33b3c6168bf93854645d33e33e
在master中执行
kubectl get nodes
此时可以在列表中看到k8s-node1、k8s-node2 状态为NotReady.(因为未安装网络插件)
k8s网络插件有很多,推荐用calico.使用其他的如Flannel请自行google.
在master中执行
wget https://docs.projectcalico.org/manifests/calico.yaml
kubectl apply -f calico.yaml
安装完成之后可以看下集群状态,状态NotReady变为Ready.
如果有问题请自行排查,如:
kubectl get pod -n kube-system
在master中执行
yum install -y bash-completion
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc
在master中执行
kubectl run web-nginx --image=nginx --port=80
查看镜像列表(默认namespace为default)
kubectl get pods -owide
响应如下:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
web-nginx 1/1 Running 0 20s 172.17.104.30 node2 <none> <none>
wget 172.17.104.30
响应的index.html 通过cat可以查看为nginx的首页
kubectl expose pod web-nginx --name=web-nginx --port=80 --type=NodePort
查看服务列表
kubectl get service -owide
响应结果
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 7d1h <none>
web-nginx NodePort 10.96.181.25 <none> 80:32002/TCP 5s run=web-nginx
查看所在node
kubectl get pods -A -owide
通过 访问 http://nodeIp:32002/
可验证nginx是否启动成功.
#输入密码 密码要记下来 例: password
openssl genrsa -des3 -out ca.key 2048
#输入上面输入的密码 后面按需求输入
openssl req -x509 -key ca.key -out ca.crt
mkdir -p /etc/pki/CA/newcerts
touch /etc/pki/CA/index.txt
echo '01' > /etc/pki/CA/serial
#输入密码 密码要记下来 例: password
openssl genrsa -des3 -out nexus.key 2048
#输入nexus证书key的密码 后面按需求输入(与ca.crt一致)
openssl req -new -key nexus.key -reqexts SAN -config <(cat /etc/pki/tls/openssl.cnf \
<(printf "[SAN]\nsubjectAltName=DNS:my.nexus.org,DNS:www.nexus.org")) -out nexus.csr
openssl x509 -req -days 3650 \
-in nexus.csr -out nexus.crt \
-CA ca.crt -CAkey ca.key -CAcreateserial \
-extensions SAN \
-extfile <(cat /etc/pki/tls/openssl.cnf \
<(printf "[SAN]\nsubjectAltName=DNS:my.nexus.org,DNS:www.nexus.org"))
注意: 如果多次签署报错请清理通过 vim etc/pki/CA/index.txt编辑文件删除文件内容.
三台机器(k8s)全部需要执行相同步骤,ca.crt复制到其他节点上即可.
chmod 644 /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
cat ca.crt >> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
chmod 444 /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
cp nexus.crt /etc/pki/ca-trust/source/anchors/my.nexus.org.crt
update-ca-trust
openssl pkcs12 -export -out nexus.pfx -in nexus.crt -inkey nexus.key
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.security.Key;
import java.security.KeyStore;
import java.security.cert.Certificate;
import java.util.Enumeration;
public class CoverP12TokeyStore {
public static final String PKCS12 = "PKCS12";
public static final String JKS = "JKS";
public static final String PFX_KEYSTORE_FILE = "/home/srchen/my.pfx";// pfx文件位置
public static final String PFX_PASSWORD = "password";// 导出为pfx文件的设的密码
public static final String JKS_KEYSTORE_FILE = "/home/srchen/software/nexus/nexus-3.40.1-01/etc/ssl/keystore.jks"; // jks文件位置
public static final String JKS_PASSWORD = "password";// JKS的密码
public static void coverTokeyStore() {
FileInputStream fis = null;
FileOutputStream out = null;
try {
KeyStore inputKeyStore = KeyStore.getInstance("PKCS12");
fis = new FileInputStream(PFX_KEYSTORE_FILE);
char[] pfxPassword = null;
if ((PFX_PASSWORD == null) || PFX_PASSWORD.trim().equals("")) {
pfxPassword = null;
} else {
pfxPassword = PFX_PASSWORD.toCharArray();
}
char[] jksPassword = null;
if ((JKS_PASSWORD == null) || JKS_PASSWORD.trim().equals("")) {
jksPassword = null;
} else {
jksPassword = JKS_PASSWORD.toCharArray();
}
inputKeyStore.load(fis, pfxPassword);
fis.close();
KeyStore outputKeyStore = KeyStore.getInstance("JKS");
outputKeyStore.load(null, jksPassword);
Enumeration<String> enums = inputKeyStore.aliases();
while (enums.hasMoreElements()) {
String keyAlias = enums.nextElement();
if (inputKeyStore.isKeyEntry(keyAlias)) {
Key key = inputKeyStore.getKey(keyAlias, pfxPassword);
Certificate[] certChain = inputKeyStore.getCertificateChain(keyAlias);
outputKeyStore.setKeyEntry(keyAlias, key, jksPassword, certChain);
}
}
out = new FileOutputStream(JKS_KEYSTORE_FILE);
outputKeyStore.store(out, jksPassword);
out.flush();
} catch (Exception e) {
e.printStackTrace();
} finally {
if (fis != null) {
try {
fis.close();
} catch (IOException e) {
e.printStackTrace();
}
}
if (out != null) {
try {
out.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
public static void main(String[] args) {
coverTokeyStore();
}
}
wget https://download.sonatype.com/nexus/3/nexus-3.40.1-01-unix.tar.gz
tar -zxvf nexus-3.40.1-01-unix.tar.gz
cd nexus-3.40.1-01/bin
./nexus run &
启动完成之后密码生成在sonatype-work/nexus3/admin.password文件中.
访问http://192.168.43.137:8081/
输入密码即可.
由于后面采用buildkit来构建镜像推送到私有库中,只能采用https方式,所以需要开启nexus的https服务.
进入nexus配置目录.
cd /home/srchen/software/nexus/nexus-3.40.1-01/etc
编辑nexus-default.properties配置文件,添加${jetty.etc}/jetty-https.xml.
exus-args=${jetty.etc}/jetty.xml,${jetty.etc}/jetty-https.xml,${jetty.etc}/jetty-http.xml,${jetty.etc}/jetty-requestlog.xml
编辑/home/srchen/software/nexus/nexus-3.40.1-01/etc/jetty/jetty-https.xml,配置上面设置的jks的密码.
<New id="sslContextFactory" class="org.eclipse.jetty.util.ssl.SslContextFactory$Server">
<Set name="KeyStorePath"><Property name="ssl.etc"/>/keystore.jks</Set>
<Set name="KeyStorePassword">password</Set>
<Set name="KeyManagerPassword">password</Set>
<Set name="TrustStorePath"><Property name="ssl.etc"/>/keystore.jks</Set>
<Set name="TrustStorePassword">password</Set>
<Set name="EndpointIdentificationAlgorithm"></Set>
<Set name="NeedClientAuth"><Property name="jetty.ssl.needClientAuth" default="false"/></Set>
<Set name="WantClientAuth"><Property name="jetty.ssl.wantClientAuth" default="false"/></Set>
<Set name="IncludeProtocols">
<Array type="java.lang.String">
<Item>TLSv1.2</Item>
</Array>
</Set>
</New>
重启nexus
./nexus restart
登陆 http://192.168.43.137:8081/
访问 http://192.168.43.137:8081/#admin/repository/repositories
点击 Create repository、docker(hosted),
Name输入 Containerd
Online 选中
HTTPS 输入8083端口
剩下全部选中,如果需要清理策略请自行配置.
保存之后会开启https的支持端口为8083.
buildkit 号称下一代构建docker的神器.
在master主机上下载
wget https://github.com/moby/buildkit/releases/download/v0.10.3/buildkit-v0.10.3.linux-arm64.tar.gz
解压并移到local下
tar -zxvf buildkit-v0.10.3.linux-arm64.tar.gz
mkdir -p /usr/local/buildkit-v0.10.3
mv bin /usr/local/buildkit-v0.10.3
vim /etc/profile
#加入
PATH=$PATH:/usr/local/buildkit/bin
source /etc/profile
mkdir /etc/buildkit
vim /etc/buildkit/buildkitd.toml
debug = true
[worker.containerd]
namespace = "k8s.io"
[registry."docker.io"]
mirrors = ["my.nexus.org:8083"]
http = false
nsecure = true
ca=["/home/srchen/keys/ca.key"]
[registry."my.nexus.org:8083"]
http = false
insecure = true
ca=["/home/srchen/keys/ca.key"]
[[registry."my.nexus.org".keypair]]
key="/home/srchen/keys/nexus.key"
cert="/home/srchen/keys/nexus.crt"
buildkitd --oci-worker=false --containerd-worker=true &
通过dawdler-series来举例子.
wget https://github.com/srchen1987/dawdler-runtime/archive/refs/tags/0.0.2-RELEASES.tar.gz
tar -zxvf 0.0.2-RELEASES.tar.gz
cd dawdler-runtime-0.0.2-RELEASES
vim Dockerfile
FROM suranagivinod/openjdk8
MAINTAINER suxuan696@gmail.com
RUN mkdir /opt/dawdler-server-0.0.2
ADD ./dawdler-server-0.0.2 /opt/dawdler-server-0.0.2
WORKDIR /opt/dawdler-server-0.0.2/bin
ENTRYPOINT ["/bin/sh","dawdler.sh","run"]
buildctl build --no-cache \
--frontend=dockerfile.v0 \
--local context=. \
--local dockerfile=. \
--output type=image,name=my.nexus.org:8083/dawdler:0.0.2,push=true \
--export-cache type=inline
这里只是演示所以采用 run方式,如果是生产环境建议采用 deployment或apply -f dawdler.yaml方式来创建.
注意: 用run方式运行 node节点中镜像如果没更新则需要通过以下脚本清理.
#查看列表
crictl images
# 449cdf80490b5是 images查看到容器ID
crictl rmi 449cdf80490b5
也可以通过 imagePullPolicy: Always 配置每次私有仓拉取镜像.
kubectl run dawdler --image=my.nexus.org:8083/dawdler:0.0.2
kubectl get pods -n default -owide
响应
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
dawdler 1/1 Running 0 11h 172.17.104.28 node2 <none> <none>
kubectl logs dawdler
响应
Welcome to use dawdler!
_____ __ __ _____ _ ______ _____
| __ \ /\ \ \ / / | __ \ | | | ____| | __ \
| | | | / \ \ \ /\ / / | | | | | | | |__ | |__) |
| | | | / /\ \ \ \/ \/ / | | | | | | | __| | _ /
| |__| | / ____ \ \ /\ / | |__| | | |____ | |____ | | \ \
|_____/ /_/ \_\ \/ \/ |_____/ |______| |______| |_| \_\
OS arch: amd64
OS availableProcessors: 4
OS name: Linux
OS version: 5.17.5-300.fc36.x86_64
Jvm totalMemory: 5.20093696E8(507904.00K)
Jvm freeMemory: 5.01973992E8(490208.98K)
Heap Memory Usage:
init = 536870912(524288K) used = 18119704(17695K) committed = 520093696(507904K) max = 520093696(507904K)
Non-Heap Memory Usage:
init = 2555904(2496K) used = 11309544(11044K) committed = 11730944(11456K) max = -1(-1K)
Java options:
[-Xms512m, -Xmx512m, -Xmn128m]
ClassPath: .:./asm-7.1.jar:./cglib-3.3.0.jar:./commons-jexl3-3.2.jar:./commons-logging-1.2.jar:./commons-net-3.6.jar:./commons-pool2-2.10.0.jar:./consul-api-1.4.5.jar:./curator-client-5.1.0.jar:./curator-framework-5.1.0.jar:./curator-recipes-5.1.0.jar:./dawdler-core-0.0.2-RELEASES.jar:./dawdler-serialization-0.0.2-RELEASES.jar:./dawdler-server-0.0.2-RELEASES.jar:./dawdler-util-0.0.2-RELEASES.jar:./dom4j-2.1.3.jar:./jaxen-1.2.0.jar:./kryo-4.0.2.jar:./logback-access-1.2.7.jar:./logback-classic-1.2.7.jar:./logback-core-1.2.7.jar:./minlog-1.3.0.jar:./objenesis-2.5.1.jar:./reflectasm-1.11.3.jar:./slf4j-api-1.7.32.jar:./zookeeper-3.6.2.jar:./zookeeper-jute-3.6.2.jar
LibraryPath: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
Server startup in 0 ms,Listening port: 9527!
|