minelona/k8s_PaaS
1
0
Fork 0
mirror of https://github.com/ben1234560/k8s_PaaS.git synced 2025-04-22 06:35:52 +08:00
Code Issues Packages Projects Releases Wiki Activity
k8s_PaaS/第六章——在K8S中集成Apollo配置中心.md

2211 lines
60 KiB
Bash
Raw Normal View History

Create 第六章——在K8S中集成Apollo配置中心.md
2020-03-15 21:41:40 +08:00
## 第六章——在K8S中集成Apollo配置中心
> **前言：**
>
> **运维八荣八耻：**
>
> - 以可配置为荣，以硬编码为耻
> - 以互备为荣，以单点为耻
> - 以随时重启为荣，以不能迁移为耻
> - 以整体交付为荣，以部分交付为耻
> - 以无状态为荣，以有状态为耻
> - 以标准化为荣，以特殊化为耻
> - 以自动化工具为荣，以手动和人肉为耻
> - 以无人值守为荣，以人工介入为耻
>
> **目前我们交付进K8S集群的两个dubbo微服务和monitor，它们的配置都是写死在容器里的**
>
> **配置管理的现状：**
>
> - 配置散乱格式不标准（XML、ini、conf、yaml...）
> - 主要采用本地静态配置，应用多副本集下配置修改麻烦
> - 易引发生产事故（测试环境、生产环境配置混用）
> - 配置缺乏安全审计和版本控制功能（config review）
> - 不同环境的应用，配置不同，造成多次打包，测试失败
>
> **配置中心是什么？：**
>
> - 顾名思义，就是集中管理应用程序配置的“中心”
> 常见的配置中心：
>
> - SprigCloudConfig
> - K8S ConfigMap
> - Apollo：基于SprigCloudConfig
> - ...
>
> 目前具有争议的一般是使用Apollo还是SprigCloudConfig，看对比图，所以我们使用Apollo，而且Apollo是基于SprigCloudConfig，也就是你交付了Apollo也相当于交付了微服务
![1582622382181](assets/1582622382181.png)
### configmap使用详解
> **WHAT：**就是为了让镜像 和 配置文件解耦，以便实现镜像的可移植性和可复用性，因为一个configMap其实就是一系列配置信息的集合，将来可直接注入到Pod中的容器使用
>
> **WHY：**为了配合Apollo使用
使用configmap管理应用配置，需要先拆分环境，拆分未test和pro环境来模拟实际工作
先把dubbo的服务（消费者/服务站/监视者）都scale成0，这样就没有容器在里面跑了
![1583162321302](assets/1583162321302.png)
![1583162339688](assets/1583162339688.png)
拆分zk成测试环境和生产环境来模拟实际情况，如下：
| 主机名 | 角色 | ip |
| ----------------- | -------------------- | --------- |
| HDSS7-11.host.com | zk1.od.com(Test环境) | 10.4.7.11 |
| HDSS7-12.host.com | zk2.od.com(Prod环境) | 10.4.7.12 |
之前是11、12、21一共3台，我们先把21拆了
~~~
# 11/12/21机器，停掉zookeeper并删掉相关data文件和log文件的内容：
~]# cd /opt/zookeeper
zookeeper]# bin/zkServer.sh stop
# 停不掉就 kill -9 id 杀掉
zookeeper]# ps aux|grep zoo
zookeeper]# cd /data/zookeeper/data/
data]# rm -fr ./*
cd ../logs/
logs]# rm -fr ./*
~~~
> **kill -9：**强制杀死该进程
![1581059149533](assets/1581059149533.png)
~~~
# 11/12机器，修改zoo.cfg配置：
zookeeper]# cd /opt/zookeeper/conf
zookeeper]# vi zoo.cfg
# 把下面的三行都删掉，不需要组成集群，如图
~~~
![1581059640346](assets/1581059640346.png)
~~~
# 11/12机器，启动zk：
cd /opt/zookeeper/
zookeeper]# bin/zkServer.sh start
zookeeper]# ps aux|grep zoo
zookeeper]# bin/zkServer.sh status
~~~
Update 第六章——在K8S中集成Apollo配置中心.md
2020-03-20 18:53:41 +08:00
![1584697777390](assets/1584697777390.png)
Create 第六章——在K8S中集成Apollo配置中心.md
2020-03-15 21:41:40 +08:00
Mode: standalone 模式，拆分完成
~~~
# 200机器，准备资源配置清单：
~]# vi /data/k8s-yaml/dubbo-monitor/cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: dubbo-monitor-cm
namespace: infra
data:
dubbo.properties: |
dubbo.container=log4j,spring,registry,jetty
dubbo.application.name=simple-monitor
dubbo.application.owner=ben1234560
dubbo.registry.address=zookeeper://zk1.od.com:2181
dubbo.protocol.port=20880
dubbo.jetty.port=8080
dubbo.jetty.directory=/dubbo-monitor-simple/monitor
dubbo.charts.directory=/dubbo-monitor-simple/charts
dubbo.statistics.directory=/dubbo-monitor-simple/statistics
dubbo.log4j.file=/dubbo-monitor-simple/logs/dubbo-monitor.log
dubbo.log4j.level=WARN
~]# vi /data/k8s-yaml/dubbo-monitor/dp2.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: dubbo-monitor
namespace: infra
labels:
name: dubbo-monitor
spec:
replicas: 1
selector:
matchLabels:
name: dubbo-monitor
template:
metadata:
labels:
app: dubbo-monitor
name: dubbo-monitor
spec:
containers:
- name: dubbo-monitor
image: harbor.od.com/infra/dubbo-monitor:latest
ports:
- containerPort: 8080
protocol: TCP
- containerPort: 20880
protocol: TCP
imagePullPolicy: IfNotPresent
volumeMounts:
- name: configmap-volume
mountPath: /dubbo-monitor-simple/conf
volumes:
- name: configmap-volume
configMap:
name: dubbo-monitor-cm
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
~~~
~~~
# 对比两个dp有什么不同，200机器：
cd /data/k8s-yaml/dubbo-monitor/
dubbo-monitor]# vimdiff dp.yaml dp2.yaml
#:qall 退出
~~~
> **vimdiff ：**编辑同一文件的不同历史版本，对各文件的内容进行比对与调整
>
> 没有vimdiff 的下载 yum install vim -y
![1583198466051](assets/1583198466051.png)
> 这里两个文件不同的地方，注意，这里的image的版本因为我的操作问题所以一个v2一个没有v，你的应该是都没有的
~~~
# 200机器，替换dp：
dubbo-monitor]# mv dp.yaml /tmp/
dubbo-monitor]# mv dp2.yaml dp.yaml
~~~
> **mv：**如果mv后是目录，则是移动，如果是文件名，则是更改名字
~~~
# 应用资源配置清单，22机器：
~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-monitor/cm.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-monitor/dp.yaml
# 看一下相关容器有没有起来
~~~
![1583198817470](assets/1583198817470.png)
![1583198829279](assets/1583198829279.png)
![1583198887392](assets/1583198887392.png)
![1583199215057](assets/1583199215057.png)
我们改成zk2
![1583199496585](assets/1583199496585.png)
在删掉这个pod重启
![1583199525277](assets/1583199525277.png)
刷新页面变成zk2了
![1583199653453](assets/1583199653453.png)
完成
#### 报错信息
> flannel重启报错问题
~~~
# flannel重启需要增加以下内容，添加在最下面:
21 ~]# vi /etc/sipervisord.d/flannel.ini
killasgroup=true
stopasgroup=true
~~~
#### 如何排错：
~~~
# 查看时间是否正常
date
# 查看kubernetes的报错
21 ~]# tail -fn 200/data/logs/kubernetes/kube-kubelet/kubelet.stdout.log
# 可以重启docker 和 kubelet
21 ~]# systemctl restart docker
21 ~]# ps aux|grep kubelet
21 ~]# kill -9 id
# 查看supervisor的状态
21 ~]# supervisorctl status
# 查看路由状态，是否能ping通其它机器
21 ~]# route -n
21 ~]# ping 172.7.22.2
# 查看flannel情况
21 ~]# tail -fn 200 /data/logs/flanneld/flanneld.stdout.log
# 12flannel主机器上get网络，如果error则加入backend
12 ~]# ./etcdctl get /coreos.com/network/config
# 报错iptables，让机器一重启就自动使用，21/22机器
~]# service iptables save
# out: [ok]
~~~
#### cm对象的另一种创建方法（可不做）：
~~~
# 22机器：
cd /opt/kubernetes/server/bin/conf
# 当前目录下要又kubelet.kubeconfig的文件
conf]# kubectl create cm kubelet-cm --from-file=./kubelet.kubeconfig
#out: configmap/kubelet-cm created
~~~
Update 第六章——在K8S中集成Apollo配置中心.md
2020-03-20 18:53:41 +08:00
可以在dashoboard的default名称空间下Config Maps看到
Create 第六章——在K8S中集成Apollo配置中心.md
2020-03-15 21:41:40 +08:00
随后记得删除
#### 官方Apollo框架：
Update 第六章——在K8S中集成Apollo配置中心.md
2020-03-20 18:53:41 +08:00
![img](https://raw.githubusercontent.com/ctripcorp/apollo/master/doc/images/overall-architecture.png)
Create 第六章——在K8S中集成Apollo配置中心.md
2020-03-15 21:41:40 +08:00
#### 简化Apollo框架：
Update 第六章——在K8S中集成Apollo配置中心.md
2020-03-20 18:53:41 +08:00
![1584698746125](assets/1584698746125.png)
![1584698613252](assets/1584698613252.png)
Create 第六章——在K8S中集成Apollo配置中心.md
2020-03-15 21:41:40 +08:00
> Client通过推拉结合和ConfigService交互，然后ConfigService去拿ConfigDB里面的配置给Client端返回去。
>
> Portal是Apollo的一个仪表盘，通过调用AdminService去同步修改ConfigDB里面的配置
>
> 先交付ConfigService，然后交付AdminService，最后加portal
### 交付Apollo-ConfigService到K8S
[官网https://github.com/ctripcorp/apollo/](https://github.com/ctripcorp/apollo/)
##### 安装部署MySQL数据库
~~~~
# 更新yum源，11机器：
~]# vi /etc/yum.repos.d/MariaDB.repo
[mariadb]
name = MariaDB
baseurl = https://mirrors.ustc.edu.cn/mariadb/yum/10.1/centos7-amd64/
gpgkey=https://mirrors.ustc.edu.cn/mariadb/yum/RPM-GPG-KEY-MariaDB
gpgcheck=1
# 导入GPG-KEY
~]# rpm --import https://mirrors.ustc.edu.cn/mariadb/yum/RPM-GPG-KEY-MariaDB
~]# yum list mariadb --show-duplicates
# out: mariadb.x86_64
~]# yum makecache
# out:Metadata Cache Created
# 更新数据库版本
~]# yum list mariadb-server --show-duplicates
~]# yum install mariadb-server -y
# 编辑基础配置
# 增加部分内容
~] vi /etc/my.cnf.d/server.cnf
[mysqld]
character_set_server = utf8mb4
collation_server = utf8mb4_general_ci
init_connect = "SET NAMES 'utf8mb4'"
~~~~
![1583202954447](assets/1583202954447.png)
~~~
# 11机器，修改基础配置,并启动：
# 在改内容下增加，字符集
~]# vi /etc/my.cnf.d/mysql-clients.cnf
[mysql]
default-character-set = utf8mb4
~]# systemctl start mariadb
~~~
![1583202999075](assets/1583202999075.png)
~~~
# 11机器，设置,mysql密码：
~]# mysqladmin -uroot password
# 这里是设置密码：123456
~]# mysql -uroot -p
# 这里是输入密码
none)]> \s
# 确定SERVER\DB\CLIENT\CONN 都是utf8
~~~
![1583203114762](assets/1583203114762.png)
~~~
# 11机器：
none)]> show databases;
none)]> drop database test;
none)]> exit
# 查看数据库是否正常
~]# ps aux|grep mysql
~]# netstat luntp|grep 3306
~]# wget https://raw.githubusercontent.com/ctripcorp/apollo/1.5.1/scripts/db/migration/configdb/V1.0.0__initialization.sql -O apolloconfig.sql
~]# cat apolloconfig.sql
~]# mysql -uroot -p < apolloconfig.sql
# 输入密码
~] mysql -uroot -p
# 输密码
none)]> show databases;
~~~
![1581148090694](assets/1581148090694.png)
~~~
# 11机器，创建其它权限用户而不是直接给root权限：
none)]> use ApolloConfigDB;
ApolloConfigDB]> show tables;
ApolloConfigDB]> grant INSERT,DELETE,UPDATE,SELECT on ApolloConfigDB.* to 'apolloconfig'@'10.4.7.%' identified by "123456";
ApolloConfigDB]> select user,host from mysql.user;
~~~
![1583203450336](assets/1583203450336.png)
~~~
# 11机器,修改初始数据：
ApolloConfigDB]> show tables;
ApolloConfigDB]> select * from ServerConfig\G
ApolloConfigDB]> update ApolloConfigDB.ServerConfig set ServerConfig.Value="http://config.od.com/eureka" where ServerConfig.Key="eureka.service.url";
ApolloConfigDB]> select * from ServerConfig\G
~~~
原：
![1583203490999](assets/1583203490999.png)
改后
![1583203522358](assets/1583203522358.png)
~~~
# 11机器，解析域名：
~]# vi /var/named/od.com.zone
serial 前滚一位
config A 10.4.7.10
~]# systemctl restart named
~~~
![1583203561891](assets/1583203561891.png)
~~~
# 21机器，测试下21机器能不能查到(11机器是肯定能查到的)：
~]# dig -t A config.od.com @192.168.0.2 +short
# out:10.4.7.10
~~~
[包的官网地址](https://github.com/ctripcorp/apollo/releases/tag/v1.5.1)![1583204125452](assets/1583204125452.png)
~~~
# 200机器，制作docker镜像：
cd /opt/src
src]# wget https://github.com/ctripcorp/apollo/releases/download/v1.5.1/apollo-configservice-1.5.1-github.zip
# 或者去官网下载或者用我的上传的包
src]# mkdir /data/dockerfile/apollo-configservice
src]# unzip -o apollo-configservice-1.5.1-github.zip -d /data/dockerfile/apollo-configservice
src]# cd /data/dockerfile/apollo-configservice/
apollo-configservice]# rm -rf apollo-configservice-1.5.1-sources.jar
apollo-configservice]# ll
~~~
![1581232250106](assets/1581232250106.png)
~~~
# 11机器，解析域名：
~]# vi /var/named/od.com.zone
serial 前滚一位
mysql A 10.4.7.11
~]# systemctl restart named
~]# dig -t A mysql.od.com @10.4.7.11 +short
# out: 10.4.7.11
~~~
![1582636827536](assets/1582636827536.png)
~~~~
# 200机器，修改账户密码：
cd /data/dockerfile/apollo-configservice/config
config]# vi application-github.properties
spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigDB?characterEncoding=utf8
spring.datasource.username = apolloconfig
spring.datasource.password = 123456
~~~~
![1581232483973](assets/1581232483973.png)
~~~
# 200机器：
cd /data/dockerfile/apollo-configservice/scripts
scripts]# rm -f shutdown.sh
# 全部删掉，换成以下内容
scripts]# vi startup.sh
#!/bin/bash
SERVICE_NAME=apollo-configservice
## Adjust log dir if necessary
LOG_DIR=/opt/logs/apollo-config-server
## Adjust server port if necessary
SERVER_PORT=8080
APOLLO_CONFIG_SERVICE_NAME=$(hostname -i)
SERVER_URL="http://${APOLLO_CONFIG_SERVICE_NAME}:${SERVER_PORT}"
## Adjust memory settings if necessary
export JAVA_OPTS="-Xms128m -Xmx128m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=256m -XX:MaxNewSize=256m -XX:SurvivorRatio=8"
## Only uncomment the following when you are using server jvm
#export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"
########### The following is the same for configservice, adminservice, portal ###########
export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"
# Find Java
if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
javaexe="$JAVA_HOME/bin/java"
elif type -p java > /dev/null 2>&1; then
javaexe=$(type -p java)
elif [[ -x "/usr/bin/java" ]]; then
javaexe="/usr/bin/java"
else
echo "Unable to find Java"
exit 1
fi
if [[ "$javaexe" ]]; then
version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
# now version is of format 009003 (9.3.x)
if [ $version -ge 011000 ]; then
JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
elif [ $version -ge 010000 ]; then
JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
elif [ $version -ge 009000 ]; then
JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
else
JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
fi
fi
printf "$(date) ==== Starting ==== \n"
cd `dirname $0`/..
chmod 755 $SERVICE_NAME".jar"
./$SERVICE_NAME".jar" start
rc=$?;
if [[ $rc != 0 ]];
then
echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
exit $rc;
fi
tail -f /dev/null
~~~
> [Apollo官方文档](https://github.com/ctripcorp/apollo/blob/1.5.1/scripts/apollo-on-kubernetes/apollo-config-server/scripts/startup-kubernetes.sh)
>
> 上述代码是从Apollo官网拉下来的，不过第7行多了一行APOLLO_CONFIG_SERVICE_NAME=$(hostname -i)
>
> 其中export JAVA_OPTS修改了下资源，改小了
~~~
# 200机器，给权限：
scripts]# chmod u+x startup.sh
scripts]# ll
~~~
![1581233647557](assets/1581233647557.png)
~~~
# 200机器，做dockerfile：
cd /data/dockerfile/apollo-configservice
apollo-configservice]# vi Dockerfile
FROM 909336740/jre8:8u112
ENV VERSION 1.5.1
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
echo "Asia/Shanghai" > /etc/timezone
ADD apollo-configservice-${VERSION}.jar /apollo-configservice/apollo-configservice.jar
ADD config/ /apollo-configservice/config
ADD scripts/ /apollo-configservice/scripts
CMD ["/apollo-configservice/scripts/startup.sh"]
~~~
> [参考Apollo官网文档](https://github.com/ctripcorp/apollo/blob/1.5.1/scripts/apollo-on-kubernetes/apollo-config-server/Dockerfile)
![1583219072270](assets/1583219072270.png)
~~~
# 200机器,制作容器：
apollo-configservice]# docker build . -t harbor.od.com/infra/apollo-configservice:v1.5.1
apollo-configservice]# docker push harbor.od.com/infra/apollo-configservice:v1.5.1
~~~
![1583205129961](assets/1583205129961.png)
~~~
# 制作资源配置清单，200机器：
~]# mkdir /data/k8s-yaml/apollo-configservice
~]# cd /data/k8s-yaml/apollo-configservice
apollo-configservice]# vi cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: apollo-configservice-cm
namespace: infra
data:
application-github.properties: |
# DataSource
spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigDB?characterEncoding=utf8
spring.datasource.username = apolloconfig
spring.datasource.password = 123456
eureka.service.url = http://config.od.com/eureka
app.properties: |
appId=100003171
apollo-configservice]# vi dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: apollo-configservice
namespace: infra
labels:
name: apollo-configservice
spec:
replicas: 1
selector:
matchLabels:
name: apollo-configservice
template:
metadata:
labels:
app: apollo-configservice
name: apollo-configservice
spec:
volumes:
- name: configmap-volume
configMap:
name: apollo-configservice-cm
containers:
- name: apollo-configservice
image: harbor.od.com/infra/apollo-configservice:v1.5.1
ports:
- containerPort: 8080
protocol: TCP
volumeMounts:
- name: configmap-volume
mountPath: /apollo-configservice/config
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
apollo-configservice]# vi svc.yaml
kind: Service
apiVersion: v1
metadata:
name: apollo-configservice
namespace: infra
spec:
ports:
- protocol: TCP
port: 8080
targetPort: 8080
selector:
app: apollo-configservice
apollo-configservice]# vi ingress.yaml
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: apollo-configservice
namespace: infra
spec:
rules:
- host: config.od.com
http:
paths:
- path: /
backend:
serviceName: apollo-configservice
servicePort: 8080
~~~
![1583205384294](assets/1583205384294.png)
~~~
# 应用资源配置清单，22机器：
~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/cm.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/dp.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/svc.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/ingress.yaml
~~~
![1583206365045](assets/1583206365045.png)
资源给的少，可能稍微慢了一些，点进去->点右上角的LOGS，日志有点多，记得点击右下角的翻页
[浏览器访问config.od.com](config.od.com)
鼠标对着起来的Apollo，可以看到右下角有网址
![1583206733615](assets/1583206733615.png)
~~~
# 22机器，curl：
~]# curl http://172.7.22.5:8080/info
~~~
![1583206773094](assets/1583206773094.png)
成功
### Apollo-ConfigService连接数据库IP分析
~~~
# 11机器：
~]# mysql -uroot -p
none)]> show processlist;
~~~
![1583207560779](assets/1583207560779.png)
![1583207586526](assets/1583207586526.png)
> 原本是172.7.22.5，连接到数据的是10.4.7.22，因为做了NAT转换，它带了面具
>
### 交付Apollo-adminservice
[官网下载https://github.com/ctripcorp/apollo/releases/tag/v1.5.1](https://github.com/ctripcorp/apollo/releases/tag/v1.5.1)
Update 第六章——在K8S中集成Apollo配置中心.md
2020-03-20 18:53:41 +08:00
![1584698854148](assets/1584698854148.png)
Create 第六章——在K8S中集成Apollo配置中心.md
2020-03-15 21:41:40 +08:00
~~~
# 200机器：
cd /opt/src
src]# wget https://github.com/ctripcorp/apollo/releases/download/v1.5.1/apollo-adminservice-1.5.1-github.zip
src]# mkdir /data/dockerfile/apollo-adminservice
src]# unzip -o apollo-adminservice-1.5.1-github.zip -d /data/dockerfile/apollo-adminservice
src]# cd /data/dockerfile/apollo-adminservice
apollo-adminservice]# rm -fr apollo-adminservice-1.5.1-sources.jar
apollo-adminservice]# rm -f apollo-adminservice.conf
apollo-adminservice]# cd scripts/
scripts]# rm -f shutdown.sh
# 删掉原来的全部内容，添加以下新的内容
scripts]# vi startup.sh
#!/bin/bash
SERVICE_NAME=apollo-adminservice
## Adjust log dir if necessary
LOG_DIR=/opt/logs/apollo-admin-server
## Adjust server port if necessary
SERVER_PORT=8080
APOLLO_ADMIN_SERVICE_NAME=$(hostname -i)
# SERVER_URL="http://localhost:${SERVER_PORT}"
SERVER_URL="http://${APOLLO_ADMIN_SERVICE_NAME}:${SERVER_PORT}"
## Adjust memory settings if necessary
#export JAVA_OPTS="-Xms2560m -Xmx2560m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=1536m -XX:MaxNewSize=1536m -XX:SurvivorRatio=8"
## Only uncomment the following when you are using server jvm
#export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"
########### The following is the same for configservice, adminservice, portal ###########
export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"
# Find Java
if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
javaexe="$JAVA_HOME/bin/java"
elif type -p java > /dev/null 2>&1; then
javaexe=$(type -p java)
elif [[ -x "/usr/bin/java" ]]; then
javaexe="/usr/bin/java"
else
echo "Unable to find Java"
exit 1
fi
if [[ "$javaexe" ]]; then
version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
# now version is of format 009003 (9.3.x)
if [ $version -ge 011000 ]; then
JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
elif [ $version -ge 010000 ]; then
JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
elif [ $version -ge 009000 ]; then
JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
else
JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
fi
fi
printf "$(date) ==== Starting ==== \n"
cd `dirname $0`/..
chmod 755 $SERVICE_NAME".jar"
./$SERVICE_NAME".jar" start
rc=$?;
if [[ $rc != 0 ]];
then
echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
exit $rc;
fi
tail -f /dev/null
~~~
> [startup.sh官方地址](https://github.com/ctripcorp/apollo/blob/1.5.1/scripts/apollo-on-kubernetes/apollo-admin-server/scripts/startup-kubernetes.sh)
>
> 修改处为：
>
> SERVER_PORT=8080 # 因为docker网络空间是互相隔离，用8080即可
>
> APOLLO_ADMIN_SERVICE_NAME=$(hostname -i)
~~~
# 200机器，制作dockerfile：
cd /data/dockerfile/apollo-adminservice
apollo-adminservice]# ll
apollo-adminservice]# vi Dockerfile
FROM 909336740/jre8:8u112
ENV VERSION 1.5.1
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
echo "Asia/Shanghai" > /etc/timezone
ADD apollo-adminservice-${VERSION}.jar /apollo-adminservice/apollo-adminservice.jar
ADD config/ /apollo-adminservice/config
ADD scripts/ /apollo-adminservice/scripts
CMD ["/apollo-adminservice/scripts/startup.sh"]
apollo-adminservice]# docker build . -t harbor.od.com/infra/apollo-adminservice:v1.5.1
apollo-adminservice]# docker push harbor.od.com/infra/apollo-adminservice:v1.5.1
~~~
![1583207853169](assets/1583207853169.png)
![1583219005442](assets/1583219005442.png)
看以下harbor仓库有没有
![1583208050305](assets/1583208050305.png)
##### 重新复习一下交付过程：
##### 搞定镜像-搞定资源配置清单-应用资源配置清单
~~~
# 200机器，制作资源配置清单：
mkdir /data/k8s-yaml/apollo-adminservice
cd /data/k8s-yaml/apollo-adminservice
apollo-adminservice]# vi cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: apollo-adminservice-cm
namespace: infra
data:
application-github.properties: |
# DataSource
spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigDB?characterEncoding=utf8
spring.datasource.username = apolloconfig
spring.datasource.password = 123456
eureka.service.url = http://config.od.com/eureka
app.properties: |
appId=100003172
apollo-adminservice]# vi dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: apollo-adminservice
namespace: infra
labels:
name: apollo-adminservice
spec:
replicas: 1
selector:
matchLabels:
name: apollo-adminservice
template:
metadata:
labels:
app: apollo-adminservice
name: apollo-adminservice
spec:
volumes:
- name: configmap-volume
configMap:
name: apollo-adminservice-cm
containers:
- name: apollo-adminservice
image: harbor.od.com/infra/apollo-adminservice:v1.5.1
ports:
- containerPort: 8080
protocol: TCP
volumeMounts:
- name: configmap-volume
mountPath: /apollo-adminservice/config
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
~~~
![1583208130534](assets/1583208130534.png)
~~~
# 22机器，应用资源配置清单：
~]# kubectl apply -f http://k8s-yaml.od.com/apollo-adminservice/cm.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/apollo-adminservice/dp.yaml
~~~
![1583217458560](assets/1583217458560.png)
~~~
# 11机器，查看数据库连接IP：
~]# mysql -uroot -p
none)]> show processlist;
~~~
多了更多的22连接
![1583217487662](assets/1583217487662.png)
[config.od.com访问地址](config.od.com)
鼠标对着起来的Apollo，可以看到右下角有网址
![1583217527372](assets/1583217527372.png)
~~~
# 22机器，curl：
~]# curl http://172.7.22.7:8080/info
~~~
![1583217575749](assets/1583217575749.png)
[^课外题1]: 交付完config和admin后，我们发现，交付到K8S里面的服务如此之容易，那么问题来了，交付到K8S和交付到物理机或者虚拟机里，那里有很大的区别，为什么K8S更好。
[^课外题2]: 尝试用点点点的方式在dashboard里面扩容adminservice和configservice
### 交付Apollo-Portal前，数据库初始化
[官网地址](https://github.com/ctripcorp/apollo/releases/tag/v1.5.1)
~~~
# 200机器，制作镜像-下载包并整理：
cd /opt/src/
~]# wget https://github.com/ctripcorp/apollo/releases/download/v1.5.1/apollo-portal-1.5.1-github.zip
src]# mkdir /data/dockerfile/apollo-portal
src]# unzip -o apollo-portal-1.5.1-github.zip -d /data/dockerfile/apollo-portal
src]# cd /data/dockerfile/apollo-portal
apollo-portal]# rm -f apollo-portal-1.5.1-sources.jar
apollo-portal]# rm -f apollo-portal.conf
apollo-portal]# rm -f scripts/shutdown.sh
~~~
[sql官网网址-需要raw](https://github.com/ctripcorp/apollo/blob/master/scripts/apollo-on-kubernetes/db/portal-db/apolloportaldb.sql)
~~~
# 11机器，数据库初始化：
~]# wget https://raw.githubusercontent.com/ctripcorp/apollo/master/scripts/apollo-on-kubernetes/db/portal-db/apolloportaldb.sql -O apolloportal.sql
~]# ll
~~~
![1583217807056](assets/1583217807056.png)
~~~
# 11机器：
~]# mysql -uroot -p
none)]> source ./apolloportal.sql
none)]> show databases;
none)]> use ApolloPortalDB
ApolloportalDB]> show tables;
ApolloportalDB]> grant INSERT,DELETE,UPDATE,SELECT on ApolloPortalDB.* to "apolloportal"@"10.4.7.%" identified by "123456";
ApolloportalDB]> select user,host from mysql.user;
ApolloportalDB]> select * from ServerConfig\G
ApolloportalDB]> update ServerConfig set Value='[{"orgId":"ben01","orgName":"Linux学院"},{"orgId":"ben02","orgName":"云计算学院"},{"orgId":"ben03","orgName":"Python学院"}]' where Id=2;
ApolloportalDB]> select * from ServerConfig\G
~~~
![1583217921425](assets/1583217921425.png)
原![1583217963849](assets/1583217963849.png)
改后
![1583218104712](assets/1583218104712.png)
完成
### 制作Portal的docker镜像，并交付
~~~
# 200机器，更新startup:
cd /data/dockerfile/apollo-portal/scripts/
# 全部删掉换成下面的
scripts]# vi startup.sh
#!/bin/bash
SERVICE_NAME=apollo-portal
## Adjust log dir if necessary
LOG_DIR=/opt/logs/apollo-portal-server
## Adjust server port if necessary
SERVER_PORT=8080
APOLLO_PORTAL_SERVICE_NAME=$(hostname -i)
# SERVER_URL="http://localhost:$SERVER_PORT"
SERVER_URL="http://${APOLLO_PORTAL_SERVICE_NAME}:${SERVER_PORT}"
## Adjust memory settings if necessary
#export JAVA_OPTS="-Xms2560m -Xmx2560m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=1536m -XX:MaxNewSize=1536m -XX:SurvivorRatio=8"
## Only uncomment the following when you are using server jvm
#export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"
########### The following is the same for configservice, adminservice, portal ###########
export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"
# Find Java
if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
javaexe="$JAVA_HOME/bin/java"
elif type -p java > /dev/null 2>&1; then
javaexe=$(type -p java)
elif [[ -x "/usr/bin/java" ]]; then
javaexe="/usr/bin/java"
else
echo "Unable to find Java"
exit 1
fi
if [[ "$javaexe" ]]; then
version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
# now version is of format 009003 (9.3.x)
if [ $version -ge 011000 ]; then
JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
elif [ $version -ge 010000 ]; then
JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
elif [ $version -ge 009000 ]; then
JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
else
JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
fi
fi
printf "$(date) ==== Starting ==== \n"
cd `dirname $0`/..
chmod 755 $SERVICE_NAME".jar"
./$SERVICE_NAME".jar" start
rc=$?;
if [[ $rc != 0 ]];
then
echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
exit $rc;
fi
tail -f /dev/null
~~~
> [portal_startup官网地址](https://github.com/ctripcorp/apollo/blob/1.5.1/scripts/apollo-on-kubernetes/apollo-portal-server/scripts/startup-kubernetes.sh)
>
> 仅有以下两处不同：
>
> SERVER_PORT=8080
> APOLLO_PORTAL_SERVICE_NAME=$(hostname -i)
~~~
# 200机器，制作dockerfile：
cd /data/dockerfile/apollo-portal/
apollo-portal]# vi Dockerfile
FROM 909336740/jre8:8u112
ENV VERSION 1.5.1
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
echo "Asia/Shanghai" > /etc/timezone
ADD apollo-portal-${VERSION}.jar /apollo-portal/apollo-portal.jar
ADD config/ /apollo-portal/config
ADD scripts/ /apollo-portal/scripts
CMD ["/apollo-portal/scripts/startup.sh"]
apollo-portal]# docker build . -t harbor.od.com/infra/apollo-portal:v1.5.1
apollo-portal]# docker push harbor.od.com/infra/apollo-portal:v1.5.1
~~~
![1583218938233](assets/1583218938233.png)
![1583219229803](assets/1583219229803.png)
~~~
# 200机器：
mkdir /data/k8s-yaml/apollo-portal
cd /data/k8s-yaml/apollo-portal
apollo-portal]# vi cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: apollo-portal-cm
namespace: infra
data:
application-github.properties: |
# DataSource
spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloPortalDB?characterEncoding=utf8
spring.datasource.username = apolloportal
spring.datasource.password = 123456
app.properties: |
appId=100003173
apollo-env.properties: |
dev.meta=http://config.od.com
apollo-portal]# vi dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: apollo-portal
namespace: infra
labels:
name: apollo-portal
spec:
replicas: 1
selector:
matchLabels:
name: apollo-portal
template:
metadata:
labels:
app: apollo-portal
name: apollo-portal
spec:
volumes:
- name: configmap-volume
configMap:
name: apollo-portal-cm
containers:
- name: apollo-portal
image: harbor.od.com/infra/apollo-portal:v1.5.1
ports:
- containerPort: 8080
protocol: TCP
volumeMounts:
- name: configmap-volume
mountPath: /apollo-portal/config
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
apollo-portal]# vi svc.yaml
kind: Service
apiVersion: v1
metadata:
name: apollo-portal
namespace: infra
spec:
ports:
- protocol: TCP
port: 8080
targetPort: 8080
selector:
app: apollo-portal
apollo-portal]# vi ingress.yaml
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: apollo-portal
namespace: infra
spec:
rules:
- host: portal.od.com
http:
paths:
- path: /
backend:
serviceName: apollo-portal
servicePort: 8080
~~~
![1583219397807](assets/1583219397807.png)
~~~
# 11机器，解析域名：
~]# vi /var/named/od.com.zone
serial 前滚一位
portal A 10.4.7.10
~]# systemctl restart named
~]# dig -t A portal.od.com @10.4.7.11 +short
# out: 10.4.7.10
~~~
![1583219385915](assets/1583219385915.png)
~~~
# 22机器，应用资源配置清单：
~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/cm.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/dp.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/svc.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/ingress.yaml
~~~
Update 第六章——在K8S中集成Apollo配置中心.md
2020-03-28 08:32:41 +08:00
可以查看起来的portal的logs日志，可能稍微有些慢
Create 第六章——在K8S中集成Apollo配置中心.md
2020-03-15 21:41:40 +08:00
[浏览器输入portal.od.com](portal.od.com)
```
Username: apollo
Password: admin
```
![1583221254589](assets/1583221254589.png)
> 第一件事，修改密码，任何开源软件的第一件事，就是修改默认密码
>
Update 第六章——在K8S中集成Apollo配置中心.md
2020-03-20 18:53:41 +08:00
![1584699514174](assets/1584699514174.png)
Create 第六章——在K8S中集成Apollo配置中心.md
2020-03-15 21:41:40 +08:00
![1583221587411](assets/1583221587411.png)
![1583222001345](assets/1583222001345.png)
去看一下对接数据库情况
![1583222020754](assets/1583222020754.png)
~~~
~~~
试试查询，查询key，也试试使用保存更新
~~~
key:organizations
value:[{"orgId":"ben01","orgName":"Linux学院"},{"orgId":"ben02","orgName":"云计算学院"},{"orgId":"ben03","orgName":"Python学院"},{"orgId":"ben03","orgName":"大数据学院"}]
~~~
![1583222152917](assets/1583222152917.png)
~~~
# 11机器，查询是否更新：
~]# mysql -uroot -p
none)]> use ApolloPortalDB;
ApolloPortalDB]> select * from ServerConfig\G
~~~
![1583222202986](assets/1583222202986.png)
成功
使用Apollo，创建项目，提交
![1583222246074](assets/1583222246074.png)
~~~
# 填入对应参数，并提交：
AppId:dubbo-demo-service
~~~
![1583222350430](assets/1583222350430.png)
![1583222393771](assets/1583222393771.png)
完成
### dubbo服务提供者连接Apollo实战
首先我们在git新建一个Apollo分支，以下是操作方法
##### git创建分支并上传代码（或者直接fork我），注意，下图配的是gitlab里面web的，现在你要操作的是自己git的service，因为service没截图，所以我用web的替代
1. 切换身份
~~~
# 上传项目的文件夹打开，然后cd到附近的新建的apollo文件夹，克隆项目并切换身份：
$ git http://gitlab.od.com:10000/909336740/dubbo-demo-web.git
$ cd dubbo-demo-web/
$ git branch apollo
$ git checkout apollo
~~~
![1583287244483](assets/1583287244483.png)
将Apollo分支的文件全部拉来这个文件夹，选择替换重名文件
![1583287180864](assets/1583287180864.png)
2. 把apollo的代码全部拉过去覆盖，并上传代码
~~~
$ git add .
$ git commit -m "apollo commit#1"
$ git push -u origin apollo
~~~
![1583287633875](assets/1583287633875.png)
![1582680560428](assets/1582680560428.png)
完成，当然你可以直接fork我的
![1582680588102](assets/1582680588102.png)
#### 继续主线
在Apollo portal里创建相应两个配置项
![1582680643369](assets/1582680643369.png)
新增配置项，并提交
![1583223092507](assets/1583223092507.png)
![1583223148433](assets/1583223148433.png)
发布
![1583223168662](assets/1583223168662.png)
![1583223190037](assets/1583223190037.png)
我们已经配置好了，然后需要用Jenkins制作镜像
~~~
# 填入对应参数
app_name: dubbo-demo-service
image_name: app/dubbo-demo-service
git_repo: https://gitee.com/benjas/dubbo-demo-service.git
git_ver: apollo
add_tag: 200303_1615
target_dir: ./dubbo-server/target
base_image: base/jre8:8u112
~~~
> 注意，我这里用的是gitlab，因为网络问题，你用自己的公网git即可
![1583223413973](assets/1583223413973.png)
去#38的console里查看情况
![1583223612917](assets/1583223612917.png)
#### 报错问题：
![1581324514061](assets/1581324514061.png)
该报错的原因是mirrors没有改对
~~~
# 200机器：
cd /data/nfs-volume/jenkins_home/maven-3.6.1-8u232/conf
conf]# vi settings.xml
~~~
修改完上面的内容后保存再去重新构建
#### 继续主线
![1583223754285](assets/1583223754285.png)
镜像制作完成
~~~
# 200机器，修改指定的镜像，并添加部分内容：
cd /data/k8s-yaml/dubbo-demo-service/
dubbo-demo-service]# vi dp.yaml
image: harbor.od.com/app/dubbo-demo-service:apollo_200303_1615
ports:
- containerPort: 20880
protocol: TCP
env:
- name: JAR_BALL
value: dubbo-server.jar
- name: C_OPTS
value: -Denv=dev -Dapollo.meta=http://config.od.com
imagePullPolicy: IfNotPresent
~~~
![1583223916007](assets/1583223916007.png)
~~~
# 22机器，应用：
~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-demo-service/dp.yaml
# out: deployment.extensions/dubbo-demo-service configured
~~~
![1583224029500](assets/1583224029500.png)
去LOGS查看，可以看到连接了Apollo
![1583227298634](assets/1583227298634.png)
> PS：这里的pod名字你可能觉得不一样，因为我镜像弄错了，重新改了一遍
![1583227711909](assets/1583227711909.png)
当你把service扩容成两个后
![1583227780562](assets/1583227780562.png)
可以看到变成两个实例
![1583227821344](assets/1583227821344.png)
在monitor里可以看到端口是20880（注意必须连接的是zk1，怎么修改上面有，先改cm，然后删掉pod让它自动重启）
![1583229093925](assets/1583229093925.png)
![1583229141970](assets/1583229141970.png)
我们去Apollo改一下端口，改成20881，并发布
![1583229265526](assets/1583229265526.png)
![1583229275645](assets/1583229275645.png)
重启两个service的pods（就是删掉让它们自动重启）
![1583229312442](assets/1583229312442.png)
刷新dubbo-monitor，可以看到端口已经变成20881
![1583229337601](assets/1583229337601.png)
还可以改成zk2
![1583229498782](assets/1583229498782.png)
> 作业：改成zk2后，需要删掉对应的pod，然后让其也在zk2，然后还需要把monitor也改成zk2，方法在上面有。这个作业一定要做，不然下面会报错，具体是哪里报错我会提示
最后我改回来了20880端口，但是依然用的zk2
完成
### dubbo服务消费者连接Apollo实战
同样，我是有Apollo分支的，操作方法和上面的service服务一样
![1582682538181](assets/1582682538181.png)
创建项目
![1583287914567](assets/1583287914567.png)
新增配置，并发布
![1583289334584](assets/1583289334584.png)
![1583289365597](assets/1583289365597.png)
用Jenkins构建dubbo消费者
~~~
# 填入对应参数
app_name: dubbo-demo-consumer
image_name: app/dubbo-demo-consumer
git_repo: http://gitlab.od.com:10000/909336740/dubbo-demo-web.git
git_ver: apollo
add_tag: 200304_1040
target_dir: ./dubbo-client/target
base_image: base/jre8:8u112
~~~
> 注意，我这里用的是gitlab，因为网络问题，你用自己的公网git即可
![1583290002445](assets/1583290002445.png)
![1583290022279](assets/1583290022279.png)
![1583290134412](assets/1583290134412.png)
~~~
# 200机器，修改配置：
cd /data/k8s-yaml/dubbo-demo-consumer/
dubbo-demo-consumer]# vi dp.yaml
image: harbor.od.com/app/dubbo-demo-service:apollo_200304_1040
ports:
- containerPort: 8080
protocol: TCP
- containerPort: 20880
protocol: TCP
env:
- name: JAR_BALL
value: dubbo-client.jar
- name: C_OPTS
value: -Denv=dev -Dapollo.meta=http://config.od.com
imagePullPolicy: IfNotPresent
~~~
![1583290251350](assets/1583290251350.png)
~~~
# 应用资源配置清单，22机器：
~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-demo-consumer/dp.yaml
# out: deployment.extensions/dubbo-demo-consumer configured
~~~
![1583290331910](assets/1583290331910.png)
![1583291244653](assets/1583291244653.png)
Update 第六章——在K8S中集成Apollo配置中心.md
2020-03-28 08:32:41 +08:00
再去Applications可以看到已经起来了
Create 第六章——在K8S中集成Apollo配置中心.md
2020-03-15 21:41:40 +08:00
~~~
# 浏览器访问demo.od.com/hello?name=apollo
~~~
![1583291263893](assets/1583291263893.png)
> 如果你登录的是这个界面，那么你没有做作业：改service的zk1成zk2，这个consumer已经是zk2的，而service还是zk1，当然不能响应
>
> ![1583290792208](assets/1583290792208.png)
#### 实现代码迭代
修改代码，然后commit
![1582682749719](assets/1582682749719.png)
![1582682870930](assets/1582682870930.png)
Jenkins构建
~~~
# 填入对应参数
app_name: dubbo-demo-consumer
image_name: app/dubbo-demo-consumer
git_repo: http://gitlab.od.com:10000/909336740/dubbo-demo-web.git
git_ver: apollo
add_tag: 200304_1145
target_dir: ./dubbo-client/target
base_image: base/jre8:8u112
~~~
> 注意，我这里用的是gitlab，因为网络问题，你用自己的公网git即可
![1583293563142](assets/1583293563142.png)
![1583293832408](assets/1583293832408.png)
~~~
# 200机器修改使用的镜像：
cd /data/k8s-yaml/dubbo-demo-consumer/
dubbo-demo-consumer]# vi dp.yaml
image: harbor.od.com/app/dubbo-demo-service:apollo_191208_1640
~~~
~~~
# 22机器应用：
~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-demo-consumer/dp.yaml
# out: deployment.extensions/dubbo-demo-consumer configured
~~~
![1583294170202](assets/1583294170202.png)
~~~
# 浏览器访问demo.od.com/hello?name=apollo
~~~
![1583294184192](assets/1583294184192.png)
### 实战Apollo分环境管理dubbo服务-交付Apollo-configservice
> 我们要实现测试环境和生产环境只需要打包一个镜像
>
先开始分环境
~~~
# 11机器，解析域名：
~]# vi /var/named/od.com.zone
serial 前滚一位
zk-test A 10.4.7.11
zk-prod A 10.4.7.12
~]# systemctl restart named
~]# dig -t A zk-test.od.com +short
# out: 10.4.7.11
~]# dig -t A zk-prod.od.com +short
# out: 10.4.7.12
~~~
关闭deployment里的消费者和服务者（改成scale 0，先消费者后服务者）
![1583301119606](assets/1583301119606.png)
~~~
# 创建两个名称空间， 21机器：
~]# kubectl create ns test
# out:namespace/test created
~]# kubectl create secret docker-registry harbor --docker-server=harbor.od.com --docker-username=admin --docker-password=Harbor12345 -n test
# out:secret/harbor created
~]# kubectl create ns prod
# out:namespace/prod created
~]# kubectl create secret docker-registry harbor --docker-server=harbor.od.com --docker-username=admin --docker-password=Harbor12345 -n prod
# out:secret/harbor created
~~~
![1583301237979](assets/1583301237979.png)
去看一下dashboard里面的Namespaces
![1583301259487](assets/1583301259487.png)
把admin、portal、config都scale成0
![1583301320962](assets/1583301320962.png)
~~~
# 11机器，建库：
~]# vi apolloconfig.sql
# 增加Test关键字
~~~
![1583301385286](assets/1583301385286.png)
~~~
# 11机器，测试环境：
~]# mysql -uroot -p < apolloconfig.sql
~]# mysql -uroot -p
none)]> show databases;
none)]> use ApolloConfigTestDB;
ApolloConfigTestDB]> select * from ServerConfig\G
ApolloConfigTestDB]> update ApolloConfigTestDB.ServerConfig set ServerConfig.Value="http://config-test.od.com/eureka" where ServerConfig.Key="eureka.service.url";
ApolloConfigTestDB]> select * from ServerConfig\G
ApolloConfigTestDB]> grant INSERT,DELETE,UPDATE,SELECT on ApolloConfigTestDB.* to "apolloconfig"@"10.4.7.%" identified by "123456";
~~~
![1583301481115](assets/1583301481115.png)
~~~
# 11机器，生产环境：
~]# vi apolloconfig.sql
# 修改成Prod关键字
~]# mysql -uroot -p < apolloconfig.sql
~]# mysql -uroot -p
none)]> show databases;
none)]> use ApolloConfigProdDB;
ApolloConfigProdDB]> select * from ServerConfig\G
ApolloConfigProdDB]> update ApolloConfigProdDB.ServerConfig set ServerConfig.Value="http://config-prod.od.com/eureka" where ServerConfig.Key="eureka.service.url";
ApolloConfigProdDB]> select * from ServerConfig\G
ApolloConfigProdDB]> grant INSERT,DELETE,UPDATE,SELECT on ApolloConfigProdDB.* to "apolloconfig"@"10.4.7.%" identified by "123456";
~~~
![1583301626767](assets/1583301626767.png)
~~~~
# 11机器，修改支持类型：
none)]> use ApolloPortalDB;
ApolloPortalDB]> show tables;
ApolloPortalDB]> select * from ServerConfig\G
ApolloPortalDB]> update ServerConfig set Value='fat,pro' where Id=1;
ApolloPortalDB]> select * from Serverconfig\G
~~~~
改后
![1583302030474](assets/1583302030474.png)
> PS按理来说你应该只有dev，但是好像Apollo更新了，一开始就有4种支持的类型，只要你确保有fat和pro即可
~~~
# 200机器：
cd /data/k8s-yaml/apollo-portal/
# 需改以下内容
apollo-portal]# vi cm.yaml
apollo-env.properties: |
fat.meta=http://config-test.od.com
pro.meta=http://config-prod.od.com
~~~
![1583302159311](assets/1583302159311.png)
~~~
# 22机器，应用：
~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/cm.yaml
# out: configmap/apollo-portal configured
~~~
![1583302232037](assets/1583302232037.png)
完成
### 实战使用Apollo分环境管理dubbo服务——交付Apollo-portal和adminservice
~~~
# 制作资源配置清单,200机器：
cd /data/k8s-yaml/
k8s-yaml]# mkdir -pv test/{apollo-configservice,apollo-adminservice,dubbo-demo-service,dubbo-demo-consumer}
k8s-yaml]# mkdir -pv prod/{apollo-configservice,apollo-adminservice,dubbo-demo-service,dubbo-demo-consumer}
k8s-yaml]# cd test/apollo-configservice/
apollo-configservice]# cp -a /data/k8s-yaml/apollo-configservice/cm.yaml .
apollo-configservice]# cp -a /data/k8s-yaml/apollo-configservice/svc.yaml .
apollo-configservice]# cp -a /data/k8s-yaml/apollo-configservice/dp.yaml .
apollo-configservice]# cp -a /data/k8s-yaml/apollo-configservice/ingress.yaml .
# 修改成test，三处
apollo-configservice]# vi cm.yaml
namespace: test
spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigTestDB?characterEncoding=utf8
spring.service.url = http://config-test.od.com/eureka
# 修改成test，一处
apollo-configservice]# vi dp.yaml
namespace: test
# 修改成test，一处
apollo-configservice]# vi svc.yaml
namespace: test
# 修改成test，两处
apollo-configservice]# vi ingress.yaml
namespace: test
- host: config-test.od.com
~~~
~~~
# 11机器，解析域名：
~]# vi /var/named/od.com.zone
serial 前滚一位
config-test A 10.4.7.10
config-prod A 10.4.7.10
~]# systemctl restart named
~~~
![1582683284142](assets/1582683284142.png)
~~~
# 应用资源配置清单，22机器：
~]# kubectl apply -f http://k8s-yaml.od.com/test/apollo-configservice/cm.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/test/apollo-configservice/dp.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/test/apollo-configservice/svc.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/test/apollo-configservice/ingress.yaml
~~~
~~~~
# 200机器，制作prod的：
cd /data/k8s-yaml/prod/apollo-configservice/
apollo-configservice]# cp ../../test/apollo-configservice/*.yaml .
# 修改成prod，三处
apollo-configservice]# vi cm.yaml
namespace: prod
spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigProdDB?characterEncoding=utf8
spring.service.url = http://config-prod.od.com/eureka
# 修改成prod，一处
apollo-configservice]# vi dp.yaml
namespace: prod
# 修改成prod，一处
apollo-configservice]# vi svc.yaml
namespace: prod
# 修改成prod，两处
apollo-configservice]# vi ingress.yaml
namespace: prod
- host: config-prod.od.com
~~~~
~~~~
# 应用资源清单，22机器：
~]# kubectl apply -f http://k8s-yaml.od.com/prod/apollo-configservice/cm.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/prod/apollo-configservice/dp.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/prod/apollo-configservice/svc.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/prod/apollo-configservice/ingress.yaml
~~~~
![1581347383718](assets/1581347383718.png)
确认你的nslookup.exe 有没有（一般是有的）
![1581347525407](assets/1581347525407.png)
~~~
# 访问config-test.od.com
config.od.com已经没了
~~~
![1583304944456](assets/1583304944456.png)
~~~~
# 访问config-prod.od.com
prod可能还没好，接着往下面做，test能起来，prod肯定也能起来
~~~~
![1583304959338](assets/1583304959338.png)
> 可以看到test和prod在两个不同的容器里，这就是我们模拟的分环境
~~~
# 200机器：
cd /data/k8s-yaml/test/apollo-adminservice/
apollo-adminservice]# cp -a /data/k8s-yaml/apollo-adminservice/*.yaml .
# 修改成test，三处
apollo-adminservice]# vi cm.yaml
namespace: test
spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigTestDB?
spring.service.url = http://config-test.od.com/eureka
# 修改成test，一处
apollo-adminservice]# vi dp.yaml
namespace: test
apollo-adminservice]# cd ../../prod/apollo-adminservice
apollo-adminservice]# cp -a ../../test/apollo-adminservice/*.yaml .
# 修改成prod，三处
apollo-adminservice]# vi cm.yaml
namespace: prod
spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigProdDB?characterEncoding=utf8
spring.service.url = http://config-prod.od.com/eureka
# 修改成prod，一处
apollo-adminservice]# vi dp.yaml
namespace: prod
~~~
~~~
# 应用，22机器：
~]# kubectl apply -f http://k8s-yaml.od.com/test/apollo-adminservice/cm.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/test/apollo-adminservice/dp.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/prod/apollo-adminservice/cm.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/prod/apollo-adminservice/dp.yaml
~~~
![1583304745055](assets/1583304745055.png)
![1583304817198](assets/1583304817198.png)
~~~
# 11机器，清除数据：
mysql -uroot -p
none)]> use ApolloPortalDB;
ApolloProtalDB]> select * from App;
ApolloProtalDB]> select * from AppNamespace;
ApolloProtalDB]> truncate table App;
ApolloProtalDB]> truncate table AppNamespace;
~~~
![1583305105464](assets/1583305105464.png)
把portal scale成1
![1583305172038](assets/1583305172038.png)
使用查询功能查询下
![1583305255615](assets/1583305255615.png)
![1583305298870](assets/1583305298870.png)
成功
### 实战发布dubbo连接Apollo到不同环境
创建项目
[portal.od.com](portal.od.com)
![1583305340551](assets/1583305340551.png)
![1583305381813](assets/1583305381813.png)
现在也可以看到环境列表有两个了
![1583305431837](assets/1583305431837.png)
测试环境新增配置项，并发布
![1583305535156](assets/1583305535156.png)
![1583305580379](assets/1583305580379.png)
![1583305601716](assets/1583305601716.png)
生产环境新增配置项，并发布
![1583305663692](assets/1583305663692.png)
![1583312777104](assets/1583312777104.png)
![1583312803655](assets/1583312803655.png)
service配置好了，再配置consumer
![1583305834462](assets/1583305834462.png)
测试环境新增配置项，并发布
![1583305939762](assets/1583305939762.png)
![1583305965712](assets/1583305965712.png)
生产环境新增配置项，并发布
![1583306013898](assets/1583306013898.png)
![1583306034551](assets/1583306034551.png)
![1583306049712](assets/1583306049712.png)
~~~
# 200机器，制作测试环境service：
cd /data/k8s-yaml/test/dubbo-demo-service/
dubbo-demo-service]# cp -a /data/k8s-yaml/dubbo-demo-service/*.yaml .
# 修改三处
dubbo-demo-service]# vi dp.yaml
namespace: test
env:
- name: JAR_BALL
value: dubbo-server.jar
- name: C_OPTS
value: -Denv=fat -Dapollo.meta=http://config-test.od.com
~~~
~~~
# 应用，22机器：
~]# kubectl apply -f http://k8s-yaml.od.com/test/dubbo-demo-service/dp.yaml
~~~
![1583306650482](assets/1583306650482.png)
~~~
# 200机器，制作测试环境consumer：
cd /data/k8s-yaml/test/dubbo-demo-consumer/
dubbo-demo-consumer]# cp -a /data/k8s-yaml/dubbo-demo-consumer/*.yaml .
# 修改三处
dubbo-demo-consumer]# vi dp.yaml
namespace: test
- name: JAR_BALL
value: dubbo-client.jar
- name: C_OPTS
value: -Denv=fat -Dapollo.meta=http://config-test.od.com
# 修改一处
dubbo-demo-consumer]# vi svc.yaml
namespace: test
# 修改一处
dubbo-demo-consumer]# vi ingress.yaml
namespace: test
- host: demo-test.od.com
~~~
~~~
# 11机器，新增解析：
~]# vi /var/named/od.com.zone
serial 前滚一位
demo-test A 10.4.7.10
~]# systemctl restart named
~~~
![1583306813921](assets/1583306813921.png)
~~~
# 应用，22机器：
~]# kubectl apply -f http://k8s-yaml.od.com/test/dubbo-demo-consumer/dp.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/test/dubbo-demo-consumer/svc.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/test/dubbo-demo-consumer/ingress.yaml
~~~
查看dashboard的启动情况
![1583307076131](assets/1583307076131.png)
此时的monitor还是zk2，我们改成zk-test
![1583307422408](assets/1583307422408.png)
![1583307483922](assets/1583307483922.png)
update然后删掉对应的pod让它自动重启
![1583307517718](assets/1583307517718.png)
![1583307567245](assets/1583307567245.png)
![1583307586774](assets/1583307586774.png)
~~~
# 浏览器输入：demo-test.od.com/hello?name=test
~~~
![1583310055796](assets/1583310055796.png)
完成
开始做生产环境
~~~
# 11机器，新增解析：
~]# vi /var/named/od.com.zone
serial 前滚一位
demo-prod A 10.4.7.10
~]# systemctl restart named
~~~
![1581350791800](assets/1581350791800.png)
~~~
# 200机器，制作生产环境service：
cd /data/k8s-yaml/prod/dubbo-demo-service/
dubbo-demo-service]# cp -a ../../test/dubbo-demo-service/*.yaml .
# 共修改三处
dubbo-demo-service]# vi dp.yaml
namespace: prod
env:
- name: JAR_BALL
value: dubbo-server.jar
- name: C_OPTS
value: -Denv=pro -Dapollo.meta=http://apollo-configservice:8080
~~~
[^关于http://apollo-configservice:8080]: 直接用这个和用http://config-test.od.com是没有区别的，http://config-test.od.com是多了一层反代，而apollo-configservice:8080是因为它们在同一名称空间里，所以可以这么使用
~~~
# 应用，22机器：
~]# kubectl apply -f http://k8s-yaml.od.com/prod/dubbo-demo-service/dp.yaml
~~~
![1583310378331](assets/1583310378331.png)
你还可以去LOGS日志里面看看
~~~
# 200机器，制作生产环境consumer：
cd /data/k8s-yaml/prod/dubbo-demo-consumer/
dubbo-demo-consumer]# cp -a /data/k8s-yaml/dubbo-demo-consumer/*.yaml .
# 共修改三处
dubbo-demo-consumer]# vi dp.yaml
namespace: prod
env:
- name: JAR_BALL
value: dubbo-client.jar
- name: C_OPTS
value: -Denv=pro -Dapollo.meta=http://apollo-configservice:8080
imagePullPolicy: IfNotPresent
# 共修改一处
dubbo-demo-consumer]# vi svc.yaml
namespace: prod
# 共修改两处
dubbo-demo-consumer]# vi ingress.yaml
namespace: prod
spec:
rules:
- host: demo-prod.od.com
~~~
~~~
# 应用，22机器：
~]# kubectl apply -f http://k8s-yaml.od.com/prod/dubbo-demo-consumer/dp.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/prod/dubbo-demo-consumer/svc.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/prod/dubbo-demo-consumer/ingress.yaml
~~~
~~~
# 浏览器输入：demo-prod.od.com/hello?name=prod
~~~
![1583313070558](assets/1583313070558.png)
完成
> 报错：![1583312899877](assets/1583312899877.png)
>
> 我去LOGS里面看了一下，发现我Apollo配得port写错了，写成了dubbo-port，应该是dubbo.port，改回来就可以了，删掉service得pod重新启动，删掉consumer的pod重新启动
### 实战演示项目提测，发版流程
> 模拟项目提测到发布上线，这里用得gitlab，你有可以了解一下gitlab是长什么样子的，还有怎么操作的
修改源代码，并commit
![1583313961163](assets/1583313961163.png)
![1583313975564](assets/1583313975564.png)
> gitlab跟gittee在更新代码的编号有所区别
![1583314181056](assets/1583314181056.png)
Jenkins构建
~~~
# 填入对应参数，然后build
app_name: dubbo-demo-consumer
image_name: app/dubbo-demo-consumer
git_repo: http://gitlab.od.com:10000/909336740/dubbo-demo-web.git
git_ver: 535826b1239fedba0df3799b7b3b8585d56e9e18
add_tag: 200304_1730
target_dir: ./dubbo-client/target
base_image: base/jre8:8u112
~~~
![1583314459402](assets/1583314459402.png)
![1583314584034](assets/1583314584034.png)
先到测试环境发布，修改tag
![1583314731675](assets/1583314731675.png)
![1583316573117](assets/1583316573117.png)
重启成功，刷新测试环境的页面，查看情况
![1583314816036](assets/1583314816036.png)
刷新生产环境的页面，查看情况，还没改变
![1583314831690](assets/1583314831690.png)
我们来修改prod的，这时候已经不需要再去Jenkins打包镜像了
![1583316530331](assets/1583316530331.png)
![1583316573117](assets/1583316573117.png)
更新完启动pod后
再来刷新生产环境的页面
![1583316620715](assets/1583316620715.png)
Update 第六章——在K8S中集成Apollo配置中心.md
2020-03-20 18:53:41 +08:00
已经改变，成功
Reference in New Issue Copy Permalink