minelona/k8s_PaaS
1
0
Fork 0
mirror of https://github.com/ben1234560/k8s_PaaS.git synced 2025-04-22 06:35:52 +08:00
Code Issues Packages Projects Releases Wiki Activity
k8s_PaaS/第六章——在K8S中集成Apollo配置中心.md

60 KiB
Raw Blame History

第六章——在K8S中集成Apollo配置中心

前言：

运维八荣八耻：

  • 以可配置为荣，以硬编码为耻
  • 以互备为荣，以单点为耻
  • 以随时重启为荣，以不能迁移为耻
  • 以整体交付为荣，以部分交付为耻
  • 以无状态为荣，以有状态为耻
  • 以标准化为荣，以特殊化为耻
  • 以自动化工具为荣，以手动和人肉为耻
  • 以无人值守为荣，以人工介入为耻

目前我们交付进K8S集群的两个dubbo微服务和monitor，它们的配置都是写死在容器里的

配置管理的现状：

  • 配置散乱格式不标准（XML、ini、conf、yaml...）
  • 主要采用本地静态配置，应用多副本集下配置修改麻烦
  • 易引发生产事故（测试环境、生产环境配置混用）
  • 配置缺乏安全审计和版本控制功能（config review）
  • 不同环境的应用，配置不同，造成多次打包，测试失败

配置中心是什么？：

  • 顾名思义，就是集中管理应用程序配置的“中心”

常见的配置中心：

  • SprigCloudConfig
  • K8S ConfigMap
  • Apollo：基于SprigCloudConfig
  • ...

目前具有争议的一般是使用Apollo还是SprigCloudConfig，看对比图，所以我们使用Apollo，而且Apollo是基于SprigCloudConfig，也就是你交付了Apollo也相当于交付了微服务

1582622382181

configmap使用详解

WHAT：就是为了让镜像 和 配置文件解耦，以便实现镜像的可移植性和可复用性，因为一个configMap其实就是一系列配置信息的集合，将来可直接注入到Pod中的容器使用

WHY：为了配合Apollo使用

使用configmap管理应用配置，需要先拆分环境，拆分未test和pro环境来模拟实际工作

先把dubbo的服务（消费者/服务站/监视者）都scale成0，这样就没有容器在里面跑了

1583162321302

1583162339688

拆分zk成测试环境和生产环境来模拟实际情况，如下：

主机名 角色 ip
HDSS7-11.host.com zk1.od.com(Test环境) 10.4.7.11
HDSS7-12.host.com zk2.od.com(Prod环境) 10.4.7.12

之前是11、12、21一共3台，我们先把21拆了

# 11/12/21机器，停掉zookeeper并删掉相关data文件和log文件的内容：
~]# cd /opt/zookeeper
zookeeper]# bin/zkServer.sh stop
# 停不掉就 kill -9 id 杀掉
zookeeper]# ps aux|grep zoo
zookeeper]# cd /data/zookeeper/data/
data]# rm -fr ./*
cd ../logs/
logs]# rm -fr ./*

kill -9：强制杀死该进程

1581059149533

# 11/12机器，修改zoo.cfg配置：
zookeeper]# cd /opt/zookeeper/conf
zookeeper]# vi zoo.cfg
# 把下面的三行都删掉，不需要组成集群，如图

1581059640346

# 11/12机器，启动zk：
cd /opt/zookeeper/
zookeeper]# bin/zkServer.sh start
zookeeper]# ps aux|grep zoo
zookeeper]# bin/zkServer.sh status

1584697777390

Mode: standalone 模式，拆分完成

# 200机器，准备资源配置清单：
~]# vi /data/k8s-yaml/dubbo-monitor/cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: dubbo-monitor-cm
  namespace: infra
data:
  dubbo.properties: |
    dubbo.container=log4j,spring,registry,jetty
    dubbo.application.name=simple-monitor
    dubbo.application.owner=ben1234560
    dubbo.registry.address=zookeeper://zk1.od.com:2181
    dubbo.protocol.port=20880
    dubbo.jetty.port=8080
    dubbo.jetty.directory=/dubbo-monitor-simple/monitor
    dubbo.charts.directory=/dubbo-monitor-simple/charts
    dubbo.statistics.directory=/dubbo-monitor-simple/statistics
    dubbo.log4j.file=/dubbo-monitor-simple/logs/dubbo-monitor.log
    dubbo.log4j.level=WARN

~]# vi /data/k8s-yaml/dubbo-monitor/dp2.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: dubbo-monitor
  namespace: infra
  labels: 
    name: dubbo-monitor
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: dubbo-monitor
  template:
    metadata:
      labels: 
        app: dubbo-monitor
        name: dubbo-monitor
    spec:
      containers:
      - name: dubbo-monitor
        image: harbor.od.com/infra/dubbo-monitor:latest
        ports:
        - containerPort: 8080
          protocol: TCP
        - containerPort: 20880
          protocol: TCP
        imagePullPolicy: IfNotPresent
        volumeMounts:
          - name: configmap-volume
            mountPath: /dubbo-monitor-simple/conf
      volumes:
        - name: configmap-volume
          configMap:
            name: dubbo-monitor-cm
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

# 对比两个dp有什么不同，200机器：
cd /data/k8s-yaml/dubbo-monitor/
dubbo-monitor]# vimdiff dp.yaml dp2.yaml
#:qall 退出

**vimdiff **：编辑同一文件的不同历史版本，对各文件的内容进行比对与调整

没有vimdiff 的下载 yum install vim -y

1583198466051

这里两个文件不同的地方，注意，这里的image的版本因为我的操作问题所以一个v2一个没有v，你的应该是都没有的

# 200机器，替换dp：
dubbo-monitor]# mv dp.yaml /tmp/
dubbo-monitor]# mv dp2.yaml dp.yaml

mv：如果mv后是目录，则是移动，如果是文件名，则是更改名字

# 应用资源配置清单，22机器：
~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-monitor/cm.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-monitor/dp.yaml
# 看一下相关容器有没有起来

1583198817470

1583198829279

1583198887392

1583199215057

我们改成zk2

1583199496585

在删掉这个pod重启

1583199525277

刷新页面变成zk2了

1583199653453

完成

报错信息

flannel重启报错问题

# flannel重启需要增加以下内容，添加在最下面:
21 ~]# vi /etc/sipervisord.d/flannel.ini
killasgroup=true
stopasgroup=true

如何排错：

# 查看时间是否正常
date

# 查看kubernetes的报错
21 ~]# tail -fn 200/data/logs/kubernetes/kube-kubelet/kubelet.stdout.log

# 可以重启docker 和 kubelet
21 ~]# systemctl restart docker 
21 ~]# ps aux|grep kubelet
21 ~]# kill -9 id

# 查看supervisor的状态
21 ~]# supervisorctl status

# 查看路由状态，是否能ping通其它机器
21 ~]# route -n
21 ~]# ping 172.7.22.2

# 查看flannel情况
21 ~]# tail -fn 200 /data/logs/flanneld/flanneld.stdout.log
# 12flannel主机器上get网络，如果error则加入backend
12 ~]# ./etcdctl get /coreos.com/network/config

# 报错iptables，让机器一重启就自动使用，21/22机器
~]# service iptables save
# out: [ok]

cm对象的另一种创建方法（可不做）：

# 22机器：
cd /opt/kubernetes/server/bin/conf
# 当前目录下要又kubelet.kubeconfig的文件
conf]# kubectl create cm kubelet-cm --from-file=./kubelet.kubeconfig
#out: configmap/kubelet-cm created

可以在dashoboard的default名称空间下Config Maps看到

随后记得删除

官方Apollo框架：

1587274472834

简化Apollo框架：

1584698746125

1584698613252

Client通过推拉结合和ConfigService交互，然后ConfigService去拿ConfigDB里面的配置给Client端返回去。

Portal是Apollo的一个仪表盘，通过调用AdminService去同步修改ConfigDB里面的配置

先交付ConfigService，然后交付AdminService，最后加portal

交付Apollo-ConfigService到K8S

官网https://github.com/ctripcorp/apollo/

安装部署MySQL数据库
# 更新yum源，11机器：
~]# vi /etc/yum.repos.d/MariaDB.repo
[mariadb]
name = MariaDB
baseurl = https://mirrors.ustc.edu.cn/mariadb/yum/10.1/centos7-amd64/
gpgkey=https://mirrors.ustc.edu.cn/mariadb/yum/RPM-GPG-KEY-MariaDB
gpgcheck=1

# 导入GPG-KEY
~]# rpm --import https://mirrors.ustc.edu.cn/mariadb/yum/RPM-GPG-KEY-MariaDB
~]# yum list mariadb --show-duplicates
# out: mariadb.x86_64
~]# yum makecache
# out:Metadata Cache Created
# 更新数据库版本
~]# yum list mariadb-server --show-duplicates
~]# yum install mariadb-server -y
# 编辑基础配置
# 增加部分内容
~] vi /etc/my.cnf.d/server.cnf
[mysqld]
character_set_server = utf8mb4
collation_server = utf8mb4_general_ci
init_connect = "SET NAMES 'utf8mb4'"

1583202954447

# 11机器，修改基础配置,并启动：
# 在改内容下增加，字符集
~]# vi /etc/my.cnf.d/mysql-clients.cnf
[mysql]
default-character-set = utf8mb4

~]# systemctl start mariadb

1583202999075

# 11机器，设置,mysql密码：
~]# mysqladmin -uroot password
# 这里是设置密码：123456
~]# mysql -uroot -p
# 这里是输入密码
none)]> \s
# 确定SERVER\DB\CLIENT\CONN 都是utf8

1583203114762

# 11机器：
none)]> show databases;
none)]> drop database test;
none)]> exit
# 查看数据库是否正常
~]# ps aux|grep mysql
~]# netstat luntp|grep 3306
~]# wget https://raw.githubusercontent.com/ctripcorp/apollo/1.5.1/scripts/db/migration/configdb/V1.0.0__initialization.sql -O apolloconfig.sql
~]# cat apolloconfig.sql
~]# mysql -uroot -p < apolloconfig.sql
# 输入密码
~] mysql -uroot -p
# 输密码
none)]> show databases;

1581148090694

# 11机器，创建其它权限用户而不是直接给root权限：
none)]> use ApolloConfigDB;
ApolloConfigDB]> show tables;
ApolloConfigDB]> grant INSERT,DELETE,UPDATE,SELECT on ApolloConfigDB.* to 'apolloconfig'@'10.4.7.%' identified by "123456";
ApolloConfigDB]> select user,host from mysql.user;

1583203450336

# 11机器,修改初始数据：
ApolloConfigDB]> show tables;
ApolloConfigDB]> select * from ServerConfig\G
ApolloConfigDB]> update ApolloConfigDB.ServerConfig set ServerConfig.Value="http://config.od.com/eureka" where ServerConfig.Key="eureka.service.url";
ApolloConfigDB]> select * from ServerConfig\G

原：

1583203490999

改后

1583203522358

# 11机器，解析域名：
~]# vi /var/named/od.com.zone
serial 前滚一位
config             A    10.4.7.10

~]# systemctl restart named

1583203561891

# 21机器，测试下21机器能不能查到(11机器是肯定能查到的)：
~]# dig -t A config.od.com @192.168.0.2 +short
# out:10.4.7.10

包的官网地址1583204125452

# 200机器，制作docker镜像：
cd /opt/src
src]# wget https://github.com/ctripcorp/apollo/releases/download/v1.5.1/apollo-configservice-1.5.1-github.zip
# 或者去官网下载或者用我的上传的包
src]# mkdir /data/dockerfile/apollo-configservice
src]# unzip -o apollo-configservice-1.5.1-github.zip -d /data/dockerfile/apollo-configservice
src]# cd /data/dockerfile/apollo-configservice/
apollo-configservice]# rm -rf apollo-configservice-1.5.1-sources.jar
apollo-configservice]# ll

1581232250106

# 11机器，解析域名：
~]# vi /var/named/od.com.zone
serial 前滚一位
mysql              A    10.4.7.11

~]# systemctl restart named
~]# dig -t A mysql.od.com @10.4.7.11 +short
# out: 10.4.7.11

1582636827536

# 200机器，修改账户密码：
cd /data/dockerfile/apollo-configservice/config
config]# vi application-github.properties
spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigDB?characterEncoding=utf8
spring.datasource.username = apolloconfig
spring.datasource.password = 123456

1581232483973

# 200机器：
cd /data/dockerfile/apollo-configservice/scripts
scripts]# rm -f shutdown.sh
# 全部删掉，换成以下内容
scripts]# vi startup.sh
#!/bin/bash
SERVICE_NAME=apollo-configservice
## Adjust log dir if necessary
LOG_DIR=/opt/logs/apollo-config-server
## Adjust server port if necessary
SERVER_PORT=8080
APOLLO_CONFIG_SERVICE_NAME=$(hostname -i)
SERVER_URL="http://${APOLLO_CONFIG_SERVICE_NAME}:${SERVER_PORT}"

## Adjust memory settings if necessary
export JAVA_OPTS="-Xms128m -Xmx128m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=256m -XX:MaxNewSize=256m -XX:SurvivorRatio=8"

## Only uncomment the following when you are using server jvm
#export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"

########### The following is the same for configservice, adminservice, portal ###########
export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"

# Find Java
if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
    javaexe="$JAVA_HOME/bin/java"
elif type -p java > /dev/null 2>&1; then
    javaexe=$(type -p java)
elif [[ -x "/usr/bin/java" ]];  then
    javaexe="/usr/bin/java"
else
    echo "Unable to find Java"
    exit 1
fi

if [[ "$javaexe" ]]; then
    version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
    version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
    # now version is of format 009003 (9.3.x)
    if [ $version -ge 011000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    elif [ $version -ge 010000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    elif [ $version -ge 009000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    else
        JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
        JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
        JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled  -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
    fi
fi

printf "$(date) ==== Starting ==== \n"

cd `dirname $0`/..
chmod 755 $SERVICE_NAME".jar"
./$SERVICE_NAME".jar" start

rc=$?;

if [[ $rc != 0 ]];
then
    echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
    exit $rc;
fi

tail -f /dev/null

Apollo官方文档

上述代码是从Apollo官网拉下来的，不过第7行多了一行APOLLO_CONFIG_SERVICE_NAME=$(hostname -i)

其中export JAVA_OPTS修改了下资源，改小了

# 200机器，给权限：
scripts]# chmod u+x startup.sh
scripts]# ll

1581233647557

# 200机器，做dockerfile：
cd /data/dockerfile/apollo-configservice
apollo-configservice]# vi Dockerfile
FROM 909336740/jre8:8u112

ENV VERSION 1.5.1

RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
    echo "Asia/Shanghai" > /etc/timezone

ADD apollo-configservice-${VERSION}.jar /apollo-configservice/apollo-configservice.jar
ADD config/ /apollo-configservice/config
ADD scripts/ /apollo-configservice/scripts

CMD ["/apollo-configservice/scripts/startup.sh"]

参考Apollo官网文档

1583219072270

# 200机器,制作容器：
apollo-configservice]# docker build . -t harbor.od.com/infra/apollo-configservice:v1.5.1
apollo-configservice]# docker push harbor.od.com/infra/apollo-configservice:v1.5.1

1583205129961

# 制作资源配置清单，200机器：
~]# mkdir /data/k8s-yaml/apollo-configservice
~]# cd /data/k8s-yaml/apollo-configservice
apollo-configservice]# vi cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: apollo-configservice-cm
  namespace: infra
data:
  application-github.properties: |
    # DataSource
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigDB?characterEncoding=utf8
    spring.datasource.username = apolloconfig
    spring.datasource.password = 123456
    eureka.service.url = http://config.od.com/eureka
  app.properties: |
    appId=100003171

apollo-configservice]# vi dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: apollo-configservice
  namespace: infra
  labels: 
    name: apollo-configservice
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: apollo-configservice
  template:
    metadata:
      labels: 
        app: apollo-configservice 
        name: apollo-configservice
    spec:
      volumes:
      - name: configmap-volume
        configMap:
          name: apollo-configservice-cm
      containers:
      - name: apollo-configservice
        image: harbor.od.com/infra/apollo-configservice:v1.5.1
        ports:
        - containerPort: 8080
          protocol: TCP
        volumeMounts:
        - name: configmap-volume
          mountPath: /apollo-configservice/config
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

apollo-configservice]# vi svc.yaml
kind: Service
apiVersion: v1
metadata: 
  name: apollo-configservice
  namespace: infra
spec:
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080
  selector: 
    app: apollo-configservice

apollo-configservice]# vi ingress.yaml
kind: Ingress
apiVersion: extensions/v1beta1
metadata: 
  name: apollo-configservice
  namespace: infra
spec:
  rules:
  - host: config.od.com
    http:
      paths:
      - path: /
        backend: 
          serviceName: apollo-configservice
          servicePort: 8080

1583205384294

# 应用资源配置清单，22机器：
~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/cm.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/dp.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/svc.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/ingress.yaml

1583206365045

资源给的少，可能稍微慢了一些，点进去->点右上角的LOGS，日志有点多，记得点击右下角的翻页

浏览器访问config.od.com

鼠标对着起来的Apollo，可以看到右下角有网址

1583206733615

# 22机器，curl：
~]# curl http://172.7.22.5:8080/info

1583206773094

成功

Apollo-ConfigService连接数据库IP分析

# 11机器：
~]# mysql -uroot -p
none)]> show processlist;

1583207560779

1583207586526

原本是172.7.22.5，连接到数据的是10.4.7.22，因为做了NAT转换，它带了面具

交付Apollo-adminservice

官网下载https://github.com/ctripcorp/apollo/releases/tag/v1.5.1

1584698854148

# 200机器：
cd /opt/src
src]# wget https://github.com/ctripcorp/apollo/releases/download/v1.5.1/apollo-adminservice-1.5.1-github.zip
src]# mkdir /data/dockerfile/apollo-adminservice
src]# unzip -o apollo-adminservice-1.5.1-github.zip -d /data/dockerfile/apollo-adminservice
src]# cd /data/dockerfile/apollo-adminservice
apollo-adminservice]# rm -fr apollo-adminservice-1.5.1-sources.jar
apollo-adminservice]# rm -f apollo-adminservice.conf
apollo-adminservice]# cd scripts/
scripts]# rm -f shutdown.sh
# 删掉原来的全部内容，添加以下新的内容
scripts]# vi startup.sh
#!/bin/bash
SERVICE_NAME=apollo-adminservice
## Adjust log dir if necessary
LOG_DIR=/opt/logs/apollo-admin-server
## Adjust server port if necessary
SERVER_PORT=8080
APOLLO_ADMIN_SERVICE_NAME=$(hostname -i)
# SERVER_URL="http://localhost:${SERVER_PORT}"
SERVER_URL="http://${APOLLO_ADMIN_SERVICE_NAME}:${SERVER_PORT}"

## Adjust memory settings if necessary
#export JAVA_OPTS="-Xms2560m -Xmx2560m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=1536m -XX:MaxNewSize=1536m -XX:SurvivorRatio=8"

## Only uncomment the following when you are using server jvm
#export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"

########### The following is the same for configservice, adminservice, portal ###########
export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"

# Find Java
if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
    javaexe="$JAVA_HOME/bin/java"
elif type -p java > /dev/null 2>&1; then
    javaexe=$(type -p java)
elif [[ -x "/usr/bin/java" ]];  then
    javaexe="/usr/bin/java"
else
    echo "Unable to find Java"
    exit 1
fi

if [[ "$javaexe" ]]; then
    version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
    version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
    # now version is of format 009003 (9.3.x)
    if [ $version -ge 011000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    elif [ $version -ge 010000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    elif [ $version -ge 009000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    else
        JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
        JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
        JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled  -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
    fi
fi

printf "$(date) ==== Starting ==== \n"

cd `dirname $0`/..
chmod 755 $SERVICE_NAME".jar"
./$SERVICE_NAME".jar" start

rc=$?;

if [[ $rc != 0 ]];
then
    echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
    exit $rc;
fi

tail -f /dev/null

startup.sh官方地址

修改处为：

SERVER_PORT=8080 # 因为docker网络空间是互相隔离，用8080即可

APOLLO_ADMIN_SERVICE_NAME=$(hostname -i)

# 200机器，制作dockerfile：
cd /data/dockerfile/apollo-adminservice
apollo-adminservice]# ll
apollo-adminservice]# vi Dockerfile
FROM 909336740/jre8:8u112

ENV VERSION 1.5.1

RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
    echo "Asia/Shanghai" > /etc/timezone

ADD apollo-adminservice-${VERSION}.jar /apollo-adminservice/apollo-adminservice.jar
ADD config/ /apollo-adminservice/config
ADD scripts/ /apollo-adminservice/scripts

CMD ["/apollo-adminservice/scripts/startup.sh"]

apollo-adminservice]# docker build . -t harbor.od.com/infra/apollo-adminservice:v1.5.1
apollo-adminservice]# docker push harbor.od.com/infra/apollo-adminservice:v1.5.1

1583207853169

1583219005442

看以下harbor仓库有没有

1583208050305

重新复习一下交付过程：
搞定镜像-搞定资源配置清单-应用资源配置清单
# 200机器，制作资源配置清单：
mkdir /data/k8s-yaml/apollo-adminservice
cd /data/k8s-yaml/apollo-adminservice
apollo-adminservice]# vi cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: apollo-adminservice-cm
  namespace: infra
data:
  application-github.properties: |
    # DataSource
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigDB?characterEncoding=utf8
    spring.datasource.username = apolloconfig
    spring.datasource.password = 123456
    eureka.service.url = http://config.od.com/eureka
  app.properties: |
    appId=100003172

apollo-adminservice]# vi dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: apollo-adminservice
  namespace: infra
  labels: 
    name: apollo-adminservice
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: apollo-adminservice
  template:
    metadata:
      labels: 
        app: apollo-adminservice 
        name: apollo-adminservice
    spec:
      volumes:
      - name: configmap-volume
        configMap:
          name: apollo-adminservice-cm
      containers:
      - name: apollo-adminservice
        image: harbor.od.com/infra/apollo-adminservice:v1.5.1
        ports:
        - containerPort: 8080
          protocol: TCP
        volumeMounts:
        - name: configmap-volume
          mountPath: /apollo-adminservice/config
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

1583208130534

# 22机器，应用资源配置清单：
~]# kubectl apply -f http://k8s-yaml.od.com/apollo-adminservice/cm.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/apollo-adminservice/dp.yaml

1583217458560

# 11机器，查看数据库连接IP：
~]# mysql -uroot -p
none)]> show processlist;

多了更多的22连接

1583217487662

config.od.com访问地址

鼠标对着起来的Apollo，可以看到右下角有网址

1583217527372

# 22机器，curl：
~]# curl http://172.7.22.7:8080/info

1583217575749

交付Apollo-Portal前，数据库初始化

官网地址

# 200机器，制作镜像-下载包并整理：
cd /opt/src/
~]# wget https://github.com/ctripcorp/apollo/releases/download/v1.5.1/apollo-portal-1.5.1-github.zip
src]# mkdir /data/dockerfile/apollo-portal
src]# unzip -o apollo-portal-1.5.1-github.zip -d /data/dockerfile/apollo-portal
src]# cd /data/dockerfile/apollo-portal
apollo-portal]# rm -f apollo-portal-1.5.1-sources.jar
apollo-portal]# rm -f apollo-portal.conf
apollo-portal]# rm -f scripts/shutdown.sh

sql官网网址-需要raw

# 11机器，数据库初始化：
~]# wget https://raw.githubusercontent.com/ctripcorp/apollo/master/scripts/apollo-on-kubernetes/db/portal-db/apolloportaldb.sql -O apolloportal.sql
~]# ll

1583217807056

# 11机器：
~]# mysql -uroot -p
none)]> source ./apolloportal.sql
none)]> show databases;
none)]> use ApolloPortalDB
ApolloportalDB]> show tables;
ApolloportalDB]> grant INSERT,DELETE,UPDATE,SELECT on ApolloPortalDB.* to "apolloportal"@"10.4.7.%" identified by "123456";
ApolloportalDB]> select user,host from mysql.user;
ApolloportalDB]> select * from ServerConfig\G
ApolloportalDB]> update ServerConfig set Value='[{"orgId":"ben01","orgName":"Linux学院"},{"orgId":"ben02","orgName":"云计算学院"},{"orgId":"ben03","orgName":"Python学院"}]' where Id=2;
ApolloportalDB]> select * from ServerConfig\G

1583217921425

原1583217963849

改后

1583218104712

完成

制作Portal的docker镜像，并交付

# 200机器，更新startup:
cd /data/dockerfile/apollo-portal/scripts/
# 全部删掉换成下面的
scripts]# vi startup.sh
#!/bin/bash
SERVICE_NAME=apollo-portal
## Adjust log dir if necessary
LOG_DIR=/opt/logs/apollo-portal-server
## Adjust server port if necessary
SERVER_PORT=8080
APOLLO_PORTAL_SERVICE_NAME=$(hostname -i)
# SERVER_URL="http://localhost:$SERVER_PORT"
SERVER_URL="http://${APOLLO_PORTAL_SERVICE_NAME}:${SERVER_PORT}"

## Adjust memory settings if necessary
#export JAVA_OPTS="-Xms2560m -Xmx2560m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=1536m -XX:MaxNewSize=1536m -XX:SurvivorRatio=8"

## Only uncomment the following when you are using server jvm
#export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"

########### The following is the same for configservice, adminservice, portal ###########
export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"

# Find Java
if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
    javaexe="$JAVA_HOME/bin/java"
elif type -p java > /dev/null 2>&1; then
    javaexe=$(type -p java)
elif [[ -x "/usr/bin/java" ]];  then
    javaexe="/usr/bin/java"
else
    echo "Unable to find Java"
    exit 1
fi

if [[ "$javaexe" ]]; then
    version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
    version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
    # now version is of format 009003 (9.3.x)
    if [ $version -ge 011000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    elif [ $version -ge 010000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    elif [ $version -ge 009000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    else
        JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
        JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
        JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled  -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
    fi
fi

printf "$(date) ==== Starting ==== \n"

cd `dirname $0`/..
chmod 755 $SERVICE_NAME".jar"
./$SERVICE_NAME".jar" start

rc=$?;

if [[ $rc != 0 ]];
then
    echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
    exit $rc;
fi

tail -f /dev/null

portal_startup官网地址

仅有以下两处不同：

SERVER_PORT=8080 APOLLO_PORTAL_SERVICE_NAME=$(hostname -i)

# 200机器，制作dockerfile：
cd /data/dockerfile/apollo-portal/
apollo-portal]# vi Dockerfile
FROM 909336740/jre8:8u112

ENV VERSION 1.5.1

RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
    echo "Asia/Shanghai" > /etc/timezone

ADD apollo-portal-${VERSION}.jar /apollo-portal/apollo-portal.jar
ADD config/ /apollo-portal/config
ADD scripts/ /apollo-portal/scripts

CMD ["/apollo-portal/scripts/startup.sh"]

apollo-portal]# docker build . -t harbor.od.com/infra/apollo-portal:v1.5.1
apollo-portal]# docker push harbor.od.com/infra/apollo-portal:v1.5.1

1583218938233

1583219229803

# 200机器：
mkdir /data/k8s-yaml/apollo-portal
cd /data/k8s-yaml/apollo-portal
apollo-portal]# vi cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: apollo-portal-cm
  namespace: infra
data:
  application-github.properties: |
    # DataSource
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloPortalDB?characterEncoding=utf8
    spring.datasource.username = apolloportal
    spring.datasource.password = 123456
  app.properties: |
    appId=100003173
  apollo-env.properties: |
    dev.meta=http://config.od.com

apollo-portal]# vi dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: apollo-portal
  namespace: infra
  labels: 
    name: apollo-portal
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: apollo-portal
  template:
    metadata:
      labels: 
        app: apollo-portal 
        name: apollo-portal
    spec:
      volumes:
      - name: configmap-volume
        configMap:
          name: apollo-portal-cm
      containers:
      - name: apollo-portal
        image: harbor.od.com/infra/apollo-portal:v1.5.1
        ports:
        - containerPort: 8080
          protocol: TCP
        volumeMounts:
        - name: configmap-volume
          mountPath: /apollo-portal/config
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

apollo-portal]# vi svc.yaml
kind: Service
apiVersion: v1
metadata: 
  name: apollo-portal
  namespace: infra
spec:
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080
  selector: 
    app: apollo-portal

apollo-portal]# vi ingress.yaml
kind: Ingress
apiVersion: extensions/v1beta1
metadata: 
  name: apollo-portal
  namespace: infra
spec:
  rules:
  - host: portal.od.com
    http:
      paths:
      - path: /
        backend: 
          serviceName: apollo-portal
          servicePort: 8080

1583219397807

# 11机器，解析域名：
~]# vi /var/named/od.com.zone
serial 前滚一位
portal             A    10.4.7.10

~]# systemctl restart named
~]# dig -t A portal.od.com @10.4.7.11 +short
# out: 10.4.7.10

1583219385915

# 22机器，应用资源配置清单：
~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/cm.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/dp.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/svc.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/ingress.yaml

可以查看起来的portal的logs日志，可能稍微有些慢

浏览器输入portal.od.com

Username: apollo
Password: admin

1583221254589

第一件事，修改密码，任何开源软件的第一件事，就是修改默认密码

1584699514174

1583221587411

1583222001345

去看一下对接数据库情况

1583222020754

试试查询，查询key，也试试使用保存更新

key:organizations
value:[{"orgId":"ben01","orgName":"Linux学院"},{"orgId":"ben02","orgName":"云计算学院"},{"orgId":"ben03","orgName":"Python学院"},{"orgId":"ben03","orgName":"大数据学院"}]

1583222152917

# 11机器，查询是否更新：
~]# mysql -uroot -p
none)]> use ApolloPortalDB;
ApolloPortalDB]> select * from ServerConfig\G

1583222202986

成功

使用Apollo，创建项目，提交

1583222246074

# 填入对应参数，并提交：
AppId:dubbo-demo-service

1583222350430

1583222393771

完成

dubbo服务提供者连接Apollo实战

首先我们在git新建一个Apollo分支，以下是操作方法

git创建分支并上传代码（或者直接fork我），注意，下图配的是gitlab里面web的，现在你要操作的是自己git的service，因为service没截图，所以我用web的替代

  1. 切换身份

    # 上传项目的文件夹打开，然后cd到附近的新建的apollo文件夹，克隆项目并切换身份：
$ git http://gitlab.od.com:10000/909336740/dubbo-demo-web.git
$ cd dubbo-demo-web/
$ git branch apollo
$ git checkout apollo

    1583287244483

    将Apollo分支的文件全部拉来这个文件夹，选择替换重名文件

    1583287180864

  2. 把apollo的代码全部拉过去覆盖，并上传代码

    $ git add .
$ git commit -m "apollo commit#1"
$ git push -u origin apollo

    1583287633875

    1582680560428

    完成，当然你可以直接fork我的

    1582680588102

继续主线

在Apollo portal里创建相应两个配置项

1582680643369

新增配置项，并提交

1583223092507

1583223148433

发布

1583223168662

1583223190037

我们已经配置好了，然后需要用Jenkins制作镜像

# 填入对应参数
app_name: dubbo-demo-service
image_name: app/dubbo-demo-service
git_repo: https://gitee.com/benjas/dubbo-demo-service.git
git_ver: apollo
add_tag: 200303_1615
target_dir: ./dubbo-server/target
base_image: base/jre8:8u112

注意，我这里用的是gitlab，因为网络问题，你用自己的公网git即可

1583223413973

去#38的console里查看情况

1583223612917

报错问题：

1581324514061

该报错的原因是mirrors没有改对

# 200机器：
cd /data/nfs-volume/jenkins_home/maven-3.6.1-8u232/conf
conf]# vi settings.xml

修改完上面的内容后保存再去重新构建

继续主线

1583223754285

镜像制作完成

# 200机器，修改指定的镜像，并添加部分内容：
cd /data/k8s-yaml/dubbo-demo-service/
dubbo-demo-service]# vi dp.yaml
        image: harbor.od.com/app/dubbo-demo-service:apollo_200303_1615
        ports:
        - containerPort: 20880
          protocol: TCP
        env:
        - name: JAR_BALL
          value: dubbo-server.jar
        - name: C_OPTS
          value: -Denv=dev -Dapollo.meta=http://config.od.com
        imagePullPolicy: IfNotPresent

1583223916007

# 22机器，应用：
~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-demo-service/dp.yaml
# out: deployment.extensions/dubbo-demo-service configured

1583224029500

去LOGS查看，可以看到连接了Apollo

1583227298634

PS：这里的pod名字你可能觉得不一样，因为我镜像弄错了，重新改了一遍

1583227711909

当你把service扩容成两个后

1583227780562

可以看到变成两个实例

1583227821344

在monitor里可以看到端口是20880（注意必须连接的是zk1，怎么修改上面有，先改cm，然后删掉pod让它自动重启）

1583229093925

1583229141970

我们去Apollo改一下端口，改成20881，并发布

1583229265526

1583229275645

重启两个service的pods（就是删掉让它们自动重启）

1583229312442

刷新dubbo-monitor，可以看到端口已经变成20881

1583229337601

还可以改成zk2

1583229498782

作业：改成zk2后，需要删掉对应的pod，然后让其也在zk2，然后还需要把monitor也改成zk2，方法在上面有。这个作业一定要做，不然下面会报错，具体是哪里报错我会提示

最后我改回来了20880端口，但是依然用的zk2

完成

dubbo服务消费者连接Apollo实战

同样，我是有Apollo分支的，操作方法和上面的service服务一样

1582682538181

创建项目

1583287914567

新增配置，并发布

1583289334584

1583289365597

用Jenkins构建dubbo消费者

# 填入对应参数
app_name: dubbo-demo-consumer
image_name: app/dubbo-demo-consumer
git_repo: http://gitlab.od.com:10000/909336740/dubbo-demo-web.git
git_ver: apollo
add_tag: 200304_1040
target_dir: ./dubbo-client/target
base_image: base/jre8:8u112

注意，我这里用的是gitlab，因为网络问题，你用自己的公网git即可

1583290002445

1583290022279

1583290134412

# 200机器，修改配置：
cd /data/k8s-yaml/dubbo-demo-consumer/
dubbo-demo-consumer]# vi dp.yaml
        image: harbor.od.com/app/dubbo-demo-service:apollo_200304_1040
        ports:
        - containerPort: 8080
          protocol: TCP
        - containerPort: 20880
          protocol: TCP
        env:
        - name: JAR_BALL
          value: dubbo-client.jar
        - name: C_OPTS
          value: -Denv=dev -Dapollo.meta=http://config.od.com
        imagePullPolicy: IfNotPresent

1583290251350

# 应用资源配置清单，22机器：
~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-demo-consumer/dp.yaml
# out: deployment.extensions/dubbo-demo-consumer configured

1583290331910

1583291244653

再去Applications可以看到已经起来了

# 浏览器访问demo.od.com/hello?name=apollo

1583291263893

如果你登录的是这个界面，那么你没有做作业：改service的zk1成zk2，这个consumer已经是zk2的，而service还是zk1，当然不能响应

1583290792208

实现代码迭代

修改代码，然后commit

1582682749719

1582682870930

Jenkins构建

# 填入对应参数
app_name: dubbo-demo-consumer
image_name: app/dubbo-demo-consumer
git_repo: http://gitlab.od.com:10000/909336740/dubbo-demo-web.git
git_ver: apollo
add_tag: 200304_1145
target_dir: ./dubbo-client/target
base_image: base/jre8:8u112

注意，我这里用的是gitlab，因为网络问题，你用自己的公网git即可

1583293563142

1583293832408

# 200机器修改使用的镜像：
cd /data/k8s-yaml/dubbo-demo-consumer/
dubbo-demo-consumer]# vi dp.yaml
image: harbor.od.com/app/dubbo-demo-service:apollo_191208_1640

# 22机器应用：
~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-demo-consumer/dp.yaml
# out: deployment.extensions/dubbo-demo-consumer configured

1583294170202

# 浏览器访问demo.od.com/hello?name=apollo

1583294184192

实战Apollo分环境管理dubbo服务-交付Apollo-configservice

我们要实现测试环境和生产环境只需要打包一个镜像

先开始分环境

# 11机器，解析域名：
~]# vi /var/named/od.com.zone
serial 前滚一位
zk-test            A    10.4.7.11
zk-prod            A    10.4.7.12

~]# systemctl restart named
~]# dig -t A zk-test.od.com +short
# out: 10.4.7.11
~]# dig -t A zk-prod.od.com +short
# out: 10.4.7.12

关闭deployment里的消费者和服务者（改成scale 0，先消费者后服务者）

1583301119606

# 创建两个名称空间， 21机器：
~]# kubectl create ns test
# out:namespace/test created
~]# kubectl create secret docker-registry harbor --docker-server=harbor.od.com --docker-username=admin --docker-password=Harbor12345 -n test
# out:secret/harbor created
~]# kubectl create ns prod
# out:namespace/prod created
~]# kubectl create secret docker-registry harbor --docker-server=harbor.od.com --docker-username=admin --docker-password=Harbor12345 -n prod
# out:secret/harbor created

1583301237979

去看一下dashboard里面的Namespaces

1583301259487

把admin、portal、config都scale成0

1583301320962

# 11机器，建库：
~]# vi apolloconfig.sql
# 增加Test关键字

1583301385286

# 11机器，测试环境：
~]# mysql -uroot -p < apolloconfig.sql
~]# mysql -uroot -p
none)]> show databases;
none)]> use ApolloConfigTestDB;
ApolloConfigTestDB]> select * from ServerConfig\G
ApolloConfigTestDB]> update ApolloConfigTestDB.ServerConfig set ServerConfig.Value="http://config-test.od.com/eureka" where ServerConfig.Key="eureka.service.url";
ApolloConfigTestDB]> select * from ServerConfig\G
ApolloConfigTestDB]> grant INSERT,DELETE,UPDATE,SELECT on ApolloConfigTestDB.* to "apolloconfig"@"10.4.7.%" identified by "123456";

1583301481115

# 11机器，生产环境：
~]# vi apolloconfig.sql
# 修改成Prod关键字
~]# mysql -uroot -p < apolloconfig.sql
~]# mysql -uroot -p
none)]> show databases;
none)]> use ApolloConfigProdDB;
ApolloConfigProdDB]> select * from ServerConfig\G
ApolloConfigProdDB]> update ApolloConfigProdDB.ServerConfig set ServerConfig.Value="http://config-prod.od.com/eureka" where ServerConfig.Key="eureka.service.url";
ApolloConfigProdDB]> select * from ServerConfig\G
ApolloConfigProdDB]> grant INSERT,DELETE,UPDATE,SELECT on ApolloConfigProdDB.* to "apolloconfig"@"10.4.7.%" identified by "123456";

1583301626767

# 11机器，修改支持类型：
none)]> use ApolloPortalDB;
ApolloPortalDB]> show tables;
ApolloPortalDB]> select * from ServerConfig\G
ApolloPortalDB]> update ServerConfig set Value='fat,pro' where Id=1;
ApolloPortalDB]> select * from Serverconfig\G

改后

1583302030474

PS按理来说你应该只有dev，但是好像Apollo更新了，一开始就有4种支持的类型，只要你确保有fat和pro即可

# 200机器：
cd /data/k8s-yaml/apollo-portal/
# 需改以下内容
apollo-portal]# vi cm.yaml
  apollo-env.properties: |
    fat.meta=http://config-test.od.com
    pro.meta=http://config-prod.od.com

1583302159311

# 22机器，应用：
~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/cm.yaml
# out: configmap/apollo-portal configured

1583302232037

完成

实战使用Apollo分环境管理dubbo服务——交付Apollo-portal和adminservice

# 制作资源配置清单,200机器：
cd /data/k8s-yaml/
k8s-yaml]# mkdir -pv test/{apollo-configservice,apollo-adminservice,dubbo-demo-service,dubbo-demo-consumer}
k8s-yaml]# mkdir -pv prod/{apollo-configservice,apollo-adminservice,dubbo-demo-service,dubbo-demo-consumer}
k8s-yaml]# cd test/apollo-configservice/
apollo-configservice]# cp -a /data/k8s-yaml/apollo-configservice/cm.yaml .
apollo-configservice]# cp -a /data/k8s-yaml/apollo-configservice/svc.yaml .
apollo-configservice]# cp -a /data/k8s-yaml/apollo-configservice/dp.yaml .
apollo-configservice]# cp -a /data/k8s-yaml/apollo-configservice/ingress.yaml .
# 修改成test，三处
apollo-configservice]# vi cm.yaml
  namespace: test
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigTestDB?characterEncoding=utf8
    spring.service.url = http://config-test.od.com/eureka

# 修改成test，一处
apollo-configservice]# vi dp.yaml
  namespace: test

# 修改成test，一处
apollo-configservice]# vi svc.yaml
  namespace: test

# 修改成test，两处
apollo-configservice]# vi ingress.yaml
  namespace: test
  - host: config-test.od.com

# 11机器，解析域名：
~]# vi /var/named/od.com.zone
serial 前滚一位
config-test        A    10.4.7.10
config-prod        A    10.4.7.10

~]# systemctl restart named

1582683284142

# 应用资源配置清单，22机器：
~]# kubectl apply -f http://k8s-yaml.od.com/test/apollo-configservice/cm.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/test/apollo-configservice/dp.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/test/apollo-configservice/svc.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/test/apollo-configservice/ingress.yaml

# 200机器，制作prod的：
cd /data/k8s-yaml/prod/apollo-configservice/
apollo-configservice]# cp ../../test/apollo-configservice/*.yaml .
# 修改成prod，三处
apollo-configservice]# vi cm.yaml
  namespace: prod
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigProdDB?characterEncoding=utf8
    spring.service.url = http://config-prod.od.com/eureka

# 修改成prod，一处
apollo-configservice]# vi dp.yaml
  namespace: prod

# 修改成prod，一处
apollo-configservice]# vi svc.yaml
  namespace: prod

# 修改成prod，两处
apollo-configservice]# vi ingress.yaml
  namespace: prod
  - host: config-prod.od.com

# 应用资源清单，22机器：
~]# kubectl apply -f http://k8s-yaml.od.com/prod/apollo-configservice/cm.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/prod/apollo-configservice/dp.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/prod/apollo-configservice/svc.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/prod/apollo-configservice/ingress.yaml

1581347383718

确认你的nslookup.exe 有没有（一般是有的）

1581347525407

# 访问config-test.od.com
config.od.com已经没了

1583304944456

# 访问config-prod.od.com
prod可能还没好，接着往下面做，test能起来，prod肯定也能起来

1583304959338

可以看到test和prod在两个不同的容器里，这就是我们模拟的分环境

# 200机器：
cd /data/k8s-yaml/test/apollo-adminservice/
apollo-adminservice]# cp -a /data/k8s-yaml/apollo-adminservice/*.yaml .
# 修改成test，三处
apollo-adminservice]# vi cm.yaml
  namespace: test
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigTestDB?
    spring.service.url = http://config-test.od.com/eureka

# 修改成test，一处
apollo-adminservice]# vi dp.yaml
  namespace: test

apollo-adminservice]# cd ../../prod/apollo-adminservice
apollo-adminservice]# cp -a ../../test/apollo-adminservice/*.yaml .
# 修改成prod，三处
apollo-adminservice]# vi cm.yaml
  namespace: prod
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigProdDB?characterEncoding=utf8
    spring.service.url = http://config-prod.od.com/eureka

# 修改成prod，一处
apollo-adminservice]# vi dp.yaml
  namespace: prod

# 应用，22机器：
~]# kubectl apply -f http://k8s-yaml.od.com/test/apollo-adminservice/cm.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/test/apollo-adminservice/dp.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/prod/apollo-adminservice/cm.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/prod/apollo-adminservice/dp.yaml

1583304745055

1583304817198

# 11机器，清除数据：
mysql -uroot -p
none)]> use ApolloPortalDB;
ApolloProtalDB]> select * from App;
ApolloProtalDB]> select * from AppNamespace;
ApolloProtalDB]> truncate table App;
ApolloProtalDB]> truncate table AppNamespace;

1583305105464

把portal scale成1

1583305172038

使用查询功能查询下

1583305255615

1583305298870

成功

实战发布dubbo连接Apollo到不同环境

创建项目

portal.od.com

1583305340551

1583305381813

现在也可以看到环境列表有两个了

1583305431837

测试环境新增配置项，并发布

1583305535156

1583305580379

1583305601716

生产环境新增配置项，并发布

1583305663692

1583312777104

1583312803655

service配置好了，再配置consumer

1583305834462

测试环境新增配置项，并发布

1583305939762

1583305965712

生产环境新增配置项，并发布

1583306013898

1583306034551

1583306049712

# 200机器，制作测试环境service：
cd /data/k8s-yaml/test/dubbo-demo-service/
dubbo-demo-service]# cp -a /data/k8s-yaml/dubbo-demo-service/*.yaml .
# 修改三处
dubbo-demo-service]# vi dp.yaml
  namespace: test
        env:
        - name: JAR_BALL
          value: dubbo-server.jar
        - name: C_OPTS
          value: -Denv=fat -Dapollo.meta=http://config-test.od.com

# 应用，22机器：
~]# kubectl apply -f http://k8s-yaml.od.com/test/dubbo-demo-service/dp.yaml

1583306650482

# 200机器，制作测试环境consumer：
cd /data/k8s-yaml/test/dubbo-demo-consumer/
dubbo-demo-consumer]# cp -a /data/k8s-yaml/dubbo-demo-consumer/*.yaml .
# 修改三处
dubbo-demo-consumer]# vi dp.yaml
  namespace: test
        - name: JAR_BALL
          value: dubbo-client.jar
        - name: C_OPTS
          value: -Denv=fat -Dapollo.meta=http://config-test.od.com

# 修改一处
dubbo-demo-consumer]# vi svc.yaml
  namespace: test

# 修改一处
dubbo-demo-consumer]# vi ingress.yaml
  namespace: test
  - host: demo-test.od.com

# 11机器，新增解析：
~]# vi /var/named/od.com.zone
serial 前滚一位
demo-test          A    10.4.7.10

~]# systemctl restart named

1583306813921

# 应用，22机器：
~]# kubectl apply -f http://k8s-yaml.od.com/test/dubbo-demo-consumer/dp.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/test/dubbo-demo-consumer/svc.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/test/dubbo-demo-consumer/ingress.yaml

查看dashboard的启动情况

1583307076131

此时的monitor还是zk2，我们改成zk-test

1583307422408

1583307483922

update然后删掉对应的pod让它自动重启

1583307517718

1583307567245

1583307586774

# 浏览器输入：demo-test.od.com/hello?name=test

1583310055796

完成

开始做生产环境

# 11机器，新增解析：
~]# vi /var/named/od.com.zone
serial 前滚一位
demo-prod          A    10.4.7.10

~]# systemctl restart named

1581350791800

# 200机器，制作生产环境service：
cd /data/k8s-yaml/prod/dubbo-demo-service/
dubbo-demo-service]# cp -a ../../test/dubbo-demo-service/*.yaml .
# 共修改三处
dubbo-demo-service]# vi dp.yaml
  namespace: prod
        env:
        - name: JAR_BALL
          value: dubbo-server.jar
        - name: C_OPTS
          value: -Denv=pro -Dapollo.meta=http://apollo-configservice:8080

# 应用，22机器：
~]# kubectl apply -f http://k8s-yaml.od.com/prod/dubbo-demo-service/dp.yaml

1583310378331

你还可以去LOGS日志里面看看

# 200机器，制作生产环境consumer：
cd /data/k8s-yaml/prod/dubbo-demo-consumer/
dubbo-demo-consumer]# cp -a /data/k8s-yaml/dubbo-demo-consumer/*.yaml .
# 共修改三处
dubbo-demo-consumer]# vi dp.yaml
  namespace: prod
        env:
        - name: JAR_BALL
          value: dubbo-client.jar
        - name: C_OPTS
          value: -Denv=pro -Dapollo.meta=http://apollo-configservice:8080
        imagePullPolicy: IfNotPresent

# 共修改一处
dubbo-demo-consumer]# vi svc.yaml
  namespace: prod

# 共修改两处
dubbo-demo-consumer]# vi ingress.yaml
  namespace: prod
spec:
  rules:
  - host: demo-prod.od.com

# 应用，22机器：
~]# kubectl apply -f http://k8s-yaml.od.com/prod/dubbo-demo-consumer/dp.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/prod/dubbo-demo-consumer/svc.yaml
~]# kubectl apply -f http://k8s-yaml.od.com/prod/dubbo-demo-consumer/ingress.yaml

# 浏览器输入：demo-prod.od.com/hello?name=prod

1583313070558

完成

报错：1583312899877

我去LOGS里面看了一下，发现我Apollo配得port写错了，写成了dubbo-port，应该是dubbo.port，改回来就可以了，删掉service得pod重新启动，删掉consumer的pod重新启动

实战演示项目提测，发版流程

模拟项目提测到发布上线，这里用得gitlab，你有可以了解一下gitlab是长什么样子的，还有怎么操作的

修改源代码，并commit

1583313961163

1583313975564

gitlab跟gittee在更新代码的编号有所区别

1583314181056

Jenkins构建

# 填入对应参数，然后build
app_name: dubbo-demo-consumer
image_name: app/dubbo-demo-consumer
git_repo: http://gitlab.od.com:10000/909336740/dubbo-demo-web.git
git_ver: 535826b1239fedba0df3799b7b3b8585d56e9e18
add_tag: 200304_1730
target_dir: ./dubbo-client/target
base_image: base/jre8:8u112

1583314459402

1583314584034

先到测试环境发布，修改tag

1583314731675

1583316573117

重启成功，刷新测试环境的页面，查看情况

1583314816036

刷新生产环境的页面，查看情况，还没改变

1583314831690

我们来修改prod的，这时候已经不需要再去Jenkins打包镜像了

1583316530331

1583316573117

更新完启动pod后

再来刷新生产环境的页面

1583316620715

已经改变，成功