kminion 설치 (docker)

0. 테스트 환경

• OS : ubuntu 22.04
• kminion
  • hostname : kminion
  • ip : 192.168.137.31
• kafka
  • hostname : kafka
  • ip : 192.168.137.32

1. kafka 설치

1.1 jdk 설치

sudo apt-get update
sudo apt install default-jdk 

1.2 kafka 설치

wget https://dlcdn.apache.org/kafka/3.5.0/kafka_2.13-3.5.0.tgz 
tar xvzf kafka_2.13-3.5.0.tgz 
sudo mv kafka_2.13-3.5.0 /usr/local/kafka

1.3 systemctl 에 zookeeper, kafka 등록

echo "[Unit]
Description=Apache Zookeeper server
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
ExecStart=/usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties
ExecStop=/usr/local/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target" | sudo tee /etc/systemd/system/zookeeper.service 

echo "[Unit]
Description=Apache Kafka Server
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service

[Service]
Type=simple
Environment="JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64"
ExecStart=/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties
ExecStop=/usr/local/kafka/bin/kafka-server-stop.sh

[Install]
WantedBy=multi-user.target" | sudo tee /etc/systemd/system/kafka.service 

sudo systemctl daemon-reload 
sudo systemctl start zookeeper 
sudo systemctl start kafka 
sudo systemctl status zookeeper

1.4. 테스트용 토픽 생성

/usr/local/kafka/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic testTopic 
/usr/local/kafka/bin/kafka-topics.sh --list --bootstrap-server localhost:9092 

2. kminion 설치

2-1. apt 로 docker 를 설치하기 위한 필요 패키지 설치

sudo apt-get update
sudo apt-get install ca-certificates curl gnupg

2-2. Docker 공식 GPC 키 추가

sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

2-3. 저장소 설정

echo   "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" |   sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

2-4. Docker 엔진 설치

sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

2-5. docker-compose 설치

sudo apt-get install docker-compose

2-6. kminion 설정

• docker-compose.yml 파일 생성시 extra_hosts 으로 kafka 의 호스트 정보를 지정해준다.
• kminion 에서 브로커에 접속할때 호스트명으로 접속하는 경우가 있는데 이때 호스트명으로 서버 정보 확인이 안되면 오류가 발생한다.

mkdir kminion
cd kminion/

# kminion.yml 파일 생성 
echo "logger:
  level: info

kafka:
  brokers: [\"192.168.137.32:9092\"]" | tee kminion.yml

# docker-compose.yml 파일 생성 
echo "version: '3'

services:
  kminion:
    container_name: kminion
    image: vectorized/kminion:latest
    restart: unless-stopped
    environment:
      CONFIG_FILEPATH: /app/kminion.yml
    ports:
      - 8080:8080
    volumes:
      - ./kminion.yml:/app/kminion.yml
    extra_hosts:
      - 'kafka:192.168.137.32'" | tee docker-compose.yml

2-7. kminion 구동

sudo docker-compose up

2-8. metrics 확인

grafana 에 연동하는건 다음에.. ^^