Managing Kafka services for Shift

KOBIL Shift can make use of the Strimzi-Kafka Operator to create the required Kafka-Services (running Kafka-cluster setup with Zookeeper Management and Control logic). See configuration in KOBIL Shift metaconfig - which is by default enabled.

# -- Configuration of the Kafka custom resources. Requires [Strimzi Kafka operator]([https://strimzi.io/)](https://strimzi.io/))`
strimzi:
    # -- Enable/disable deployment of Kafka custom resources [true|false]`
    enabled: true

Configure for external Kafka services:

Find README details here:  External Kafka Services integration to KOBIL Shift   When using external Kafka service it is required to disable the creation of the custom resources for Strimzi Kafka operator by KOBIL Shift deployment - see parameter strimzi.enabled:false 

Configure KOBIL Shift metaconfiguration - external kafka

In case of running multiple KOBIL Shift deployments accessing one centralized Kafkaservice - then the kafkatopic names need to customized to ensure being unique per deployment using configuration parameter "topics.prefix"

external kafka configuration

datastores:
..
    kafka:
        auth:
            # -- Enable authentication
            enabled: true
            username: external.kafka.user
        external:
            enabled: true
            # -- The broker/bootstrap hostname and port of the external Kafka cluster.
            broker:
                host: <external-kafka.bootstrap.domain>
                port: 9092
            topics:
                prefix: "shift"
            # -- Optional TLS configuration for connection to external cluster
            tls:
                enabled: true
                trustStoreSecret: shift-kafka-tls-truststore

You may contact KOBIL ProjectTeam for assistance if required. The default design of KOBIL Shift is to exclusively use the KafkaTopics with pre-defined names within the used namespace.

Configure KOBIL Shift metaconfiguration - required "istio" annotation configuration

To allow outbound communication for the KOBIL Shift services the servicemesh (istio) configuration required specific setup to allow kafka egress on i.e. default port 9092 session establishment. For this update the metaconfig as appropriate:

istio annotation for kafka outbound sessions

..
    istio:
        enabled: true
        resourceAnnotations: |
            proxy.istio.io/config: |
                holdApplicationUntilProxyStarts: true
            # 1521 == oracle, 9092 == kafka, 6379 == redis
            traffic.sidecar.istio.io/excludeOutboundPorts: "1521,6379,9092"

Configure namespace/cluster network policy resources

Configure namespace/cluster network policy resources KOBIL Shift metaconfiguration - required "istio" annotation configuration

When enabled outbound communication for the KOBIL Shift services from servicemesh (istio) then also at namespace/cluster level the network policy is required to allow outbound session establishment. 

openshift network policy sample

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
    name: egress-external-kafka
    namespace: ${NAMESPACE}
spec:
    podSelector: {}
    egress:
    - to:
        - ipBlock:
            cidr: 10.10.196.0/24  ### this segment 10.10.196.0/24 covering a kafka cluster servers / bootstrap / broker
    policyTypes:
    - Egress