Enterprise Logging

The logging stack consists of AWS Open Search , OpenSearch Dashboards, and Fluent Bit.

Installation

The installation was directly copied from AWS EKS Workshop with one exception. The first item is to create an OIDC identity provider. This had already been done when installing the AWS load balancer.

Components

  1. Fluent Bit: an open source and multi-platform Log Processor and Forwarder which allows you to collect data/logs from different sources, unify and send them to multiple destinations. It’s fully compatible with Docker and Kubernetes environments.

  2. Amazon OpenSearch Service: OpenSearch is an open source, distributed search and analytics suite derived from Elasticsearch. Amazon OpenSearch Service offers the latest versions of OpenSearch, support for 19 versions of Elasticsearch (1.5 to 7.10 versions), and visualization capabilities powered by OpenSearch Dashboards and Kibana (1.5 to 7.10 versions).

  3. OpenSearch Dashboards: OpenSearch Dashboards, the successor to Kibana, is an open-source visualization tool designed to work with OpenSearch. Amazon OpenSearch Service provides an installation of OpenSearch Dashboards with every OpenSearch Service domain.

CU Boulder Initial Domain

OpenSearch Dashboards URL

Username and Password in Keepass

Domain endpoint

Issues

  1. Multiline logs are split into single line logs. Therefore, we need to configure the parser for multiline logs within Fluent Bit.