What's on this page

☰
Navigation :

Log Management

Logging and Log Management

Telco Network Cloud Orchestration (TNC-O) generates logs across all of its microservices. These logs are generated locally and shipped to Elasticsearch where they are persisted and can be retrieved/searched

Log tracing across TNC-O can be a challenge if not done correctly:

This creates multiple possible paths an action request can take for the total set of internal API calls required to service that request. This in turn means that any associated logs are spread across multiple log files. By taking these and shipping them to Elasticsearch a single searchable chronologically ordered log stream is created. This eliminates a considerable challenge when viewing logs.

However in a busy system there remains an issue of being able to isolate logs for a specific request.

To eliminate this and make it easier to follow the set of communication requests and the resulting logs for a specific request, TNC-O provides supports Tracing Contexts and primarily a Transaction Id. At the point of ingress to TNC-O a Tracing Context (traceCtx) can be provided. In API calls this is achieved through the addition of a X-TraceCtx-TransactionId header. If one is not provided then the point of ingress (typically the API gateway) will generate one so that it is available for the remainder of the of the transaction

This Tracing Context is passed between services on each interaction. By providing this as in indexed field in any resulting logs it is possible trace the end to end set of logs for a given transaction and to present them chronologically.

Logs generated by the core TNC-O services and optionally any service feeding into it or which it interacts with (such as the RM) are supplied with the set of X-TracingCtx headers so true end to end log isolation can be achieved in highly concurrent system even when under load This can be used in conjunction with an intent (process) Id where applicable

View Log Files

TNC-O Logging is at standard levels:

Each service instance can have its own log level set individually. This is achieved in one of two ways and details of how to do so are discussed below. The default level, if not overridden, is INFO

All TNC-O Logs are generated locally to the docker logs and are accessible on the individual pod via kubectl logs on the Kubernetes cluster

Core TNC-O service generated logs are in JSON format to facilitate ingestion to Elasticsearch

Logs are pulled by Filebeat from the individual log streams on the service instances (pods) and inserted into appropriate indices in Elasticsearch. This creates a single chronologically ordered* set of log records ( indices )

Kibana is the preferred tool for examining logs and it is recommended that it be used whenever viewing logs, especially in a production system

Setting log levels

As referenced earlier, the default logging level for TNC-O defaults to INFO, the lowest level of logging generating the least output. There are two ways in which the logging level can be altered

Setting default logging

Default Logging Level is set on a per TNC-O microservice level by configuring the preferred level in the service configuration (secret) in Vault. The Default Logging Level for the service is persisted such that on startup, the individual TNC-O service instances read this value and set the log level accordingly.

logging.level.com.accantosystems: DEBUG

To set the Default Logging Level for a service the user should

  1. log on to Vault and navigate to the appropriate service secret (if one does not exist for the target service then it should be created).
  2. Add the logging level statement to the config at the top level. In the following example this is as a peer to "alm"
{
  "alm": {
    "apollo": {
      "cassandra": {
        "keyspaceManager": {
          "replicationFactor": 1
        }
      },
      "janus": {
        "cluster.max-partitions": 4,
        "index": {
          "search": {
            "elasticsearch.create.ext.index.number_of_replicas": 0,
            "elasticsearch.create.ext.index.number_of_shards": 1
          }
        },
        "storage.cql.replication-factor": 1
      }
    }
  },
  "logging.level.com.accantosystems": "DEBUG"
}
  1. If being set on a running system, all the impacted service instances (pods) will need to be bounced in order to retrieve the new setting.

Altering the log level at run-time

Alternatively the level on service instances can be set individually at runtime without requiring a restart of the service instance.

Multiple service instances per service will exist in a HA deployment. The means that the level will effectively need to be set on all service instances for target service so that desired logs are generated.

This is achieved by calling an internal API for a running pod. It is post simply POST to the management endpoint on each service. For example to set the logs to debug for all TNC-O logging in Daytona instance, you could exec onto the pod and do this:

curl -i -X POST -H 'Content-Type: application/json' -d '{"configuredLevel": "DEBUG"}' http://localhost:<8281>/management/loggers/com.accantosystems

in general the form is;

curl -i -X POST -H 'Content-Type: application/json' -d '{"configuredLevel": "\<level\>"}' http://\<IP Address | localhost\>:\<port\>/management/loggers/com.accantosystems

Additional authentication header needs to be added when setting the log level for the Ishtar service. please see here for details

Log level

One of the TNC-O supported Log Levels (INFO being the most terse and DEBUG the most verbose):

IP Address

The internal, cluster IP address of the pod being targeted. Alternatively if the call is being made from the pod itself then ‘localhost’ should be used

Port

The port for the service is deployed on for that Pod. If not know, this can be identified using kubectl:

kubectl get svc

This will return the TCP port for the target TNC-O service.

Authentication headers for Ishtar

In a secure environment the log level for Ishtar gateway can only be set via an authenticated API call. This can be performed in either the internal or external API endpoint (this is unique to Ishtar)

To do so, the user must first obtain an access token.

Obtaining an authentication token

An accessToken can be obtained using the published TNC-O API endpoint using appropriate user/client credentials.

Once obtained, the accessToken should be used in the API request as follows when using curl, replacing \<level\> with the desired Log Level

curl -i -X POST -H 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR............Nh5QJEdZSyf6YxOOeJ9FRlUWcBQJPENd-xTDEDs' -H 'Content-Type: application/json' -d '{"configuredLevel": "\<level\>"}' http://\<IP Address | localhost\>:\<port\>/management/loggers/com.accantosystems

Log Management

TNC-O, and the associated Resource Manager(s), Resource drivers can collectively generate a large number of logs which can have an impact on load of both the system itself but also the downstream services such as Elasticsearch. Thus it is suggested that this is either allowed for in the engineering of the TNC-O deployment of the system be kept at a low level of logging (INFO). When required the level of the logging can be ‘raised’. This of course needs to be done prior to any event which needs to be captured and is a judgment call.

Elastic Search can be configured to remove historical logs based on their Elasticsearch index.

Docker Logs

Log management on individual containers should be performed in accordance with individual deployment policy. For details on how to do so the it is recommended that the user consult docker documentation. Though not typically an issue the configuration for docker must be compatible with the polling rate of Filebeat so that the docker log policy is not deleting logs quicker than Filebeat can retrieve them.

Elasticsearch Indices

It is recommend that in creation of index patterns and the design of Kibana dashboards, that the appropriate index pattens be created such that they it is possible for the user select Index Patterns which match the real scope of applicable logs (records). This helps in the performance of the system, managing disk usage and in ensuring that all relevant logs are in scope.

Additionally, it should be noted that Opensource Elasticsearch does not have an inbuilt facility for deleting logs (records/indices). However a separate Opensource tool, Elasticsearch-Curator, exists which can be used to perform management of logs (indices). This can be leveraged to manage (delete) older logs from elastic search. Documentation for how to use this tool is accessible here. It is possible to design curator actions with appropriate filters to meet the specific needs of your organization.