Logging

The Spark operator installs a vector agent as a side-car container in every application Pod except the job Pod that runs spark-submit. It also configures the logging framework to output logs in XML format. This is the same format used across all Stackable products, and it enables the vector aggregator to collect logs across the entire platform.

It is the user’s responsibility to install and configure the vector aggregator, but the agents can discover the aggregator automatically using a discovery ConfigMap as described in the logging concepts.

Only logs produced by the application’s driver and executors are collected. Logs produced by spark-submit are discarded.

The logs of the containerdebug process are collected as well, from the driver and executor Pods and from the history server. They are labelled container: containerdebug, although no such Kubernetes container exists. The process runs in the background of the spark and spark-history containers. It also runs in the spark-submit Pod, whose logs are discarded as described above. The label comes from /stackable/log/containerdebug, the directory which the operator sets with the FILE_LOG_DIRECTORY environment variable and from which the Vector agent derives the container name. Overriding FILE_LOG_DIRECTORY with envOverrides therefore changes how these logs are aggregated. The agent only collects files one level below /stackable/log, so a sibling directory such as /stackable/log/spark relabels the events instead of dropping them, while any deeper or shallower path stops the aggregation. A path outside /stackable/log also leaves the size-capped log volume.

History server

The following snippet shows how to configure log aggregation for the history server:

apiVersion: spark.stackable.tech/v1alpha1
kind: SparkHistoryServer
metadata:
  name: spark-history
spec:
  vectorAggregatorConfigMapName: spark-vector-aggregator-discovery (1)
  nodes:
    roleGroups:
      default:
        config:
          logging:
            enableVectorAgent: true (2)
            containers:
              spark-history: (3)
                console:
                  level: INFO
                file:
                  level: INFO
                loggers:
                  ROOT:
                    level: INFO
...

1	Name of a ConfigMap that referenced the vector aggregator. See example below.
2	Enable the vector agent in the history pod.
3	Configure log levels for file and console outputs.

Example vector aggregator configuration

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: spark-vector-aggregator-discovery
data:
  ADDRESS: http://spark-vector-aggregator:6123