Apache Iceberg

Apache Iceberg is a format for huge analytic tables designed to address some of the scaling issues with traditional Hive tables. Iceberg stores table metadata in a metadata catalog, which is configured in spec.connector.iceberg.catalog. The following catalogs are supported:

rest: An Iceberg REST catalog
hiveMetastore: A Hive metastore, using the same metastore ConfigMap as the Apache Hive connector
userProvided: The operator doesn’t configure a catalog, you need to configure it yourself using configOverrides

Example Iceberg REST catalog configuration

apiVersion: trino.stackable.tech/v1alpha2
kind: TrinoCatalog
metadata:
  name: iceberg  (1)
  labels:
    trino: simple-trino
spec:
  connector:
    iceberg:
      catalog:
        rest:
          hostname: iceberg-rest-catalog.example.com  (2)
          port: 9001  (3)
          path: /catalog  (4)
          tls:  (5)
            verification:
              server:
                caCert:
                  secretClass: tls
          authentication:  (6)
            oAuth2:
              server:  (7)
                hostname: idp.example.com
                path: /oauth2/token
                tls:
                  verification:
                    server:
                      caCert:
                        webPki: {}
              clientCredentials:  (8)
                secretName: trino-iceberg-client  (9)
                scopes:  (10)
                  - lakehouse
      s3:
        inline:
          host: test-minio
          port: 9000
          accessStyle: Path
          credentials:
            secretClass: minio-credentials
---
apiVersion: v1
kind: Secret
metadata:
  name: trino-iceberg-client
stringData:
  clientId: trino
  clientSecret: my-client-secret

1	The name of the catalog as it will appear in Trino
2	The hostname of the REST catalog server
3	Optional, defaults to 443 if TLS is used and to 80 otherwise
4	Optional, defaults to `/`
5	Optional, uses the same TLS configuration as S3 connections. If not specified, Trino connects over plaintext HTTP. Disabling the TLS verification (`verification.none`) is not supported.
6	Optional, defaults to no authentication
7	The token endpoint of the OAuth2 server, see OAuth2 server. It takes the same `hostname`, `port`, `path` and `tls` fields as the REST catalog, so it connects over plaintext HTTP unless `tls` is set. TLS is strongly recommended here, as client credentials and tokens are sent to this endpoint.
8	The client credentials are exchanged for a token using the OAuth2 client credentials flow. Alternatively, use `token.secretName` to pass a Secret containing a static bearer token in the `token` key.
9	Name of a Secret containing the `clientId` and `clientSecret` keys
10	Optional, the scopes to request from the OAuth2 server. If not specified, the `catalog` scope is requested. Specifying scopes replaces this default.

The operator only supports a subset of the REST catalog properties. Use configOverrides to set additional ones.

OAuth2 server

oAuth2.server points to the token endpoint of the OAuth2 server. What it is used for depends on the credential:

clientCredentials: Trino fetches its token from this endpoint on startup and refreshes it there before it expires.
token: Trino uses the token as is. If the token is a JWT with an exp claim, Trino tries to refresh it before it expires by exchanging it at this endpoint (OAuth2 token exchange). If the refresh fails, Trino logs a warning and keeps using the current token. A token that isn’t a JWT is never refreshed, so the endpoint is never called.

The OAuth2 server is required even for tokens that are never refreshed. Without it, the Iceberg client used by Trino would fall back to the deprecated /v1/oauth/tokens endpoint of the REST catalog, which is planned to be removed (see apache/iceberg#10537).

Example Iceberg Hive metastore configuration

You can deploy a Hive Metastore Stacklet with the Stackable Operator for Apache Hive.

apiVersion: trino.stackable.tech/v1alpha2
kind: TrinoCatalog
metadata:
  name: iceberg  (1)
  labels:
    trino: simple-trino
spec:
  connector:
    # Specify iceberg here when defining a iceberg catalog
    iceberg:
      catalog:
        hiveMetastore:
          configMap: simple-hive  (2)
      s3:
        inline:
          host: test-minio
          port: 9000
          accessStyle: Path
          credentials:
            secretClass: minio-credentials
          tls: (3)
            verification:
              server:
                caCert:
                  secretClass: minio-tls-certificates

1	The name of the catalog as it will appear in Trino
2	This is the name of your Hive Stacklet
3	TLS is recommended for S3 connections, see Connecting Trino to S3

Connect to S3 store or HDFS

The Iceberg connector can connect to S3 or HDFS in the same way the Apache Hive connector does. Check that documentation on how to configure the access.