How to connect
Contact data.delivery@spglobal.com to get:
- your OAuth2 client id and client secret
- your catalog's warehouse
| Catalog URI | https://iceberg.spglobal.com |
| Token | <token>
|
| Token endpoint | https://secureapi.signin.spglobal.com/gettoken/token |
| Scope | api |
| Warehouse | your catalog's warehouse |
Examples
Amazon EMR
Prerequisites:
- An EMR cluster on
emr-spark-8.1.0or later: Spark 4.1, with EMR's Iceberg, 1.11, enabled by theiceberg-defaultsclassification - Library:
org.apache.iceberg:iceberg-gcp-bundleof EMR's Iceberg version,1.11.0on 8.1, for Cloud Storage - Network access to:
https://iceberg.spglobal.com, the cataloghttps://secureapi.signin.spglobal.com/gettoken/token, the token endpointhttps://storage.googleapis.com, the data files
- The client's secret in Secrets Manager, set by the job, rather than in the cluster's configuration
The cluster's configuration, configurations.json:
[
{
"Classification": "iceberg-defaults",
"Properties": {"iceberg.enabled": "true"}
},
{
"Classification": "spark-defaults",
"Properties": {
"spark.jars.packages": "org.apache.iceberg:iceberg-gcp-bundle:1.11.0",
"spark.sql.catalog.<warehouse>": "org.apache.iceberg.spark.SparkCatalog",
"spark.sql.catalog.<warehouse>.type": "rest",
"spark.sql.catalog.<warehouse>.uri": "https://iceberg.spglobal.com",
"spark.sql.catalog.<warehouse>.warehouse": "<warehouse>",
"spark.sql.catalog.<warehouse>.rest.auth.type": "oauth2",
"spark.sql.catalog.<warehouse>.oauth2-server-uri": "https://secureapi.signin.spglobal.com/gettoken/token",
"spark.sql.catalog.<warehouse>.scope": "api",
"spark.sql.catalog.<warehouse>.token-refresh-enabled": "false",
"spark.sql.catalog.<warehouse>.header.X-Iceberg-Access-Delegation": "vended-credentials",
"spark.sql.catalog.<warehouse>.io-impl": "org.apache.iceberg.gcp.gcs.GCSFileIO"
}
}
]
aws emr create-cluster --release-label emr-spark-8.1.0 --applications Name=Spark \ --configurations file://configurations.json ...
Or with Terraform:
resource "aws_emr_cluster" "iceberg" {
name = "<cluster>"
release_label = "emr-spark-8.1.0"
applications = ["Spark"]
service_role = "<service role>"
# ec2_attributes, master_instance_group and core_instance_group as for any cluster
configurations_json = jsonencode([
{
Classification = "iceberg-defaults"
Properties = { "iceberg.enabled" = "true" }
},
{
Classification = "spark-defaults"
Properties = {
"spark.jars.packages" = "org.apache.iceberg:iceberg-gcp-bundle:1.11.0"
"spark.sql.catalog.<warehouse>" = "org.apache.iceberg.spark.SparkCatalog"
"spark.sql.catalog.<warehouse>.type" = "rest"
"spark.sql.catalog.<warehouse>.uri" = "https://iceberg.spglobal.com"
"spark.sql.catalog.<warehouse>.warehouse" = "<warehouse>"
"spark.sql.catalog.<warehouse>.rest.auth.type" = "oauth2"
"spark.sql.catalog.<warehouse>.oauth2-server-uri" = "https://secureapi.signin.spglobal.com/gettoken/token"
"spark.sql.catalog.<warehouse>.scope" = "api"
"spark.sql.catalog.<warehouse>.token-refresh-enabled" = "false"
"spark.sql.catalog.<warehouse>.header.X-Iceberg-Access-Delegation" = "vended-credentials"
"spark.sql.catalog.<warehouse>.io-impl" = "org.apache.iceberg.gcp.gcs.GCSFileIO"
}
},
])
}
In the job, before the first query:
spark.conf.set("spark.sql.catalog.<warehouse>.credential", "<client id>:<client secret>")
spark.conf.set("spark.sql.catalog.<warehouse>.token", "<token>")
Documentation:
Apache Spark
Prerequisites:
- Spark 4.0 or later
- Libraries:
org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.12.0andorg.apache.iceberg:iceberg-gcp-bundle:1.12.0. in Gradle:dependencies { runtimeOnly("org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.12.0") runtimeOnly("org.apache.iceberg:iceberg-gcp-bundle:1.12.0") } - Network access to:
https://iceberg.spglobal.com, the cataloghttps://secureapi.signin.spglobal.com/gettoken/token, the token endpointhttps://storage.googleapis.com, the data files
Session configuration:
spark.sql.catalog.<warehouse>=org.apache.iceberg.spark.SparkCatalog spark.sql.catalog.<warehouse>.type=rest spark.sql.catalog.<warehouse>.uri=https://iceberg.spglobal.com spark.sql.catalog.<warehouse>.warehouse=<warehouse> spark.sql.catalog.<warehouse>.rest.auth.type=oauth2 spark.sql.catalog.<warehouse>.oauth2-server-uri=https://secureapi.signin.spglobal.com/gettoken/token spark.sql.catalog.<warehouse>.credential=<client id>:<client secret> spark.sql.catalog.<warehouse>.scope=api spark.sql.catalog.<warehouse>.token=<token> spark.sql.catalog.<warehouse>.token-refresh-enabled=false spark.sql.catalog.<warehouse>.header.X-Iceberg-Access-Delegation=vended-credentials spark.sql.catalog.<warehouse>.io-impl=org.apache.iceberg.gcp.gcs.GCSFileIO
Querying:
SHOW NAMESPACES IN <warehouse>; SHOW TABLES IN <warehouse>.<namespace>; SELECT * FROM <warehouse>.<namespace>.<table> LIMIT 10;
Documentation:
Databricks
Prerequisites:
- Databricks Runtime 17.0 or later: Spark 4.0, Scala 2.13
- A cluster in dedicated (single user) or no isolation shared access mode
- Libraries:
org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.12.0andorg.apache.iceberg:iceberg-gcp-bundle:1.12.0 - The client's
<client id>:<client secret>in a Databricks secret scope
In the cluster's Spark config:
spark.sql.catalog.<warehouse> org.apache.iceberg.spark.SparkCatalog spark.sql.catalog.<warehouse>.type rest spark.sql.catalog.<warehouse>.uri https://iceberg.spglobal.com spark.sql.catalog.<warehouse>.warehouse <warehouse> spark.sql.catalog.<warehouse>.rest.auth.type oauth2 spark.sql.catalog.<warehouse>.oauth2-server-uri https://secureapi.signin.spglobal.com/gettoken/token spark.sql.catalog.<warehouse>.credential {{secrets/<scope>/<key>}} spark.sql.catalog.<warehouse>.scope api spark.sql.catalog.<warehouse>.token <token> spark.sql.catalog.<warehouse>.token-refresh-enabled false spark.sql.catalog.<warehouse>.header.X-Iceberg-Access-Delegation vended-credentials spark.sql.catalog.<warehouse>.io-impl org.apache.iceberg.gcp.gcs.GCSFileIO
Then query as in Spark:
SELECT * FROM <warehouse>.<namespace>.<table> LIMIT 10;
Documentation:
DuckDB
Prerequisites:
- DuckDB 1.5 or later with extensions: httpfs, avro and iceberg
- Network access to:
https://iceberg.spglobal.com, the cataloghttps://secureapi.signin.spglobal.com/gettoken/token, the token endpointhttps://storage.googleapis.com, the data files
INSTALL httpfs; INSTALL avro; INSTALL iceberg; LOAD httpfs; LOAD avro; LOAD iceberg; CREATE SECRET iceberg_client ( CREATE SECRET iceberg_token ( TYPE iceberg, CLIENT_ID '<client id>', CLIENT_SECRET '<client secret>', OAUTH2_SERVER_URI 'https://secureapi.signin.spglobal.com/gettoken/token', OAUTH2_SCOPE 'api' TOKEN '<token>' ); ATTACH '<warehouse>' AS <warehouse> ( TYPE iceberg, SECRET iceberg_client, SECRET iceberg_token, ENDPOINT 'https://iceberg.spglobal.com' ); SHOW ALL TABLES; SELECT * FROM <warehouse>.<namespace>.<table> LIMIT 10;
Documentation:
Google Managed Service for Apache Spark (formerly Dataproc)
Prerequisites:
- A cluster on image 3.0 or later: Spark 4.0, with the optional Iceberg component, Iceberg 1.11
- Library:
org.apache.iceberg:iceberg-gcp-bundleof the component's Iceberg version,1.11.0on 3.0, for Cloud Storage io-implset toGCSFileIO, as below: else Iceberg reads through the cluster's Cloud Storage connector, as its service account, not with the vended credentials- Network access to:
https://iceberg.spglobal.com, the cataloghttps://secureapi.signin.spglobal.com/gettoken/token, the token endpointhttps://storage.googleapis.com, the data files
- The client's secret in Secret Manager, set by the job, rather than in the cluster's properties
The cluster:
gcloud dataproc clusters create <cluster> --region=<region> --image-version=3.0 \ --optional-components=ICEBERG \ --properties="^#^spark:spark.jars.packages=org.apache.iceberg:iceberg-gcp-bundle:1.11.0#\ spark:spark.sql.catalog.<warehouse>=org.apache.iceberg.spark.SparkCatalog#\ spark:spark.sql.catalog.<warehouse>.type=rest#\ spark:spark.sql.catalog.<warehouse>.uri=https://iceberg.spglobal.com#\ spark:spark.sql.catalog.<warehouse>.warehouse=<warehouse>#\ spark:spark.sql.catalog.<warehouse>.rest.auth.type=oauth2#\ spark:spark.sql.catalog.<warehouse>.oauth2-server-uri=https://secureapi.signin.spglobal.com/gettoken/token#\ spark:spark.sql.catalog.<warehouse>.scope=api#\ spark:spark.sql.catalog.<warehouse>.token-refresh-enabled=false#\ spark:spark.sql.catalog.<warehouse>.header.X-Iceberg-Access-Delegation=vended-credentials#\ spark:spark.sql.catalog.<warehouse>.io-impl=org.apache.iceberg.gcp.gcs.GCSFileIO"
Or with Terraform:
resource "google_dataproc_cluster" "iceberg" {
name = "<cluster>"
region = "<region>"
cluster_config {
software_config {
image_version = "3.0"
optional_components = ["ICEBERG"]
override_properties = {
"spark:spark.jars.packages" = "org.apache.iceberg:iceberg-gcp-bundle:1.11.0"
"spark:spark.sql.catalog.<warehouse>" = "org.apache.iceberg.spark.SparkCatalog"
"spark:spark.sql.catalog.<warehouse>.type" = "rest"
"spark:spark.sql.catalog.<warehouse>.uri" = "https://iceberg.spglobal.com"
"spark:spark.sql.catalog.<warehouse>.warehouse" = "<warehouse>"
"spark:spark.sql.catalog.<warehouse>.rest.auth.type" = "oauth2"
"spark:spark.sql.catalog.<warehouse>.oauth2-server-uri" = "https://secureapi.signin.spglobal.com/gettoken/token"
"spark:spark.sql.catalog.<warehouse>.scope" = "api"
"spark:spark.sql.catalog.<warehouse>.token-refresh-enabled" = "false"
"spark:spark.sql.catalog.<warehouse>.header.X-Iceberg-Access-Delegation" = "vended-credentials"
"spark:spark.sql.catalog.<warehouse>.io-impl" = "org.apache.iceberg.gcp.gcs.GCSFileIO"
}
}
}
}
A job, which sets the credential before its first query:
gcloud dataproc jobs submit pyspark job.py --cluster=<cluster> --region=<region>
spark.conf.set("spark.sql.catalog.<warehouse>.credential", "<client id>:<client secret>")
spark.conf.set("spark.sql.catalog.<warehouse>.token", "<token>")
Documentation:
PyIceberg
Prerequisites:
- PyIceberg with Cloud Storage and pandas:
pip install "pyiceberg[gcsfs,pandas]" - Network access to:
https://iceberg.spglobal.com, the cataloghttps://secureapi.signin.spglobal.com/gettoken/token, the token endpointhttps://storage.googleapis.com, the data files
Then:
from pyiceberg.catalog import load_catalog
catalog = load_catalog(
"<warehouse>",
type="rest",
uri="https://iceberg.spglobal.com",
warehouse="<warehouse>",
auth={
"type": "oauth2",
"oauth2": {
"client_id": "<client id>",
"client_secret": "<client secret>",
"token_url": "https://secureapi.signin.spglobal.com/gettoken/token",
"scope": "api",
},
},
token="<token>",
**{"header.X-Iceberg-Access-Delegation": "vended-credentials"},
)
print(catalog.list_namespaces())
print(catalog.list_tables("<namespace>"))
print(catalog.load_table(("<namespace>", "<table>")).scan(limit=10).to_pandas())
Documentation:
Snowflake
Prerequisites:
- A role that may create catalog integrations: ACCOUNTADMIN, or one granted
CREATE INTEGRATION ON ACCOUNT
CREATE CATALOG INTEGRATION data_fabric
CATALOG_SOURCE = ICEBERG_REST
TABLE_FORMAT = ICEBERG
CATALOG_NAMESPACE = '<namespace>'
REST_CONFIG = (
CATALOG_URI = 'https://iceberg.spglobal.com'
WAREHOUSE = '<warehouse>'
ACCESS_DELEGATION_MODE = VENDED_CREDENTIALS
)
REST_AUTHENTICATION = (
TYPE = OAUTH
OAUTH_TOKEN_URI = 'https://secureapi.signin.spglobal.com/gettoken/token'
OAUTH_CLIENT_ID = '<client id>'
OAUTH_CLIENT_SECRET = '<client secret>'
OAUTH_ALLOWED_SCOPES = ('api')
TYPE = BEARER
BEARER_TOKEN = '<token>'
)
ENABLED = TRUE;
CREATE ICEBERG TABLE <table>
CATALOG = 'data_fabric'
CATALOG_TABLE_NAME = '<table>';
SELECT * FROM <table> LIMIT 10;
Documentation:
Trino
Prerequisites:
- Trino 481 or later
- Network access to:
https://iceberg.spglobal.com, the cataloghttps://secureapi.signin.spglobal.com/gettoken/token, the token endpointhttps://storage.googleapis.com, the data files
A catalog, etc/catalog/<warehouse>.properties:
connector.name=iceberg iceberg.catalog.type=rest iceberg.rest-catalog.uri=https://iceberg.spglobal.com iceberg.rest-catalog.warehouse=<warehouse> iceberg.rest-catalog.security=OAUTH2 iceberg.rest-catalog.oauth2.server-uri=https://secureapi.signin.spglobal.com/gettoken/token iceberg.rest-catalog.oauth2.credential=<client id>:<client secret> iceberg.rest-catalog.oauth2.scope=api iceberg.rest-catalog.oauth2.token=<token> iceberg.rest-catalog.oauth2.token-refresh-enabled=false iceberg.rest-catalog.oauth2.token-exchange-enabled=false iceberg.rest-catalog.vended-credentials-enabled=true fs.gcs.enabled=true
SHOW SCHEMAS FROM <warehouse>; SHOW TABLES FROM <warehouse>.<namespace>; SELECT * FROM <warehouse>.<namespace>.<table> LIMIT 10;
Documentation: