Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/spiceai/spiceai/llms.txt

Use this file to discover all available pages before exploring further.

​
Secrets Management

Spice.ai provides a flexible secrets management system to securely handle credentials, API keys, and other sensitive configuration data. Secrets are never stored in plain text in your spicepod configuration.

​
Overview

The secrets system allows you to:
  • Reference secrets from multiple stores: Environment variables, Kubernetes, AWS Secrets Manager, and more
  • Inject secrets at runtime: Secrets are loaded when Spice starts
  • Prevent credential exposure: Secrets are redacted in logs and error messages
  • Use consistent syntax: ${store:key} format across all configuration

​
Secret Reference Syntax

Reference secrets using the format: 
${<store>:<key>}
  • <store>: The name of the secret store (e.g., env, secrets, kubernetes)
  • <key>: The key/name of the secret within that store
Examples: 
${env:DATABASE_PASSWORD}          # From environment variable
${secrets:API_KEY}                # From any configured secret store
${kubernetes:db-credentials}      # From Kubernetes secret
${aws:prod/database/password}     # From AWS Secrets Manager

​
Secret Stores

​
Environment Variables (default)

The default secret store reads from environment variables and .env files. Configuration: No explicit configuration needed - the env store is always available: 
datasets:
  - from: postgres:production
    name: customers
    params:
      pg_host: db.example.com
      pg_user: ${env:PG_USER}
      pg_pass: ${env:PG_PASSWORD}
Environment variable resolution: The env store searches for variables in this order:
  1. SPICE_<KEY> (prefixed with SPICE_)
  2. <KEY> (exact match)
  3. SPICE_<KEY_UPPERCASE> (prefixed and uppercased)
  4. <KEY_UPPERCASE> (uppercased)
# Example: ${env:database_password} will match any of:
export SPICE_database_password="secret"  # 1. Prefixed exact
export database_password="secret"        # 2. Exact match
export SPICE_DATABASE_PASSWORD="secret"  # 3. Prefixed uppercase
export DATABASE_PASSWORD="secret"        # 4. Uppercase
Using .env files: Spice automatically loads secrets from:
  1. .env.local (highest priority)
  2. .env
  3. Custom path (if specified)
# .env.local
DATABASE_PASSWORD=my-secret-password
API_KEY=sk-1234567890abcdef
SPICE_OPENAI_API_KEY=sk-proj-...
Add .env.local and .env files to your .gitignore to prevent committing secrets.
Custom .env file path: 
secrets:
  - name: custom_env
    from: env
    params:
      file_path: /etc/spice/production.env

runtime:
  auth:
    api_key:
      keys:
        - ${custom_env:ADMIN_KEY}:rw

​
Kubernetes Secrets

Load secrets from Kubernetes Secret resources. Configuration: 
secrets:
  - name: k8s
    from: kubernetes:my-secret-name

datasets:
  - from: postgres:db
    name: users
    params:
      pg_user: ${k8s:username}
      pg_pass: ${k8s:password}
Create Kubernetes secret: 
kubectl create secret generic my-secret-name \
  --from-literal=username=dbuser \
  --from-literal=password=dbpass123
Key naming conventions: Kubernetes store searches for keys with spice_ prefix first:
  1. spice_<key> (prefixed)
  2. <key> (exact match)
# Kubernetes secret with spice_ prefix
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
stringData:
  spice_pg_user: "admin"    # Preferred
  pg_pass: "secret"         # Also works

​
AWS Secrets Manager

Store secrets in AWS Secrets Manager for centralized, cloud-based secret management. Configuration: 
secrets:
  - name: aws
    from: aws_secrets_manager:spice-production-secrets

datasets:
  - from: postgres:production
    name: customers
    params:
      pg_host: ${aws:db-host}
      pg_user: ${aws:db-username}
      pg_pass: ${aws:db-password}
Creating secrets in AWS: 
# Create a secret in AWS Secrets Manager
aws secretsmanager create-secret \
  --name spice-production-secrets \
  --description "Database credentials for Spice" \
  --secret-string '{
    "db-host": "prod-db.example.com",
    "db-username": "spice_app",
    "db-password": "super-secret-password"
  }'
AWS Authentication: Spice uses the default AWS credential chain:
  1. Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
  2. AWS credentials file (~/.aws/credentials)
  3. IAM role (for EC2, ECS, EKS)

​
Keyring (macOS/Linux)

Use the operating system’s secure credential storage. Configuration: 
secrets:
  - name: keyring
    from: keyring

runtime:
  auth:
    api_key:
      keys:
        - ${keyring:spice-api-key}:rw
Storing secrets in keyring: 
# macOS: Use Keychain Access app or security command
security add-generic-password \
  -a "$USER" \
  -s "spice-api-key" \
  -w "my-secret-key"

# Linux: Use secret-tool (GNOME Keyring)
secret-tool store \
  --label="Spice API Key" \
  application spice \
  key spice-api-key
# Enter password when prompted

​
Multiple Secret Stores

Configure multiple secret stores with different priorities: 
secrets:
  # Higher priority (checked first)
  - name: prod_secrets
    from: aws_secrets_manager:production
  
  # Medium priority
  - name: k8s_secrets
    from: kubernetes:spice-secrets
  
  # Lower priority (checked last)
  - name: env
    from: env

runtime:
  auth:
    api_key:
      keys:
        # Tries prod_secrets first, then k8s_secrets, then env
        - ${secrets:API_KEY}:rw
The special secrets store searches all configured stores in order until it finds the key.

​
Common Use Cases

​
Database Credentials

secrets:
  - name: env
    from: env

datasets:
  - from: postgres:analytics
    name: events
    params:
      pg_host: ${env:POSTGRES_HOST}
      pg_port: "5432"
      pg_db: analytics
      pg_user: ${env:POSTGRES_USER}
      pg_pass: ${env:POSTGRES_PASSWORD}
      pg_sslmode: require
  
  - from: mysql:production
    name: users
    params:
      mysql_host: ${env:MYSQL_HOST}
      mysql_tcp_port: "3306"
      mysql_db: users
      mysql_user: ${env:MYSQL_USER}
      mysql_pass: ${env:MYSQL_PASSWORD}

​
API Keys for Data Sources

secrets:
  - name: env
    from: env

datasets:
  - from: databricks:my_catalog.my_schema.my_table
    name: analytics_data
    params:
      databricks_endpoint: ${env:DATABRICKS_ENDPOINT}
      databricks_token: ${env:DATABRICKS_TOKEN}
  
  - from: snowflake:database.schema.table
    name: warehouse_data
    params:
      snowflake_account: ${env:SNOWFLAKE_ACCOUNT}
      snowflake_username: ${env:SNOWFLAKE_USER}
      snowflake_password: ${env:SNOWFLAKE_PASSWORD}

​
LLM and AI Model Credentials

secrets:
  - name: env
    from: env

models:
  - name: gpt-4o
    from: openai:gpt-4o
    params:
      openai_api_key: ${env:OPENAI_API_KEY}
  
  - name: claude
    from: anthropic:claude-3-5-sonnet-20241022
    params:
      anthropic_api_key: ${env:ANTHROPIC_API_KEY}

​
TLS Certificates

secrets:
  - name: k8s
    from: kubernetes:tls-certificates

runtime:
  tls:
    enabled: true
    certificate: ${k8s:tls-cert}
    key: ${k8s:tls-key}

​
Runtime Authentication

secrets:
  - name: env
    from: env

runtime:
  auth:
    api_key:
      enabled: true
      keys:
        - ${env:SPICE_API_KEY_ADMIN}:rw
        - ${env:SPICE_API_KEY_READONLY}:ro
        - ${env:SPICE_API_KEY_ANALYTICS}:ro

​
Security Best Practices

​
1. Never Commit Secrets

Never commit secrets to version control. Use .gitignore to exclude secret files.
# .gitignore
.env
.env.local
.env.*.local
*.pem
*.key
secrets/

​
2. Use Environment-Specific Secrets

Separate secrets for different environments: 
# Development: .env.local
DATABASE_URL=postgres://localhost:5432/dev
API_KEY=dev-key-12345

# Production: Kubernetes/AWS Secrets
# Never stored in files

​
3. Principle of Least Privilege

Grant minimal access to secret stores: 
# AWS IAM Policy - read-only access to specific secrets
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": "secretsmanager:GetSecretValue",
    "Resource": "arn:aws:secretsmanager:us-east-1:123456789:secret:spice-*"
  }]
}

​
4. Rotate Secrets Regularly

Implement a rotation policy: 
# Example: AWS Secrets Manager rotation
aws secretsmanager rotate-secret \
  --secret-id spice-production-secrets \
  --rotation-lambda-arn arn:aws:lambda:us-east-1:123456789:function:rotate-secret

​
5. Monitor Secret Access

Enable audit logging: 
# AWS CloudTrail for Secrets Manager access
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceType,AttributeValue=AWS::SecretsManager::Secret

# Kubernetes audit logs
kubectl logs -n kube-system kube-apiserver-* | grep "secrets"

​
6. Use Secure Communication

Combine secrets management with TLS: 
runtime:
  tls:
    enabled: true
    certificate_file: ${secrets:TLS_CERT_PATH}
    key_file: ${secrets:TLS_KEY_PATH}
  auth:
    api_key:
      keys:
        - ${secrets:API_KEY}:rw

​
Troubleshooting

​
Secret Not Found

# Error: Key 'DATABASE_PASSWORD' not found in secret store 'env'

# Check environment variables
env | grep DATABASE

# Check .env files
cat .env
cat .env.local

# Verify secret store configuration
grep -A 5 "secrets:" spicepod.yaml

​
AWS Secrets Manager Connection Issues

# Verify AWS credentials
aws sts get-caller-identity

# Test secret access
aws secretsmanager get-secret-value --secret-id spice-production-secrets

# Check IAM permissions
aws iam get-user
aws iam list-attached-user-policies --user-name your-user

​
Kubernetes Secret Access Issues

# Verify secret exists
kubectl get secret my-secret-name

# Check secret contents
kubectl get secret my-secret-name -o yaml

# Verify service account permissions
kubectl get serviceaccount -n spice-namespace
kubectl describe serviceaccount spice-sa -n spice-namespace

​
Case Sensitivity Issues

# Environment variables are case-sensitive on Linux/macOS
# Use consistent casing:

# Good
export DATABASE_PASSWORD="secret"
params:
  pg_pass: ${env:DATABASE_PASSWORD}

# Bad - case mismatch
export database_password="secret"
params:
  pg_pass: ${env:DATABASE_PASSWORD}  # Won't find it

​
Production Deployment Patterns

​
Kubernetes with External Secrets Operator

# ExternalSecret custom resource
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: spice-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: spice-secrets
  data:
  - secretKey: database-password
    remoteRef:
      key: prod/spice/db-password
---
# Spice deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spice
spec:
  template:
    spec:
      containers:
      - name: spice
        envFrom:
        - secretRef:
            name: spice-secrets

​
Docker Compose with Environment Files

# docker-compose.yml
version: '3.8'
services:
  spice:
    image: spiceai/spiceai:latest
    env_file:
      - .env.production
    volumes:
      - ./spicepod.yaml:/app/spicepod.yaml
    ports:
      - "3000:3000"
      - "50051:50051"

# .env.production (not committed)
DATABASE_PASSWORD=super-secret
API_KEY=sk-prod-12345
AWS_ACCESS_KEY_ID=AKIA...
AWS_SECRET_ACCESS_KEY=...

​
Cloud Run / Cloud Functions

# Deploy with secrets from Google Secret Manager
gcloud run deploy spice \
  --image spiceai/spiceai:latest \
  --set-secrets="DATABASE_PASSWORD=database-password:latest" \
  --set-secrets="API_KEY=api-key:latest"