Skip to main content
Version: 1.0

Troubleshooting

Quick reference for when things break. Each section starts with what you see, then why it happens, then how to fix it.

OPA bundle not loading​

Symptom: OPA starts but token requests fail with policy denials immediately, or OPA logs show repeated bundle fetch errors like bundle load failed or Bundle download failed.

Causes:

  1. Wrong or missing X-Bundle-API-Key — OPA's config sets this via the PBAC_BUNDLE_API_KEY environment variable. If OPA was started without it, or with a different value than the AS expects, every bundle fetch returns 401.
  2. AS not running yet — OPA starts before the AS (correct order: MySQL → AS → OPA). If OPA starts first, it will fail to fetch the bundle until the AS is up.
  3. Wrong bundle URL — opa/config.yaml points to http://host.docker.internal:8080/bundles/pbac/bundle.tar.gz. On Linux hosts, host.docker.internal may not resolve; you may need 172.17.0.1 or the host's actual IP.
  4. AS bundle endpoint returning an error — the AS logs a Bundle build failed error if the DB is unreachable when OPA requests the bundle.

Diagnose:

Check OPA's console logs (they are set to debug level by default):

docker logs pbac-opa --tail 50

Test the bundle endpoint directly from the host. Replace dev-bundle-key with whatever PBAC_BUNDLE_API_KEY is set to:

curl -v -H "X-Bundle-API-Key: dev-bundle-key" \
  http://localhost:8080/bundles/pbac/bundle.tar.gz \
  -o /dev/null

A 200 with a non-empty body means the AS side is fine. A 401 means the key is wrong or missing. A 500 means the AS had an internal error (check the AS logs).

Check that OPA has loaded a bundle successfully:

curl http://localhost:8181/v1/data/oauth/token -s | jq .

If this returns {} or a missing-rule error instead of a policy result structure, OPA has no policies loaded yet.

Fix:

  • Ensure PBAC_BUNDLE_API_KEY is set to the same value when starting both the AS and OPA:

    PBAC_BUNDLE_API_KEY=my-secret ./opa/run_opa.sh

  • Start services in order: MySQL first, then the AS, then OPA.

  • On Linux, replace host.docker.internal in opa/config.yaml with the actual host IP or use --network host for the OPA container.

  • Wait 5–10 seconds after startup; OPA polls the bundle every 5–10 seconds (polling.min_delay_seconds / max_delay_seconds in opa/config.yaml).

Token request denied unexpectedly​

Symptom: POST /token returns {"error":"access_denied"} or {"error":"unauthorized_client"} when you expect a token to be issued. The request looks correct.

Causes:

  1. OPA is unreachable — if the AS cannot connect to OPA at OPA_URL, token issuance fails.
  2. OPA has no policies loaded — if the bundle hasn't been fetched yet, OPA returns an empty result which the AS treats as a denial.
  3. Missing or incorrect policy data — the token policy depends on data.oauth entries (resource definitions, subject entitlements). If policy_data rows are missing from the DB, OPA evaluates against an empty dataset and denies.
  4. grant_type not in the client's software statement — the token policy checks that the requested grant type is authorised for the client.
  5. Client is disabled or revoked — the AS checks enabled = true and revoked_at IS NULL before calling OPA.
  6. Denylist — the token policy may include explicit deny rules for certain subjects or clients.

Diagnose:

Enable OPA decision logs (already on by default — decision_logs.console: true in opa/config.yaml). Look for the decision immediately after the failed token request:

docker logs pbac-opa --tail 20

You will see JSON lines with "result" and "input" — check result.allow, result.reason, and result.decision.

Check OPA directly with the same input your AS would send:

curl -s http://localhost:8181/v1/data/oauth/token \
  -H "Content-Type: application/json" \
  -d '{"input": {"request": {"grant_type": "client_credentials", "scopes": ["read"]}, "client": {"client_id": "my-client"}}}' \
  | jq .

Check that OPA is reachable from the AS:

curl -s http://localhost:8181/health | jq .

Check the AS application logs for PDP evaluation failed or OPA returned error.

Fix:

  • If OPA is unreachable: verify OPA_URL is set correctly and OPA is running.
  • If policies are missing: wait for the bundle to reload (up to 10 seconds), or check GET /bundles/pbac/bundle.tar.gz returns a valid tarball.
  • If policy data is missing: provision the required policy_data rows via the Admin API (POST /admin/policy-data) or seed the DB.
  • If the client config is wrong: check the client's software_statement field in the Admin UI or via GET /admin/clients/{id}.

Introspect returns active: false​

Symptom: POST /introspect returns {"active": false} for a token you just issued.

Causes:

  1. Token is expired — the AS checks expires_at before calling OPA.
  2. JIT single-use token already consumed — if the PDP set jit_single_use: true in the token's pdpOutput, the first successful introspect deletes the token. A second call returns active: false.
  3. Wrong scope on the calling token — the caller must present a Bearer token with introspection or uma_protection scope, or authenticate via Basic auth with a client that is entitled to those scopes. An incorrect or insufficient caller token causes a 401/403, not active: false, but misconfigured clients can appear as denials.
  4. OPA re-evaluation denied — introspect re-evaluates the token policy at call time. If policy data has changed (e.g. a subject was removed), the re-evaluation may deny.
  5. Client disabled or revoked — the client that originally received the token has been disabled since issuance.
  6. Token simply does not exist — the token was never issued (typo, wrong environment, wrong AS instance).

Diagnose:

Check OPA decision logs immediately after the introspect call:

docker logs pbac-opa --tail 20

Look for the allow and reason fields in the decision. The AS logs a deny audit entry at WARN level when introspection fails.

Check whether the token exists and when it expires (Admin API):

curl -s -H "X-Admin-API-Key: dev-admin-key" \
  "http://localhost:8080/admin/audit-logs?limit=5" | jq .

Check the audit log for the specific token's most recent /introspect entries to see whether jit_single_use revocation happened.

For JIT single-use, look for an audit entry with "action": "token_revoked" and "reason": "jit_single_use" immediately after the first successful introspect.

Fix:

  • Expired token: re-request a token from /token.
  • JIT single-use: by design — request a new token for each use.
  • Wrong caller scope: ensure the RS client has introspection or uma_protection in its granted scopes. Check via GET /admin/clients/{id} and verify the software_statement.granted_resources or the policy data grants the right scopes.
  • OPA re-evaluation denied: check policy data for the subject/resource — provision missing entries via POST /admin/policy-data.

Admin API returns 401 or 403​

Symptom: Calls to /admin/** or /access/** return 401 Unauthorized or 403 Forbidden.

Cause:

The X-Admin-API-Key header is missing, blank, or does not match PBAC_ADMIN_API_KEY. When PBAC_ADMIN_API_KEY is set to a non-blank value, all /admin/** and /access/** requests require this header. In local dev the variable defaults to blank (open), so this only triggers in environments where the key was explicitly configured.

Diagnose:

Check what key the AS started with by looking at startup logs:

# If the key is set, you will NOT see this warning at startup:
# "SECURITY WARNING: Admin API is unprotected"
# If you do see that warning, the key is blank and the endpoint should be open.

Try the request with the key:

curl -v -H "X-Admin-API-Key: your-key-here" \
  http://localhost:8080/admin/clients

Fix:

  • Pass X-Admin-API-Key: <value> in every admin request.
  • Verify the key matches the PBAC_ADMIN_API_KEY environment variable the AS was started with.
  • In local dev without the variable set, omit the header entirely — the endpoint is open.

MySQL connection errors​

Symptom: The AS fails to start with an error like Communications link failure, Access denied for user 'pbac', or Unknown database 'pbac'. Or the AS starts but requests fail with DB errors.

Causes:

  1. Docker not running or MySQL container not started — MySQL runs as a Docker container defined in docker-compose.yml.
  2. Wrong credentials — default credentials are pbac/pbac-dev, database pbac. If overridden via SPRING_DATASOURCE_USERNAME, SPRING_DATASOURCE_PASSWORD, or SPRING_DATASOURCE_URL, a mismatch causes auth failure.
  3. SSL misconfiguration — the base application.yml JDBC URL does not set useSSL=false. The dev profile does add useSSL=false&allowPublicKeyRetrieval=true. In production, if your MySQL instance requires SSL, ensure the keystore is configured.
  4. Flyway migration failure — if the schema is out of sync (e.g., you pulled new code with new migrations but the DB was not updated), the AS will refuse to start with Validate failed: detected resolved migration not applied.
  5. Port conflict — MySQL is mapped to 3306. If another process holds that port, Docker fails to bind.

Diagnose:

Check if MySQL is running:

docker-compose ps
docker-compose logs mysql --tail 30

Test connectivity from the host:

mysql -h 127.0.0.1 -P 3306 -u pbac -ppbac-dev pbac -e "SELECT 1;"

Check AS startup logs for the exact exception. Flyway errors look like:

FlywayException: Validate failed: Detected resolved migration not applied to database

Fix:

  • Start MySQL: docker-compose up -d
  • Wait ~10 seconds for MySQL to initialise on first run (the AS may need to be restarted if it started before MySQL was ready).
  • For Flyway migration failures: run mvn flyway:migrate or let the AS apply migrations automatically on next startup (Flyway will apply pending migrations in order).
  • For SSL issues in production: either configure the JDBC URL with the appropriate SSL parameters, or confirm useSSL=false is acceptable for your network topology.
  • If port 3306 is in use: stop the conflicting process or change the port mapping in docker-compose.yml and update SPRING_DATASOURCE_URL.

Startup warnings about insecure defaults​

Symptom: The AS logs lines like:

SECURITY WARNING: Admin API is unprotected — set PBAC_ADMIN_API_KEY to require authentication on /admin/** and /access/**
SECURITY WARNING: Bundle endpoint is unprotected — set PBAC_BUNDLE_API_KEY to require authentication on /bundles/**

What it means:

These warnings are emitted by SecurityConfig.warnIfUnprotected() at startup. They indicate:

  • PBAC_ADMIN_API_KEY is blank (the default) — /admin/** and /access/** are completely open to anyone who can reach the AS. Fine for local dev; dangerous in any shared or production environment.
  • PBAC_BUNDLE_API_KEY is dev-bundle-key or blank — the bundle endpoint has no real secret protecting it. OPA can fetch policy data without authentication.

Fix for production:

Set both variables to strong, randomly generated values before deployment:

export PBAC_ADMIN_API_KEY=$(openssl rand -hex 32)
export PBAC_BUNDLE_API_KEY=$(openssl rand -hex 32)

Then configure OPA to use the bundle key by setting PBAC_BUNDLE_API_KEY when starting OPA:

PBAC_BUNDLE_API_KEY="$PBAC_BUNDLE_API_KEY" ./opa/run_opa.sh

OPA's config.yaml passes this as the X-Bundle-API-Key header:

services:
  as:
    headers:
      X-Bundle-API-Key: ${PBAC_BUNDLE_API_KEY}

The warnings disappear once both keys are set to non-blank values. If you intentionally want an open admin endpoint (e.g., behind a network firewall), you can acknowledge the warning and move on — it does not block startup.

Registration denied​

Symptom: POST /register returns {"error":"registration_denied"} or {"error":"invalid_client_metadata: ..."}.

Causes:

  1. jwks_uri not using HTTPS — if the registration request includes a jwks_uri, it must use the https:// scheme. The AS rejects http:// values with invalid_client_metadata: jwks_uri must use HTTPS scheme. This check runs in both RegisterService (DCR) and ClientAssertionValidator.
  2. Policy denied the registration — the OPA register policy (/v1/data/oauth/register) evaluated the DCR request and returned decision: deny. This typically means the client's software statement does not satisfy the configured registration rules.
  3. Untrusted software statement issuer — if PBAC_SOFTWARE_STATEMENT_VERIFY=true, the AS verifies the JWT signature of the software statement. An unknown issuer or invalid signature will cause rejection.
  4. Missing required claims — the register policy checks for required claims in the software statement (e.g., grant_types, redirect_uris for public clients, software_id). Missing claims produce a denial with a reason field in the OPA output.
  5. private_key_jwt without JWKS — if the client registers with token_endpoint_auth_method: private_key_jwt but provides neither jwks_uri nor an inline jwks, registration is rejected.
  6. Step-up required — the policy may return decision: step_up with obligations (e.g., requiring additional verification before registration proceeds). The AS returns a 403 with an obligations field.

Diagnose:

The error response body includes a reason field from the OPA policy when available. Check it first:

curl -s -X POST http://localhost:8080/register \
  -H "Content-Type: application/json" \
  -d '{"client_name":"test","grant_types":["client_credentials"]}' \
  | jq .

Check OPA decision logs for the register evaluation:

docker logs pbac-opa --tail 20

The input to the register policy is logged in the AS audit log — check GET /admin/audit-logs for recent /register entries to see the full pdpInput and policyDecision.

Test the register policy directly against OPA with your input:

curl -s http://localhost:8181/v1/data/oauth/register \
  -H "Content-Type: application/json" \
  -d '{"input": {"request": {"grant_types": ["client_credentials"], "client_name": "test"}, "software_statement": {}}}' \
  | jq .result

Fix:

  • Use https:// for jwks_uri. In local dev, use a tunnelling tool (e.g., ngrok) to expose a local JWKS endpoint over HTTPS, or register with an inline jwks instead.
  • If the policy is denying: inspect the reason field and adjust the software statement claims to match what the policy requires. See opa/policies/oauth/register.rego for the rules.
  • If signature verification is failing: either set PBAC_SOFTWARE_STATEMENT_VERIFY=false for testing, or ensure the software statement is signed by an issuer with a reachable JWKS endpoint.
  • For step-up: check the obligations field in the 403 response — it describes what additional steps are required before registration can proceed.

Next steps​

  • OPA Setup — Bundle server integration and OPA configuration details
  • Configuration — Environment variables and production hardening checklist
  • Observability — Structured logging and Grafana dashboards for monitoring