Logging & Observability

PBAC includes structured JSON logging and an optional Grafana/Loki observability stack. In the dev profile, logs are pushed directly from the application to Loki — no log scrapers or agents required.

---

Quick start

# Start MySQL + Loki + Grafana
docker compose up -d

# Run the AS (dev profile — human-readable console + Loki push)
mvn spring-boot:run -Dspring-boot.run.profiles=dev

# Open the dashboard
open http://localhost:3000

The pre-built PBAC Overview dashboard shows:

• Token, authorize, introspect, and callback decisions per minute
• A live decision log with filtering by instance and event type

Grafana is configured with anonymous viewer access — no login required for read-only use. Admin access: admin / admin.

---

Architecture

The logging stack uses three Logback appenders:

Appender	When	Format	Purpose
CONSOLE_TEXT	dev profile	Human-readable text	Local development
CONSOLE_JSON	Non-dev profiles	Structured JSON (LogstashEncoder)	Container/production stdout
LOKI	dev profile	JSON push to Loki	Log aggregation and dashboards

---

Running without Loki

If you only need MySQL (the default), Loki and Grafana won't start:

docker compose up -d   # Only MySQL

The Loki appender gracefully degrades when Loki is unreachable — it logs a warning once and continues without errors. Console logging is unaffected.

---

Structured log events

Key OAuth/PDP decision points emit structured log events with consistent fields:

Event	Service	Outcomes
token_decision	TokenService	allow, deny
authorize_decision	AuthorizeService	allow, deny
introspect_decision	IntrospectService	active, inactive, error
callback_result	OAuth2CallbackService	success, error
register_decision	RegisterService	allow, deny

Every event includes event, outcome, and client_id. Additional fields vary by type (e.g., grant_type for token decisions, idp for callbacks, reason for denials).

Example (JSON console output)

{
  "@timestamp": "2026-03-11T14:30:00.123Z",
  "level": "INFO",
  "message": "Token decision: event=token_decision outcome=allow client_id=my-app grant_type=client_credentials",
  "event": "token_decision",
  "outcome": "allow",
  "client_id": "my-app",
  "grant_type": "client_credentials",
  "request_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
}

---

MDC context

Every HTTP request is tagged with MDC fields by the MdcRequestFilter:

Field	Source	Description
request_id	Auto-generated UUID	Unique per-request correlation ID
client_id	Authorization: Basic header	OAuth client ID (when Basic auth is used)

These fields appear in all log lines for the request — not just decision events.

---

Configuration

Environment variables

Variable	Default	Purpose
LOKI_URL	http://localhost:3100	Loki push endpoint. Set to empty to effectively disable Loki push.
INSTANCE_NAME	local	Instance label sent to Loki. Used for multi-instance filtering in Grafana.

Spring properties

loki:
  url: ${LOKI_URL:http://localhost:3100}

instance:
  name: ${INSTANCE_NAME:local}

Managed instances

The deploy manager automatically passes these env vars when creating instances:

./deploy/manager.sh create demo1
# → INSTANCE_NAME=demo1, LOKI_URL configured per mode (docker/host)

Each instance's logs appear with its name in the Grafana instance selector dropdown.

---

Grafana dashboard

The PBAC Overview dashboard is auto-provisioned when Grafana starts. It includes:

Row 1 — Metrics

Four stat panels showing decisions per minute, split by outcome:

• Token Decisions (allow vs deny)
• Authorize Decisions (allow vs deny)
• Introspect Decisions (active vs inactive)
• Callback Results (success vs error)

Row 2 — Decision log

A live log panel showing recent decision events with all structured fields. Filterable by instance and event type.

Instance selector

A dropdown at the top filters all panels by instance label — useful when running multiple AS instances.

Custom queries

Use Grafana Explore (http://localhost:3000/explore) to run ad-hoc LogQL queries:

# All token denials in the last hour
{app="pbac"} | json | event="token_decision" outcome="deny"

# All events for a specific client
{app="pbac"} | json | client_id="my-client"

# Error rate by event type
sum by (event, outcome)(count_over_time({app="pbac"} | json | outcome="deny" [5m]))

---

Production considerations

Log volume

Structured logging adds ~200 bytes per decision event. At 1000 requests/second, this is ~200 KB/s of log data. Loki's default retention (168 hours) handles this comfortably.

Loki backend

The default Loki configuration uses filesystem storage, suitable for single-node deployments. For production clusters, configure S3/GCS object storage in observability/loki-config.yml.

External log aggregation

The CONSOLE_JSON appender (active in non-dev profiles) outputs structured JSON to stdout. This integrates with any container log collector (Fluentd, Filebeat, CloudWatch Logs agent) without additional configuration.

---

Next steps

• Configuration — Environment variables for Loki URL and instance naming
• Kubernetes — Deploy the observability stack alongside the AS in Kubernetes
• Troubleshooting — Use decision logs and audit queries to diagnose issues