Operations

Configure, monitor, and troubleshoot the PolicyOne → BillingOne event pipeline in every environment.

← Payload Reference

Required Services

Both services must be reachable for the pipeline to function.

Redis is not included in docker-compose.yml. Run it locally (brew install redis && redis-server) or add a service to compose.

Environment Variables

PolicyOne (apps/policyone/.env.local)

BillingOne (apps/billingone/.env.local)

Minimal .env.local for local development

# PolicyOne
DATABASE_URL=postgresql://policyone_app:localdevpass@localhost:5480/policyone
REDIS_HOST=localhost
REDIS_PORT=6379
BILLING_ONE_URL=http://localhost:3048
BILLING_ONE_API_KEY=bo_dev_seed_key_for_local_testing_only
DISABLE_OUTBOX_WORKER=false

Worker Startup

OutboxPublisher starts automatically when PolicyOne's Next.js server boots, via apps/policyone/instrumentation.ts:

// instrumentation.ts (simplified)
export async function register() {
  if (
    process.env.NEXT_RUNTIME === 'nodejs' &&
    process.env.DISABLE_OUTBOX_WORKER !== 'true'
  ) {
    const { OutboxPublisher } = await import('./lib/workers/outbox-publisher.ts')
    const { ClaimsEventsConsumer } = await import('./lib/workers/claims-events-consumer.ts')

    await new OutboxPublisher().start()
    await new ClaimsEventsConsumer().start()
  }
}

BillingOne workers (including PolicyEventsConsumer) start automatically when the BillingOne server boots. No manual step needed — just ensure both servers are running and Redis is reachable.

Verifying the Pipeline is Working

1. Check outbox rows

-- How many rows are pending / processing / published / failed?
SELECT status, COUNT(*) FROM policy.event_outbox GROUP BY status;

-- Latest published events
SELECT id, event_type, published_at
FROM   policy.event_outbox
WHERE  status = 'published'
ORDER  BY published_at DESC
LIMIT  10;

-- Any failures?
SELECT id, event_type, failure_reason, failed_at
FROM   policy.event_outbox
WHERE  status = 'failed';

2. Check Redis stream

# Count messages in stream
redis-cli XLEN events:policy

# Inspect last 5 messages
redis-cli XREVRANGE events:policy + - COUNT 5

# Check BillingOne consumer group lag
redis-cli XINFO GROUPS events:policy
# Look for: "billing-center" → "pending" count (should be 0 when caught up)

3. Check DLQ

# Any dead-lettered messages?
redis-cli XLEN events:dlq

# Inspect DLQ entries
redis-cli XRANGE events:dlq - + COUNT 10

Troubleshooting

Outbox rows stuck in pending

Cause: OutboxPublisher is not running (Redis unreachable, or DISABLE_OUTBOX_WORKER=true).

Fix:

1. Verify Redis is running: redis-cli ping → PONG
2. Check DISABLE_OUTBOX_WORKER is not set to true
3. Restart PolicyOne server

Outbox rows stuck in processing

Cause: The server crashed mid-publish. These are automatically reset to pending on the next boot by resetStalledRows().

Manual fix (if needed):

UPDATE policy.event_outbox
SET    status = 'pending'
WHERE  status = 'processing';

BillingOne consumer group lag is growing

Cause: BillingOne's PolicyEventsConsumer is not running or is falling behind.

Fix:

1. Restart BillingOne server
2. Check REDIS_HOST / REDIS_PORT match PolicyOne's Redis
3. Increase STREAM_BATCH_SIZE or WORKER_CONCURRENCY in BillingOne's env

policy.bound event published but BillingOne has no billing account

Cause: policy.quote_billing row was missing at bind time — agent skipped the payment plan step.

Check:

-- Does this quote have a billing row?
SELECT * FROM policy.quote_billing WHERE quote_id = '<quote-id>';

-- Check the outbox payload
SELECT payload->'payload'->'billing' AS billing
FROM   policy.event_outbox
WHERE  event_type = 'policy.bound'
ORDER  BY created_at DESC
LIMIT  5;

Payment plan preview fails with effectiveDate is required

Cause: quotes.effective_start_date is NULL — the quote was rated before the effective date was saved.

Check:

SELECT effective_start_date FROM policy.quotes WHERE id = '<quote-id>';

If NULL, the agent needs to re-enter the effective date in the transaction step and re-rate.

Local Development Quick-Start

# 1. Start Redis
redis-server &

# 2. Start PostgreSQL (via docker-compose)
docker-compose up -d

# 3. Apply migrations
cd packages/db && pnpm drizzle-kit migrate

# 4. Start BillingOne (seeds payment plans automatically)
cd apps/billingone && pnpm dev      # → http://localhost:3048

# 5. Start PolicyOne
cd apps/policyone && pnpm dev       # → http://localhost:3047
# OutboxPublisher and ClaimsEventsConsumer start automatically

Tip: Set DISABLE_OUTBOX_WORKER=true when running PolicyOne in unit tests to prevent the worker from connecting to Redis.