When you update policies, domains, or users in Housecarl, you need to understand what guarantees the system provides and how changes propagate. This guide explains the transactional semantics, timing guarantees, and safe update procedures.
Short answer: Immediately for most operations, with important caveats for JWT tokens.
Detailed answer: It depends on what you're updating. Let's break it down.
Policy changes are immediately effective for all new authorization requests.
# You deploy a new policy (requires domain UUID)
housectl domain put-policies 550e8400-e29b-41d4-a716-446655440000 new-policy.toml
# Immediately after this returns successfully:
# - New authz requests use the new policy
# - In-flight requests continue with old policies (see below)
# - No service restart needed
# - No cache warming needed
Housecarl provides ACID (Atomicity, Consistency, Isolation, Durability) guarantees for policy updates:
Policy updates are atomic at the domain level. When you use put-policies:
# Deploy multiple policies atomically (requires domain UUID)
housectl domain put-policies 550e8400-e29b-41d4-a716-446655440000 policy1.toml policy2.toml policy3.toml
Either:
No partial updates. You never end up with only policy1 and policy2 deployed but policy3 missing.
Policy updates maintain consistency invariants:
If any invariant is violated, the entire update is rejected:
$ housectl domain put-policies 550e8400-e29b-41d4-a716-446655440000 invalid-policy.toml
Error: PolicyValidationError: Invalid regex pattern in policy 'bad-regex-policy'
# No changes are made
Concurrent policy updates are serialized at the domain level:
Scenario: Two admins update the same domain simultaneously:
Result: One update completes, then the other. The second update sees the effects of the first. The final state is the result of the second update (last write wins).
Scenario: Two admins update different domains:
Result: Updates happen in parallel with no blocking. Fully isolated.
Once housectl domain put-policies returns successfully:
Q: What happens to authorization requests that are being evaluated when I update policies?
A: In-flight requests continue with the policy snapshot they started with.
Timeline:
14:00:00.000 - Request A starts (using old policies)
14:00:00.100 - Policy update begins
14:00:00.200 - Policy update commits
14:00:00.300 - Request A completes (still using old policies)
14:00:00.400 - Request B starts (using new policies)
Why: Policy evaluation loads the policy set at the start of the request. This prevents race conditions where a policy changes mid-evaluation.
Practical impact: For a brief moment (milliseconds to seconds depending on request complexity), some requests use old policies while others use new policies. This is generally acceptable and prevents inconsistent evaluation states.
Here's the safe procedure for updating policies in production:
# Switch to dev tenant
housectl config use dev-context
# Deploy policies to dev domain (get domain UUID first with: housectl domain list)
housectl domain put-policies 660e8400-e29b-41d4-a716-446655440001 new-policies/
# Test authorization
housectl authz can-i test-request.json
# Verify expected behavior
# Parse policies locally (fast validation)
housectl authz parse-policies new-policies/
# Expected output:
All policies are valid.
# Test policies offline without touching any server
housectl authz can-i-local --request test-request.json new-policies/policy1.toml
# Switch to production tenant
housectl config use prod-context
# Get your production domain UUID
housectl domain list
# Atomic deployment (replace UUID with your production domain UUID)
housectl domain put-policies 550e8400-e29b-41d4-a716-446655440000 new-policies/
# Verify deployment (list-policies accepts name or UUID)
housectl domain list-policies production
After deploying:
If you need to revert a policy change:
# Option 1: Redeploy the previous version (recommended)
# (Assumes you keep policy versions in git)
git checkout HEAD~1 policies/
housectl domain put-policies 550e8400-e29b-41d4-a716-446655440000 policies/
# Option 2: Remove the problematic policy
# (Deploy only the policies you want to keep)
housectl domain put-policies 550e8400-e29b-41d4-a716-446655440000 good-policy1.toml good-policy2.toml
Important: There's no "undo" button. You roll back by deploying the previous policy set. This is why version control (git) for policy files is critical.
Domain creation is immediate and atomic:
housectl domain create new-domain
Once this returns, the domain exists and can be used immediately.
Changing superior domain relationships affects policy evaluation immediately:
# Add global as a superior for engineering
housectl domain add-superior engineering global
Impact:
Risk: Changing domain hierarchies can dramatically change authorization behavior. Test thoroughly before updating production hierarchies.
Domain deletion is permanent and immediate:
housectl domain delete old-domain --yes
What gets deleted:
What is NOT deleted:
Effect on authorization:
Recommendation: Before deleting a domain:
housectl domain list and review superior relationshipsUser creation is immediate:
housectl user create alice Secret123! alice@example.com
Security note: Passing passwords as command-line arguments exposes them in shell history and process listings. In sensitive environments, disable shell history before running this command, or use a secrets manager to supply the value.
Alice can authenticate immediately after this returns.
User attribute updates (email, username, active status) are immediate:
housectl user update alice --email newalice@example.com
However, there's an important caveat: JWT tokens.
Housecarl uses JWT tokens for authentication. User attributes are embedded in the JWT when the user logs in:
{
"user_id": "550e8400-e29b-41d4-a716-446655440000",
"username": "alice",
"tenant_id": "660e8400-e29b-41d4-a716-446655440001",
"email": "alice@example.com",
"exp": 1704153600
}
The problem: JWTs are stateless. Once issued, they're valid until they expire (typically 12 hours by default).
Scenario:
14:00 - Alice logs in, gets JWT token with email "alice@example.com"
14:30 - Admin updates Alice's email to "newalice@example.com"
15:00 - Alice makes an authz request using her JWT token
- JWT still says "alice@example.com" (old email)
- Database says "newalice@example.com" (new email)
Impact:
user_id, not email, so they're unaffectedemail attribute, they'll see the old email until token expiresSolution: After updating user attributes that policies might match on:
# Option 1: Ask user to log out and log back in
# New JWT will have updated attributes
# Option 2: Wait for token expiration (default: 12 hours)
# Option 3: Invalidate user's sessions (if supported)
# Force re-authentication
Best practice: Design policies to match on stable identifiers (user_id, tenant_id) rather than mutable attributes (email, username).
Password changes take effect immediately:
housectl user change-password alice NewSecret456!
Security note: Passing passwords as command-line arguments exposes them in shell history and process listings. In sensitive environments, disable shell history before running this command, or use a secrets manager to supply the value.
Important: Existing JWT tokens remain valid. Changing a password does NOT invalidate active sessions.
If you need to force re-authentication (e.g., suspected compromise):
Setting a user to inactive:
housectl user update alice --active false
Effect:
Recommendation: For immediate effect, implement token revocation checking in your application, or wait for token expiration.
Tenant creation is immediate and atomic:
housectl admin create "New Corp" "Description"
What gets created:
All created atomically - either all succeed or none do.
Tenant name, description, subscription updates are immediate:
housectl tenant update new-corp --name "New Corporation Inc" --description "Updated description"
No impact on authorization decisions (metadata only).
Disabling a tenant:
housectl admin disable tenant-id
Effect:
Immediate - takes effect on the next request after the disable operation completes.
Use case: Suspend a customer for non-payment, security incident, or account closure.
housectl admin enable tenant-id
Immediate - tenant becomes active again, authorization resumes normally.
housectl tenant associate-user acme-corp alice
Effect:
housectl tenant disassociate-user acme-corp alice
Effect:
Recommendation: After disassociating, wait for JWT expiration or implement token revocation.
Admin A: Updates policy in engineering domain at 14:00:00.100
Admin A: Immediately queries engineering domain at 14:00:00.200
Result: Admin A sees the updated policy
Housecarl guarantees read-your-writes consistency. After a write completes, subsequent reads by the same or different users see the updated state.
Time 0: Policy allows alice to read documents
Time 1: Admin updates policy to deny alice
Time 2: Alice request A (started at time 1.5, finishes at time 3)
Time 3: Alice request B (started at time 2.5, finishes at time 4)
Result:
Guarantee: No request sees an inconsistent mix of old and new policies. Each request sees a consistent snapshot.
Admin A: Updates engineering domain at time 1
Admin B: Updates product domain at time 1
Authorization request: Needs policies from both domains
Guarantee: The request sees a consistent snapshot of the policy database. Either:
Never: Sees a partial update of one domain (ACID atomicity guarantees this).
After making updates, monitor these metrics:
# Watch for unexpected spikes
# (Assumes you have metrics/observability set up)
# Manual testing:
housectl authz can-i test-request.json
An unexpected increase in denies might indicate:
Check your application's authorization error logs:
2024-01-15 14:05:23 [WARN] Authorization denied for alice: read /documents/project-alpha/spec.md
Spike in authorization denials after a policy update indicates potential issues.
Policy updates should not significantly impact authorization request latency. If latency spikes after an update:
Before updating policies in production:
housectl authz parse-policieshousectl authz can-i-local for critical pathsBefore updating domain hierarchies:
Before deactivating users or tenants:
Policy updates are fast for typical workloads. Time scales with the number of policies being deployed and includes:
Authorization requests during a policy update:
Housecarl uses PostgreSQL row-level locking:
Implication: Multiple admins can update different domains simultaneously with no contention.
Issue: User attributes in JWTs don't update until token expires.
Solution: Design policies to match on stable IDs, not mutable attributes.
Issue: Deleting a superior domain doesn't delete child domains; they lose inherited policies.
Solution: Before deleting, reassign child domains to a new superior or delete them explicitly.
Issue: Concurrent updates to the same domain result in last-write-wins. No merge.
Solution: Use version control, coordinate with team, or implement optimistic locking in your workflow.
Issue: Policy updates are all-or-nothing, no gradual rollout built-in.
Solution: For critical policy changes, use domain hierarchy:
Issue: A deny policy overrides all allows. If you add a deny policy by mistake, all access is blocked immediately.
Solution:
| Operation | Takes Effect | Notes |
|---|---|---|
| Create policy | Immediately | Atomic at domain level |
| Update policy | Immediately | In-flight requests use old policy |
| Delete policy | Immediately | Affected requests denied |
| Create domain | Immediately | Can be used in authz requests |
| Update domain hierarchy | Immediately | In-flight requests use old hierarchy |
| Delete domain | Immediately | Orphans resources and child domains |
| Create user | Immediately | Can authenticate immediately |
| Update user attributes | Immediately for DB | JWTs lag until expiration |
| Change password | Immediately for new logins | Existing JWTs still valid |
| Deactivate user | Immediately for new logins | Existing JWTs still valid until expiration |
| Create tenant | Immediately | Atomic creation |
| Disable tenant | Immediately | All requests denied |
| Associate user to tenant | Immediately for new logins | Existing JWTs unchanged |
| Disassociate user | Immediately for new logins | Existing JWTs valid until expiration |
Housecarl provides ACID guarantees for updates:
No distributed transaction coordination required - each operation is atomic within PostgreSQL.
Key Takeaway: Housecarl updates are immediate and ACID-compliant, but JWT token lag is the primary exception. Design policies around stable identifiers and implement token revocation if you need immediate effect for user changes.