Troubleshooting

This guide helps diagnose and resolve common issues when deploying and integrating Rafiki with your digital wallet.

Pre-deployment checklist

Before troubleshooting issues, ensure you’ve completed all required customizations:

Required customizations

File/Location	Variables to Update
terraform.tfvars	project_id, domain_name, region
cluster-issuer.yaml	email field
argocd/ingress.yaml	host fields
ingress-nginx/values.yaml	loadBalancerIP
rafiki/values.yaml	All YOUR_DOMAIN.com references
wallet/values.yaml	YOUR_DOMAIN.com, YOUR_REGISTRY
monitoring/values.yaml	Domain references, passwords, SMTP
backup/postgres-backup.yaml	GCS bucket, project ID
Environment secrets	Database passwords, API tokens
DNS records	Point domains to static IP

Security checklist

Security Item	Description
TLS Certificates	Let’s Encrypt configured for all domains
Database Passwords	Strong, randomly generated passwords
API Secrets	32-byte secrets for auth and webhooks
Network Policies	Enabled to restrict pod-to-pod communication
RBAC	Proper service accounts and permissions
Image Security	Using official images with known vulnerabilities patched
Backup Encryption	KMS encryption for backup data

Common issues and solutions

Infrastructure issues

Issue: Error creating cluster

Solutions:

1. Check GCP permissions:

   # Verify current user has required permissions
   gcloud auth list
   gcloud projects get-iam-policy PROJECT_ID
   
   # Add required roles
   gcloud projects add-iam-policy-binding PROJECT_ID \
     --member="user:your-email@domain.com" \
     --role="roles/container.admin"

2. Enable required APIs:

   gcloud services enable container.googleapis.com
   gcloud services enable compute.googleapis.com
   gcloud services enable dns.googleapis.com

3. Check quota limits:

   gcloud compute project-info describe --project=PROJECT_ID

Issue: GKE cluster creation hangs

Symptoms:

• Terraform hangs on cluster creation
• Cluster shows “PROVISIONING” status for extended time

Solutions:

1. Check region availability:

   # List available zones in region
   gcloud compute zones list --filter="region:us-central1"
   
   # Try different region
   terraform apply -var="region=us-east1"

2. Reduce initial node count:

   terraform apply -var="min_node_count=1"

3. Check for resource conflicts:

   # List existing clusters
   gcloud container clusters list
   
   # Clean up if needed
   gcloud container clusters delete OLD_CLUSTER_NAME --region=REGION

DNS and certificate issues

Issue: certificate not issued

Symptoms:

• Certificate shows READY: False status
• TLS certificate secret not created
• HTTPS connections fail with certificate errors
• Let’s Encrypt certificate challenges failing

kubectl get certificates -A
NAME               READY   SECRET           AGE
rafiki-auth-tls    False   rafiki-auth-tls  10m

Solutions:

1. Verify DNS records are correctly configured:

   # Check if domain resolves to your cluster IP
   nslookup auth.YOUR_DOMAIN.com
   dig auth.YOUR_DOMAIN.com
   
   # Verify static IP is assigned
   kubectl get ingress -A

2. Check cert-manager is running properly:

   # Verify cert-manager pods are healthy
   kubectl get pods -n cert-manager
   
   # Check cluster-issuer status
   kubectl get clusterissuer
   kubectl describe clusterissuer letsencrypt-prod

3 . Check firewall rules allow HTTP/HTTPS traffic:

# Ensure HTTP challenge can reach your domain
gcloud compute firewall-rules list
curl -I http://auth.YOUR_DOMAIN.com/.well-known/acme-challenge/test

Debugging:

# Check certificate status
kubectl describe certificate rafiki-auth-tls -n rafiki

# Check certificate request
kubectl get certificaterequests -A
kubectl describe certificaterequest <name> -n rafiki

# Check cert-manager logs
kubectl logs -n cert-manager deployment/cert-manager
kubectl logs -n cert-manager deployment/cert-manager-webhook
kubectl logs -n cert-manager deployment/cert-manager-cainjector

# Check ingress-nginx logs for HTTP challenges
kubectl logs -n ingress-nginx deployment/ingress-nginx-controller