A.M.P. Troubleshooting Guide
When (not if) something goes wrong — Ala Malū Privacy Field Reference
Page 1 of 3 · Revised 2026-06-14
Quick Reference — Key Addresses & Log Files
Blue Triangle (Cloud Hub)
# SSH into Blue Triangle ssh root@100.113.114.32 # Check all running containers docker ps # Tail dashboard API logs docker logs -f amp-dashboard # Tail Telegram bridge logs docker logs -f amp-bridge # Bridge health check curl http://100.113.114.32:5001
Spoke Nodes (Hawaii Pis)
# SSH via Tailscale (Leonard only) ssh pi@<tailscale-ip-of-spoke> # or use: tailscale ssh pi@<hostname> # Check serial cat /etc/amp-serial # Check heartbeat log tail -20 /var/log/amp-heartbeat.log # Check first-boot log tail -30 /var/log/amp-firstboot.log # Tailscale status sudo tailscale status
NodeLocationNode IDTailscale TagFind Tailscale IP
Blue TriangleDigitalOcean NYC1N/Atag:blue-triangle100.113.114.32 (fixed)
Node 20Leonard · Kent HQ (Founder)20tag:exit-nodetailscale status on BT or laptop
Node 21Mom · Waimea21tag:exit-nodetailscale status on BT or laptop
Node 22Charla · Nanakuli22tag:exit-nodetailscale status on BT or laptop
Node 23Tammy · Waianae23tag:exit-nodetailscale status on BT or laptop

1 — Spoke Node Issues (Hawaii Pis)

🔴Telegram alert received: "Node Offline" — node hasn't checked in for 10+ minutes
  1. Check if it's a power issue — ask the family member if the Pi's LEDs are on. Green LED = running; no lights = unplugged or dead power supply.
  2. Check Tailscale from your laptop — if the node is still in the mesh, it has internet:
tailscale status # Look for the node's hostname (AMP...). If it shows "offline" → internet/Pi issue
  1. If Tailscale shows online but heartbeat is still missing — SSH in and check the cron:
ssh pi@<tailscale-ip> crontab -l # Confirm "*/5 * * * * /usr/local/bin/amp-heartbeat.sh" is present cat /var/log/amp-heartbeat.log | tail -20 # Look for errors sudo /usr/local/bin/amp-heartbeat.sh # Run manually to test
  1. If the node is truly unreachable — have the family member unplug it, wait 15 seconds, plug back in. The heartbeat cron will resume automatically. Alert will auto-clear when BT receives the next heartbeat.

Alert auto-clears when heartbeat resumes. No manual database cleanup needed — the alert.py script handles recovery detection automatically.

🔑Can't SSH into a spoke node
  1. Confirm you're on a Tailscale-connected network (your laptop must be on the mesh).
  2. Find the node's Tailscale IP: tailscale status — look for the AMP hostname.
  3. Try Tailscale SSH directly: tailscale ssh pi@AMPxxxxxxxx
  4. If that fails, check Tailscale service on the spoke: ask family to reboot the Pi, which restarts tailscaled automatically.
  5. If still unreachable — the Pi is likely powered off or lost internet. See "Node Offline" scenario above.

Password auth is disabled. Only SSH keys work (Leonard's key + BT deploy key). Never share SSH passwords — there are none.

🌐Tailscale is not connected on the spoke
  1. SSH in via local network if possible (connect to family's WiFi) or ask them to hold the Pi near the router.
  2. Check Tailscale status and logs:
sudo systemctl status tailscaled sudo tailscale status sudo journalctl -u tailscaled -n 50
  1. If "needs login" or "not authorized" — the auth key expired. Generate a new reusable auth key at tailscale.com/admin/settings/keys and re-authenticate:
sudo tailscale up \ --authkey=tskey-auth-xxxxx \ --advertise-exit-node \ --advertise-tags=tag:exit-node
  1. If tailscaled won't start: sudo systemctl restart tailscaled — then re-run step 3.
⏰Heartbeat cron is not sending (log is old or missing)
  1. Check if the cron entry exists: crontab -l (should show */5 * * * * /usr/local/bin/amp-heartbeat.sh)
  2. Check if cron service is running: sudo systemctl status cron
  3. Test the heartbeat script manually:
sudo /usr/local/bin/amp-heartbeat.sh # Check what it outputs — look for WAN IP or error messages
  1. If /etc/amp-serial is missing or empty, the heartbeat can't identify itself. Re-run first-boot if needed (see below).
  2. Reinstall cron entry if missing:
(crontab -l 2>/dev/null; echo "*/5 * * * * /usr/local/bin/amp-heartbeat.sh") | crontab -
A.M.P. Troubleshooting Guide
First-Boot Problems · Blue Triangle · Telegram Bridge · Registration Conflicts
Page 2 of 3 · Revised 2026-06-14

2 — First-Boot Problems

🥾First boot didn't complete — node has no serial, no Tailscale, no heartbeat
  1. SSH in locally (connect to family's WiFi, or use default pi/raspberry credentials if it's a fresh image). Check the log:
cat /var/log/amp-firstboot.log # Look for the last successful line and where it stopped
  1. Check if the guard file exists — if it does, the script thinks it already ran:
ls -la /etc/amp-firstboot-done # Exists = script won't re-run
  1. If the script stopped at Tailscale setup — the auth key file was missing or wrong:
ls /boot/amp-tailscale.key # Must exist with a valid key # If missing, create the key file: echo "tskey-auth-xxxxx" | sudo tee /boot/amp-tailscale.key
  1. To re-run first-boot completely (resets everything):
sudo rm /etc/amp-firstboot-done # Remove guard flag sudo bash /usr/local/bin/first-boot.sh # Re-run (or reboot)

Removing the guard file and rebooting is safe for a fresh Pi — it won't create a duplicate serial if the MAC isn't registered yet. If it IS already registered, the API returns 200 OK (idempotent).

💥Registration returned 409 CONFLICT — serial collision detected
  1. This means the MAC address was already registered under a different serial — usually because the SD card was re-flashed and the Pi booted fresh, generating a new salt.
  2. Check the audit_logs table on Blue Triangle:
ssh root@100.113.114.32 docker exec -it amp-db psql -U amp -d ampdb -c \ "SELECT * FROM audit_logs WHERE hardware_mac = '<MAC>' ORDER BY timestamp DESC LIMIT 10;"
  1. Decide: do you want to keep the old serial or accept the new one?
    • Keep old serial: Delete the new record from gateways and reinstall the original serial file on the Pi: echo "AMPXXXXXXXX" | sudo tee /etc/amp-serial
    • Accept new serial: Update the gateways row with the new serial. Update /etc/amp-serial on the Pi to match.

3 — Blue Triangle (Cloud Hub) Issues

🔴Blue Triangle API unreachable — spokes can't register or send heartbeats
  1. SSH into BT (you must be on Tailscale): ssh root@100.113.114.32
  2. If SSH fails, BT may be down. Open DigitalOcean console in browser → Power → Power cycle the droplet.
  3. Once SSH works, check Docker containers:
docker ps -a # All containers should show "Up". If any show "Exited": docker start amp-dashboard docker start amp-db # Or restart all: cd /opt/amp && docker compose up -d
  1. Check if the API is responding locally on BT:
curl http://127.0.0.1:5000/api/health # Should return {"status":"ok"} — if not, check amp-dashboard logs docker logs --tail=50 amp-dashboard
  1. Check Tailscale on BT is still connected: tailscale status
🐳Docker container is down or crashing in a restart loop
docker ps -a # Check exit codes and status docker logs --tail=100 <container> # Read the error docker inspect <container> | grep -A5 '"State"' # Check restart count
cd /opt/amp && docker compose down && docker compose up -d # Full restart
🌊Telegram war room flooded with hundreds of tailscale_restart / reboot alerts after BT recovery

This happens when BT has been offline for hours — pukas queue events locally and flush everything at once when BT comes back, overwhelming the API's 2 Gunicorn workers.

  1. Snooze puka alerts immediately (works via aikai over LAN, no Tailscale needed):
ssh -i ~/.ssh/id_ed25519 bullaz@10.0.0.105 python3 -c "import json,time; open('/tmp/amp-snooze.json','w').write(json.dumps({'snoozed':True,'until_ts':int(time.time())+7200,'until':'2h from now'}))" # Or type /snooze 120 in the war room if bridge is responsive
  1. Block Telegram outbound from BT to stop the spam immediately:
ssh root@100.113.114.32 iptables -I FORWARD -d 149.154.160.0/20 -j DROP iptables -I FORWARD -d 91.108.4.0/22 -j DROP
  1. Clear event queues on all pukas to stop new events being sent:
for IP in 100.110.69.51 100.116.250.85 100.86.244.86 100.73.221.51; do ssh -i ~/.ssh/amp-deploy pi@$IP "sudo rm -f /tmp/amp-events-queue" done
  1. Wait for API to clear (poll for HTTP 200), then unblock Telegram:
until curl -s --max-time 5 http://172.19.0.3:5000/api/gateways -o /dev/null -w '%{http_code}' | grep -q 200; do sleep 10; done iptables -D FORWARD -d 149.154.160.0/20 -j DROP iptables -D FORWARD -d 91.108.4.0/22 -j DROP

If puka processes are stuck mid-flush (SSH to pukas also timing out), wait it out — they will drain naturally in 15–30 min. The event queue is finite.

📊Hōʻike (hoike.alamalu.com:5000) returns connection timeout / HTTP 000
  1. Check if the API is responding at all:
ssh root@100.113.114.32 curl -s --max-time 8 http://172.19.0.3:5000/api/gateways -o /dev/null -w '%{http_code}' # HTTP 200 = API fine, DNS/proxy issue. HTTP 000 = API overwhelmed or down.
  1. If API is overwhelmed (usually after event flood) — check logs and restart:
docker logs amp-api --tail=20 # Look for "WORKER TIMEOUT" or OOM errors cd /opt/amp/blue-triangle && docker compose restart api sleep 15 && curl -s --max-time 8 http://172.19.0.3:5000/api/gateways -o /dev/null -w '%{http_code}'
  1. If Docker containers can't reach internet after any iptables flush — rebuild Docker NAT rules:
systemctl restart docker # Rebuilds iptables NAT — containers can reach internet again # Verify: docker exec amp-api python3 -c "import urllib.request; print(urllib.request.urlopen('https://api.telegram.org',timeout=5).status)"
  1. If Hōʻike loads but shows stale data — check heartbeats are flowing:
docker exec amp-db psql -U amp -d amp -c "SELECT node_id, last_telemetry_seen FROM gateways ORDER BY node_id"
🔗wg_aikai failover tunnel won't establish handshake with aikai
  1. Check BT's wg_aikai config has Table = off (CRITICAL — without it, all BT outbound traffic routes through aikai, breaking internet for all Docker containers):
cat /etc/wireguard/wg_aikai.conf | grep -E 'Table|Address|ListenPort' # Must show: Table = off # If missing: wg-quick down wg_aikai, add "Table = off" under [Interface], wg-quick up wg_aikai
  1. Verify aikai's endpoint uses BT's PUBLIC IP (not Tailscale) — Tailscale ACLs block UDP 51845:
ssh -i ~/.ssh/id_ed25519 bullaz@10.0.0.105 sudo grep Endpoint /etc/wireguard/wg_failover.conf # Must show: Endpoint = 159.65.218.220:51845 (NOT 100.113.114.32) # Fix: sudo sed -i 's/100.113.114.32/159.65.218.220/' /etc/wireguard/wg_failover.conf sudo systemctl restart wg-quick@wg_failover
  1. Verify UDP 51845 is open in DO cloud firewall (Networking → Firewalls → amp-no-front-door).
  2. Check handshake on both sides:
wg show wg_aikai latest-handshakes # On BT — should be < 180 seconds ago ssh bullaz@10.0.0.105 "sudo wg show wg_failover latest-handshakes" # On aikai

4 — Telegram Bridge Not Responding (Ala-C / Ala-G silent)

✈Sent a Telegram message and got no response from Ala-C or Ala-G
  1. Quick health check — if this returns anything other than {"status":"up"}, the bridge is down:
curl http://100.113.114.32:5001 # Also check from BT itself to rule out network: ssh root@100.113.114.32 "curl http://127.0.0.1:5001"
  1. Check bridge container status and restart it:
ssh root@100.113.114.32 docker ps | grep bridge # Is it running? docker logs --tail=50 amp-bridge # Look for errors docker restart amp-bridge # Restart it
  1. If the bridge logs show "403 Forbidden" from Telegram — a bot token has expired or been revoked. Generate a new token in BotFather and update /opt/amp/.env, then restart.
  2. If the bridge logs show Anthropic API errors — check your API key balance/status at console.anthropic.com
  3. If the bridge is running but only ONE bot responds — one bot token is bad. Check which one is failing in the logs and fix that token.

/clear in the group chat resets the in-memory conversation history. Useful if the bots are giving confused or looping responses.

A.M.P. Troubleshooting Guide
Client Device Issues · Nessus · Subscriber Portal · Nuclear Recovery
Page 3 of 3 · Revised 2026-06-14

5 — Family Device / Subscriber Issues

📱Family member's device isn't routing through the exit node — no privacy protection
  1. On their phone/laptop — open the Tailscale app. Check that:
    • Tailscale is connected (toggle is ON)
    • "Use exit node" is selected, and their home Pi is chosen (e.g., AMP56F020949F)
  2. Confirm the Pi itself is advertising as an exit node:
ssh pi@<spoke-tailscale-ip> sudo tailscale status # Should show: offers exit node # If not advertising, re-run: sudo tailscale up --advertise-exit-node --advertise-tags=tag:exit-node
  1. On the Tailscale admin console (tailscale.com/admin) — confirm the exit node is "Approved". New exit node advertisements must be manually approved.
  2. Verify internet is routing through the Pi: have the family member visit api.ipify.org — their IP should match the Pi's WAN IP shown in the dashboard.
🐢Internet is slow when connected through the exit node
  1. Check the Pi's load and temperature (they're fanless — heat causes throttling):
ssh pi@<spoke-tailscale-ip> vcgencmd measure_temp # Pi temperature — should be under 65°C top -bn1 | head -5 # CPU load — should be under 60% cat /proc/net/dev # Network throughput counters
  1. If temperature is high (>75°C) — Pi is throttling. Check that the Argon NEO case heatsink is properly seated and the unit has airflow around it. Avoid placing in enclosed spaces.
  2. Test the Pi's own internet speed (eliminates the home ISP as the bottleneck):
curl -s https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py | python3 -
  1. If Pi speed is fine but VPN is slow — check Tailscale relay vs. direct connection:
sudo tailscale ping <client-tailscale-ip> # "via DERP" = relayed (slow). "direct" = direct UDP (fast) # Direct should form within 30 seconds after both sides are connected
🌐Can't reach puka.alamalu.com or setup.alamalu.com
  1. Check NPM (Nginx Proxy Manager) is running on BT: docker ps | grep proxy
  2. Check NPM proxy host configuration at http://100.113.114.32:81 (NPM admin UI)
  3. Check DNS: nslookup puka.alamalu.com — should point to Blue Triangle's public IP
  4. Check SSL cert status in NPM — expired certs will show certificate errors. Use NPM to renew Let's Encrypt certs.
ssh root@100.113.114.32 docker logs --tail=30 amp-proxy # Look for proxy errors docker restart amp-proxy # Restart if necessary

6 — Nessus Scan Issues

🔬Nessus scan fails to reach the Pi or returns incomplete results
  1. Nessus runs on Blue Triangle at https://100.113.114.32:8834 — must be on Tailscale to access.
  2. The scan target is the Pi's local IP (on Kent Lab network during burn-in). Ensure Pi is on the same network as BT's scan range or use Tailscale IP as target.
  3. If Nessus says "Host unreachable" — check that the Pi is on (LEDs), has internet, and you can ping it from BT:
ssh root@100.113.114.32 ping <pi-ip-or-tailscale-ip> -c 3
  1. If Nessus service itself is down on BT:
ssh root@100.113.114.32 sudo systemctl status nessusd sudo systemctl start nessusd # Nessus runs as host service, NOT Docker

7 — Nuclear Recovery Procedures

☢Blue Triangle droplet is completely unreachable — cannot SSH, Tailscale shows offline
  1. Step 1 — netplan fix (try this first). Most likely cause after account suspension/reactivation: netplan vs. legacy ifupdown conflict. Use DO web console (Actions → Web Console) and run:
netplan --debug generate netplan --debug apply # Restores inbound connectivity immediately. No reboot needed.
  1. If web console shows "timed out waiting for handshake" — file a DO support ticket: "SSH listening on 0.0.0.0:22, no firewall, netplan apply run, connections still time out. Please reset backend networking."
  2. SSH key check — after restoring connectivity, verify ~/.ssh/authorized_keys contains the amp-deploy key. If missing, add it (or enable password auth temporarily):
cat ~/.ssh/authorized_keys # Should contain: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIA92/nhiyYQqT16MEumPW+3mRMXtLZXCNqBOdxvYS8S6 amp-deploy # If missing, append it: echo "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIA92/nhiyYQqT16MEumPW+3mRMXtLZXCNqBOdxvYS8S6 amp-deploy" >> ~/.ssh/authorized_keys # Temporary password auth (remove after use): echo 'PasswordAuthentication yes' > /etc/ssh/sshd_config.d/70-enable-password.conf && systemctl restart ssh
  1. After reboot, Tailscale should auto-reconnect. If auth expired:
tailscale up --authkey=tskey-auth-xxxxx --advertise-tags=tag:blue-triangle --ssh
  1. Once SSH works — restart Docker stack and verify:
cd /opt/amp/blue-triangle && docker compose up -d docker ps # All 5 containers should show "Up" # If containers can't reach internet after iptables changes: systemctl restart docker
  1. If the entire droplet is gone — rebuild from scratch using DR/README.md in the repo (full rebuild runbook including event flood warning).

Pukas stay operational even when BT is offline — they use direct Telegram alerts as fallback. Family internet access is unaffected. Use /snooze 120 in the war room to silence alert noise during recovery.

💾Need to rebuild a spoke node from scratch (SD card corrupted / Pi replaced)
  1. Flash a fresh Raspberry Pi OS Bullseye image to a new SD card.
  2. Add to /boot: empty ssh file + amp-tailscale.key with a valid reusable auth key.
  3. Ensure first-boot.sh is installed as systemd one-shot service in the image.
  4. Boot Pi — first-boot runs automatically. It will re-register with BT API using the new MAC.
    • If same MAC as before: gets same serial back (idempotent) ✓
    • If new hardware (different MAC): gets new serial, new node_id — update gateways table to reassign to correct subscriber
  5. Check the first-boot log to confirm success: tail -30 /var/log/amp-firstboot.log

8 — Symptom Quick-Lookup

What You SeeMost Likely CauseFirst Command to RunPage
🔴 Telegram "Node Offline" alert Pi unplugged or lost internet tailscale status on laptop P.1
Telegram bots not responding Bridge container down curl http://100.113.114.32:5001 P.2
SSH to spoke refused Tailscale disconnected on spoke tailscale status on laptop P.1
Heartbeat log is stale Cron not running or missing crontab -l on spoke P.1
First boot log stops mid-way Missing Tailscale key file ls /boot/amp-tailscale.key P.2
409 CONFLICT at registration SD re-flashed, MAC already registered Check audit_logs table in Postgres P.2
Docker containers showing Exited BT rebooted without auto-start set docker compose up -d on BT P.2
Internet slow through exit node Pi thermal throttle or DERP relay vcgencmd measure_temp on spoke P.3
Exit node not showing in Tailscale app Not approved in Tailscale admin Approve at tailscale.com/admin P.3
puka.alamalu.com unreachable NPM down or SSL cert expired docker ps | grep proxy on BT P.3
Blue Triangle SSH unreachable Tailscale down on BT or droplet frozen Use DigitalOcean web console P.3
Telegram flooded with 100s of alerts after BT recovery Pukas flushing 11-day backlog of queued events /snooze 120 in war room, then clear puka queues P.2
Hōʻike port 5000 times out / HTTP 000 Gunicorn workers overwhelmed by event flood docker compose restart api on BT P.2
All BT outbound broken after wg_aikai up wg_aikai missing Table = off — hijacks default route wg-quick down wg_aikai, add Table = off, bring back up P.2
wg_aikai handshake stuck at 0 (never connects) aikai endpoint uses Tailscale IP — blocked by ACLs Change endpoint to 159.65.218.220:51845 in wg_failover.conf P.2
Docker containers can't reach internet iptables/nftables flushed — Docker NAT rules wiped systemctl restart docker on BT P.2
Nessus won't start nessusd service not running sudo systemctl start nessusd P.3
Required Environment Variables — /opt/amp/.env on Blue Triangle
VariableUsed ByWhere to Get / Check
DATABASE_URLDashboard API, Alert scriptPostgres connection string on BT: postgresql://amp:<pass>@amp-db/ampdb
TELEGRAM_CLAUDE_TOKENAlert script, Telegram bridge (Ala-C bot)BotFather → Ala-C bot token
TELEGRAM_GEMINI_TOKENTelegram bridge (Ala-G bot)BotFather → Ala-G bot token
TELEGRAM_GROUP_CHAT_IDAlert script, Telegram bridgeWar Room group chat ID (negative number)
ANTHROPIC_API_KEYTelegram bridgeconsole.anthropic.com → API Keys

If any key is missing or wrong, the container will crash on startup. Check with: docker logs amp-bridge --tail=20 for KeyError or authentication error messages.