PACI Kuwait proxy (geo-bypass)

PACI (kuwaitportal.paci.gov.kw) and Baladia (gismaps.baladia.gov.kw) geo-block non-Kuwait IPs — especially cloud/datacenter ranges. That breaks two things when the app runs from outside Kuwait:

1. The paci-proxy edge function's server-side calls to PACI (Supabase functions run on a global, non-KW network).
2. Client-side PACI vector-tile / Baladia overlay fetches for any user not on a Kuwaiti residential IP.

The fix is a tiny reverse proxy hosted on a Kuwaiti VPS that relays requests to PACI/Baladia from a Kuwaiti IP. As a bonus it adds permissive CORS, which also resolves the web PACI-CORS limitation (F-007 in docs/MAPS.md).

app / edge fn  ──►  Cloudflare (HTTPS + tile cache)  ──►  KW VPS (nginx)  ──►  PACI / Baladia
                     proxied DNS record                    docs/infra/        kuwaitportal.paci.gov.kw
                     paci-kw.<domain>                       cloud-init         gismaps.baladia.gov.kw

1. Provision the VPS

Provider: LightNode, Kuwait region (any KW VPS works). Cheapest tier is plenty — this is a network-I/O relay, not compute:

Field	Value
Type	CPU Shared
Location	Kuwait
Image	Ubuntu 24.04 LTS (System Image)
Instance	1 vCPU / 2 GB
Network	smallest tier (1 Mbps) or Pay-By-Traffic — Cloudflare absorbs repeat-tile bandwidth
Storage	50 GB system disk
Auth	SSH key
Preinstall	none (do NOT install BT-Panel — it would overwrite Custom Data)

LightNode's Custom Data has two tabs — use whichever matches the script:

• user-data (shell) → paste paci-kw-proxy.cloud-init.sh and set SECRET (top of the script).
• cloud-config → paste paci-kw-proxy.cloud-config.yaml and replace __PROXY_SECRET__ (5 occurrences). A #!/bin/bash script pasted on this tab fails with "must start with #cloud-config" — that's the wrong tab.

Generate the secret with:

openssl rand -hex 24

2. Public HTTPS — Cloudflare Tunnel (required; do NOT expose port 80)

LightNode's network resets plaintext HTTP on port 80. A middlebox on the (China-transiting) backhaul RST-injects HTTP requests — verified from inside Kuwait too. SSH and TLS survive because they're encrypted; plain http:// does not. So a normal proxied DNS A-record → VPS:80 does not work. Use a Cloudflare Tunnel — an outbound encrypted connection from the box, so no inbound port is exposed and the middlebox can't touch it. (This is what's deployed.)

1. Cloudflare Zero Trust → Networks → Tunnels → Create a tunnel (cloudflared), name paci-kw; copy the install token.
2. On the box:

   curl -fsSL https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64.deb -o /tmp/cf.deb
   dpkg -i /tmp/cf.deb
   cloudflared service install <TOKEN>   # installs + starts the systemd service

3. In the tunnel, add a Public Hostname: paci-kw.aldilaijan.com → HTTP → localhost:80. Cloudflare auto-creates the proxied DNS + HTTPS cert.

Recommended cache rule: responses are application/octet-stream (.pbf) / json, which Cloudflare won't edge-cache by default (Cf-Cache-Status: DYNAMIC). Add a Cache Rule for paci-kw.aldilaijan.com → Eligible for cache, Edge TTL ~7d, so repeat tiles serve from Cloudflare's edge instead of consuming VPS tunnel bandwidth. (nginx already caches the VPS→PACI hop; this caches the client→VPS hop.)

Two gotchas that silently keep it DYNAMIC:

• The rule's match must be on the host: (http.host eq "paci-kw.aldilaijan.com"). A URI Full eq "paci-kw.aldilaijan.com" matcher never matches the real URL (which is https://paci-kw.aldilaijan.com/<secret>/…) → nothing caches.
• PACI sends Set-Cookie (AGS_ROLES, TS*) and Vary: Origin on tiles; Cloudflare won't cache a response with Set-Cookie or a non-Accept-EncodingVary. The nginx snippet strips both (proxy_hide_header Set-Cookie; Vary;) — required for the cache rule to take effect. Verified MISS→HIT once both the expression and the header-strip are in place.

Caching layers (perf). This edge rule caches the client→VPS hop; nginx caches the VPS→PACI hop; and the app caches the PACI style document itself (root.json) in-memory + SharedPreferences so repeat map opens don't re-fetch it (see warmPaciBasemapStyle / loadPaciKfBasemapRootJson in lib/shared/maps/paci_style_loader.dart and the "Caching" section of docs/MAPS.md). A larger structural win — self-hosting the PACI basemap tileset as PMTiles on Cloudflare R2 + a Worker, dropping this proxy round trip for the basemap entirely — is scoped there as a paid follow-up (check PACI redistribution/attribution first).

3. Verify (from any machine)

curl https://paci-kw.<domain>/healthz
# -> ok

curl -I "https://paci-kw.<domain>/<SECRET>/paci/arcgisportal/rest/services/Hosted/PACIKFBasemap/VectorTileServer?f=json"
# -> 200, plus `X-Cache-Status: MISS` then `HIT` on a 2nd call.
# A 200 here proves the Kuwait IP reaches PACI.

3b. Kuwait Finder API (/kfapp/) — needed by Hermes resolve_paci_link

kfappsrv.paci.gov.kw (the Kuwait Finder backend behind gis.paci.gov.kw/?id=<UUID> share links) is geo-blocked from the Hermes VPS the same way the ArcGIS portal is: direct calls hang until the timeout, and the WhatsApp copilot then tells the customer "PACI timed out". It is not covered by the /paci/ route, so it has its own /kfapp/ location.

Fresh installs get it from the cloud-init / cloud-config files. On the already-deployed proxy, add the location by hand (the placeholder was already replaced with the real secret):

sudo nano /etc/nginx/sites-available/paci-proxy.conf   # add inside the server { } block:
#   location ^~ /<SECRET>/kfapp/ {
#       rewrite ^/<SECRET>/kfapp/(.*)$ /$1 break;
#       proxy_pass       https://kfappsrv.paci.gov.kw;
#       proxy_set_header Host kfappsrv.paci.gov.kw;
#       proxy_ssl_name   kfappsrv.paci.gov.kw;
#       include          snippets/paci-common.conf;
#       proxy_cache_bypass 1;   # NOT `proxy_cache off` — the snippet already sets
#       proxy_no_cache     1;   # proxy_cache, so that is a duplicate-directive error
#   }
sudo nginx -t && sudo systemctl reload nginx

Applied to the live proxy (38.54.124.114) on 2026-10-03 — backup at /root/paci-proxy.conf.bak-kfapp-<ts> (kept outside sites-*/, which nginx globs). Verified from the box (/kfapp/…searchdocument → 200 with cadastral JSON) and from inside the hermes-server container via PACI_PROXY_BASE.

Then make sure hermes-server has PACI_PROXY_BASE=https://paci-kw.<domain>/<SECRET> in its environment (same value as the Flutter build secret). kfFetch (paciKfFetch.ts) then tries the proxy first and falls back to a direct call, with bounded retries and an overall 15 s budget. With PACI_PROXY_BASE unset it behaves as before (direct).

Verify the route end to end from the VPS (an HTTP 200 with JSON, not a hang):

curl -s -o /dev/null -w "%{http_code} %{time_total}s\n" \
  "https://paci-kw.<domain>/<SECRET>/kfapp/kuwaitfinder/server/api/search/searchdocument?params=x"
# -> a fast HTTP response from PACI (even a 4xx for the bogus params) proves egress works.

4. Wire the app

Flutter (this repo)

Set one build-time value — everything else is automatic:

PACI_PROXY_BASE=https://paci-kw.<domain>/<SECRET>

It's read by AppEnv.paciProxyBase (env.dart) and consumed by paciUrlViaProxy(...) in paci_basemap_config.dart, which rewrites every PACI/Baladia URL:

• …/paci/<original PACI path> — basemap VectorTileServer, style root, tiles
• …/baladia/<original Baladia path> — parcels MapServer overlay

Empty (default) → direct upstream URLs, unchanged. So this is inert until the value is set. env/dev.json is set (local dev). CI does not use env files — the three shipping workflows (flutter-web, flutter-android, flutter-ios) pass --dart-define=PACI_PROXY_BASE=$PACI_PROXY_BASE from the GitHub Actions secret PACI_PROXY_BASE (already set via gh secret set).

Web repo (aldilaijanre) — mostly already handled

The paci-proxy edge function does not call PACI directly. On origin/main it routes through the hermes-server PACI bridge (supabase/functions/_shared/paciBridge.ts → PACI_BRIDGE_URL /api/paci/fetch, auth PACI_BRIDGE_SECRET) because Supabase Edge egress can't reach PACI's perimeter but hermes-server can. So the web server-side PACI path is already solved — do not redirect it at this proxy.

The only remaining (optional) web change is client-side: the React ArcGIS basemap/Baladia constants (src/utils/paciMapUtils.ts, src/components/maps/arcgis-paci-helpers.ts, src/components/maps/BaladiaMap.tsx, src/constants/baladiaSources.ts) still fetch tiles directly from kuwaitportal.paci.gov.kw / gismaps.baladia.gov.kw, which is CORS- + geo-blocked in the browser. Pointing those at https://paci-kw.aldilaijan.com/<SECRET>/paci|baladia (behind a VITE_PACI_PROXY_BASE env) would let the web render real PACI vector tiles. Do it as a separate PR from a fresh origin/main worktree if wanted.

Troubleshooting: map renders blank (tiles 302 to a login page)

Symptom: the PACI vector basemap renders empty/blank in the app, but property pins on top of it still work fine (they come from Supabase, a separate data path).

Root cause (found 2026-07-03): nginx's connection(s) to kuwaitportal.paci.gov.kw can get stuck in a state where every request — including the style JSON, not just tiles — gets redirected (302) to an ArcGIS Portal login page, even though PACI itself is completely healthy. A single fresh connection (e.g. curl run directly on the box, bypassing nginx) always succeeds immediately. The failure is specific to nginx's proxied connection, not the network path, not geo-blocking, and not a config regression — nginx -T matched this doc exactly with no drift when this was diagnosed.

Diagnose:

ssh -i ~/.ssh/paci_kw root@<VPS_IP>   # lightnode_kw key does NOT work for this box
SECRET=$(cat /etc/paci-proxy/secret)

# Does PACI itself work from this box, bypassing nginx entirely?
curl -sD - -o /dev/null "https://kuwaitportal.paci.gov.kw/arcgisportal/rest/services/Hosted/PACIKFBasemap/VectorTileServer/tile/12/1698/2593.pbf"
# -> 200 real tile data means PACI/the VPS's IP are fine.

# Does the SAME tile via nginx locally reproduce the failure?
curl -sD - -o /dev/null "http://localhost:80/$SECRET/paci/arcgisportal/rest/services/Hosted/PACIKFBasemap/VectorTileServer/tile/12/1698/2593.pbf?cb=1"
# -> 302 to a `.../login?returnUrl=...` confirms this exact bug.

Fix: systemctl restart nginx on the box. This alone resolved it immediately and durably in testing (repeated + concurrent requests across multiple rounds all succeeded afterward — see tool/paci_proxy_stress_test.py in the Flutter repo, which exercises this exact failure signature against a grid of tiles across Kuwait: python tool/paci_proxy_stress_test.py --rounds 5 --sleep 30).

Watchdog (deployed 2026-07-03): a systemd timer runs paci-proxy-healthcheck.sh every 5 minutes — it probes a spread of PACI assets (style JSON, sprites, several tiles) through nginx (not /healthz, which never touches PACI), cache-busted. The spread matters: the wedge can be partial — some assets 302 to the login page while a single canary tile still returns 200 — so it restarts nginx if any asset gets a 302/5xx and logs the outcome to /var/log/paci-proxy-healthcheck.log on the box. The same run also probes a Baladia export tile: because Baladia outages are upstream-side (see the next section), it does not restart nginx for those — it only alerts (log + optional webhook) after 3 consecutive Baladia failures. Drop a Slack/Discord webhook URL into /etc/paci-proxy/alert-webhook on the box to get push alerts; without it Baladia failures are logged only. To (re)install after a VPS rebuild:

scp docs/infra/paci-proxy-healthcheck.sh root@<VPS_IP>:/tmp/
scp docs/infra/paci-proxy-healthcheck.service docs/infra/paci-proxy-healthcheck.timer root@<VPS_IP>:/tmp/
ssh root@<VPS_IP> '
  install -m 755 /tmp/paci-proxy-healthcheck.sh /usr/local/bin/paci-proxy-healthcheck.sh
  install -m 644 /tmp/paci-proxy-healthcheck.service /etc/systemd/system/
  install -m 644 /tmp/paci-proxy-healthcheck.timer /etc/systemd/system/
  systemctl daemon-reload
  systemctl enable --now paci-proxy-healthcheck.timer
'

Check journalctl -u paci-proxy-healthcheck.service or the log file above if you suspect the watchdog itself needs attention.

Client-side (app): the app's PACI→Google fallback is now recoverable — after it falls back it periodically re-attempts the PACI style (every 45s) and restores the vector basemap on its own once the proxy is healthy. Before, a brief wedge stranded a user on Google for the whole session (only a reload or screen change fixed it, because the fallback never retried PACI). See _scheduleRetry in lib/shared/widgets/paci_flutter_map_layers.dart.

Troubleshooting: parcels blank but PACI basemap is fine (Baladia 504s)

Symptom: the map basemap renders fine, but the Baladia parcels overlay (the "Municipality parcels" toggle) shows blank/missing tiles. Because Baladia is an overlay drawn on top of whichever basemap is selected — including the default PACI basemap — this reads to users as "the PACI map is broken." This is a different failure from the login-redirect above: PACI is healthy.

Root cause (found 2026-07-04): the Baladia GIS upstream (gismaps.baladia.gov.kw / 52.233.182.142) is intermittently overloaded and takes longer than nginx's proxy_read_timeout to return response headers, so nginx returns 504. It comes in bursts and self-recovers — one day had 533 × 504 (89% of Baladia requests) while PACI stayed 100% healthy. This is upstream-side: systemctl restart nginx does not help (restarting only fixes the PACI wedge above).

Diagnose:

ssh -i ~/.ssh/paci_kw root@<VPS_IP>
# How bad, right now? (504 vs 200 for the overlay)
grep '/baladia/' /var/log/nginx/access.log | grep '" 504 ' | wc -l
grep '/baladia/' /var/log/nginx/access.log | tail -n 50 | awk '{match($0,/" [0-9]{3} /); print substr($0,RSTART+2,3)}' | sort | uniq -c

# Is Baladia itself responding right now (correct SNI, bypassing nginx)?
curl -sk --resolve gismaps.baladia.gov.kw:443:52.233.182.142 -m 60 -o /dev/null \
  -w "%{http_code} ttfb=%{time_starttransfer}s\n" \
  "https://gismaps.baladia.gov.kw/arcgis/rest/services/KA/ParcelLocation_MKM/MapServer?f=json"
# 200 in ~1s = Baladia recovered; a hang/504 = still in a burst.

Or, from the Flutter repo on any machine, characterize it with the load test:

python tool/paci_proxy_stress_test.py --baladia --rounds 5 --sleep 30

Fix / mitigation: there is no server-side fix — Baladia is external government infrastructure. What we do control:

• App (shipped): the overlay's tile provider tracks sustained failures and the switcher shows a "temporarily unavailable — retrying" hint; it fails fast (~15s) and auto-recovers when Baladia does. See lib/shared/maps/basemap_layer_builder.dart.
• Proxy: the /baladia/ location uses a longer proxy_read_timeout (60s vs PACI's 30s) so slow-but-valid renders complete and warm the cache; and nginx serves stale-on-504 for tiles it has already cached.
• Monitoring: the watchdog alerts (not restarts) on sustained Baladia 504s.

Notes & caveats

• The secret ships in the client bundle (it's in the URL), so it only deters drive-by scanning, not a determined extractor. For real abuse protection add a Cloudflare rate-limit / WAF rule on paci-kw.<domain>. Rotating the secret means redeploying clients — keep it long and stable.
• Absolute URLs in ArcGIS metadata: if the basemap metadata loads but tiles still go straight to PACI (and get blocked), enable the sub_filter block in the cloud-init snippet to rewrite embedded kuwaitportal.paci.gov.kw URLs to the proxy, then systemctl reload nginx.
• Cache: nginx keeps a 7-day, 4 GB on-disk tile cache (/var/cache/nginx/paci) and serves stale on PACI outages. Cloudflare is the primary cache; nginx is the second tier.
• The live secret is not committed — it lives only on the box (/etc/paci-proxy/secret) and in the deploy env files.