https://authzed.com logo
Title
j

Joey

11/23/2022, 4:11 PM
and you are still seeing the same error?
what do you see for
zed context list
?
e

ensonic

11/23/2022, 4:14 PM
zed context list
CURRENT    NAME     ENDPOINT           TOKEN         INSECURE 
           local    localhost:50051    <redacted>       ✓        
   ✓       blog     localhost:50051    <redacted>       ✓
let me reboot my machine and start over, something weird and I don't want you to waste time debugging if that fixes it
reinstalled everything, still happening
where is this connection to the dispatcher originating from? spicedb or zed?
because of:
zed --log-level=debug permission check blog/post:1 read  blog/user:emilia --revision "${ZEDTOKEN}"
8:08PM DBG set log level new level=debug
8:08PM DBG extracted response dispatch metadata cached=0 dispatch=1
Error: rpc error: code = Unavailable desc = last connection error: connection error: desc = "transport: Error while dialing dial tcp 192.168.9.34:50053: connect: connection refused"
j

Joey

11/23/2022, 7:14 PM
spicedb
the question is: why can't it hit itself
do you know the pod's IP by any chance?
e

ensonic

11/24/2022, 11:13 AM
the pod ip is the one shows in the github gist and it the one where it complain that I can't connect to
since I restarted everything:
kubectl describe pod dev-spicedb-6cf866cfcb-8qnz4
...
IP:               192.168.9.34
Containers:
  dev-spicedb:
    Container ID:  containerd://6c1896278d7614e27d48fc405d12310a510c66a55543b36f76f7cf71bd2f5e50
    Image:         authzed/spicedb:v1.15.0
    Image ID:      docker.io/authzed/spicedb@sha256:f1cd59fb584ebb06eb20c7ca30a57106ee86e22bd3708cafe7c10052dc352bc5
    Ports:         50051/TCP, 50053/TCP, 8443/TCP, 9090/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP
...

zed --log-level=debug permission check blog/post:1 read  blog/user:emilia --revision "${ZEDTOKEN}"
12:12PM DBG set log level new level=debug
12:12PM DBG extracted response dispatch metadata cached=0 dispatch=1
Error: rpc error: code = Unavailable desc = last connection error: connection error: desc = "transport: Error while dialing dial tcp 192.168.9.34:50053: connect: connection refused"
The env-vars say:
SPICEDB_DISPATCH_CLUSTER_ENABLED:  false
- is the dispatch actually running?
If I alos port forward the dispatch port:
kubectl port-forward deployment/dev-spicedb 50051:50051 50053:50053
Forwarding from 127.0.0.1:50051 -> 50051
Forwarding from [::1]:50051 -> 50051
Forwarding from 127.0.0.1:50053 -> 50053
Forwarding from [::1]:50053 -> 50053
Handling connection for 50053
E1124 12:19:22.623092  859177 portforward.go:406] an error occurred forwarding 50053 -> 50053: error forwarding port 50053 to pod e0565e58875b91a9dfa1ac9858865068e576c4fc7d9fef2ef1af87bd1ae6cf22, uid : failed to execute portforward in network namespace "/var/run/netns/cni-172be34b-ae63-1a8e-2520-77c3c58434a5": failed to connect to localhost:50053 inside namespace "e0565e58875b91a9dfa1ac9858865068e576c4fc7d9fef2ef1af87bd1ae6cf22", IPv4: dial tcp4 127.0.0.1:50053: connect: connection refused IPv6 dial tcp6 [::1]:50053: connect: connection refused 
E1124 12:19:22.623635  859177 portforward.go:234] lost connection to pod
and then poke it:
nc -zvw1 localhost 50053
Connection to localhost (::1) 50053 port [tcp/*] succeeded!
It seems to confirm that dispatch is actually not running
In addition, if I launch a shell into the cluster and probe the ports, it confirms that no dispatch runs
kubectl run -it --rm --restart=Never alpine --image=alpine sh
/ # nc -zvw1 dev.default.svc.cluster.local 50051
dev.default.svc.cluster.local (10.104.111.61:50051) open
/ # nc -zvw1 dev.default.svc.cluster.local 50053
/ #
j

Joey

11/24/2022, 6:17 PM
Ah, that’d do it