SpiceDB is an open source, Google Zanzibar-inspired, database system for creating and managing security-critical application permissions.

SpiceDB

my config has now:
```
spec:
  config:
    datastoreEngine: memory
    image: authzed/spicedb:v1.15.0
    dispatchUpstreamAddr: ""
```
but still get the error. That was my lunch break, 4 hours of meeting next and I'll update the buug reports

I did reproduce the same behaviour locally with a fresh kube+operator+spicedb 1.15

So I think I've identified 2 problems:
- operator does not support the dispatchUpstreamAddr flag, so it ignores it. Hence it didn't work for you
- operator sets `SPICEDB_CLUSTER_DISPATCH_ENABLED` which disables the dispatch server, but `SPICEDB_DISPATCH_UPSTREAM_ADDR` which enables dispatching. This is a bug and it should be set to empty string.

The only option that I'm aware of right now is you will have to modify the `Deployment`, by changing the environment variable `SPICEDB_DISPATCH_UPSTREAM_ADDR` to empty.

https://github.com/authzed/spicedb-operator/issues/30

Why would you want this? I was under the impression dispatching made it better?

https://github.com/authzed/zed/issues/169

normally, you wouldn't, but it appears its not working on his cluster for some reason?

do you see a configured Service pointing to the SpiceDB nodes?

yep (btw its minikube, akak single node cluster):
```
kubectl get service dev 
NAME   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                 AGE
dev    ClusterIP   10.105.248.177   <none>        50051/TCP,50053/TCP,8443/TCP,9090/TCP   9d
```

where is the IP `192.168.9.11` from then?

I redeployed with the suggested `dispatchUpstreamAddr: ''` in the config

sure, but the operator doesn't support disabling dispatch AFAIK

I'm more curious as to where dispatch is getting the 192 addres

dispatching does not make sense for the memory datastore

this a good question, the service as a cluster ip

can you `kubectl describe` the SpiceDB pod and put it here for us to see (make sure to redact any secrets in env vars, if there are any)

https://gist.github.com/ensonic/5a134ac50b73dbdaed2c6c9aaed75a6a

```
zed context list
CURRENT    NAME     ENDPOINT           TOKEN         INSECURE 
           local    localhost:50051    <redacted>       ✓        
   ✓       blog     localhost:50051    <redacted>       ✓  
```

let me reboot my machine and start over, something weird and I don't want you to waste time debugging if that fixes it

where is this connection to the dispatcher originating from? spicedb or zed?

because of:
```
 zed --log-level=debug permission check blog/post:1 read  blog/user:emilia --revision "${ZEDTOKEN}"
8:08PM DBG set log level new level=debug
8:08PM DBG extracted response dispatch metadata cached=0 dispatch=1
Error: rpc error: code = Unavailable desc = last connection error: connection error: desc = "transport: Error while dialing dial tcp 192.168.9.34:50053: connect: connection refused"
```

the pod ip is the one shows in the github gist and it the one where it complain that I can't connect to

since I restarted everything:
```
kubectl describe pod dev-spicedb-6cf866cfcb-8qnz4
...
IP:               192.168.9.34
Containers:
  dev-spicedb:
    Container ID:  containerd://6c1896278d7614e27d48fc405d12310a510c66a55543b36f76f7cf71bd2f5e50
    Image:         authzed/spicedb:v1.15.0
    Image ID:      docker.io/authzed/spicedb@sha256:f1cd59fb584ebb06eb20c7ca30a57106ee86e22bd3708cafe7c10052dc352bc5
    Ports:         50051/TCP, 50053/TCP, 8443/TCP, 9090/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP
...

zed --log-level=debug permission check blog/post:1 read  blog/user:emilia --revision "${ZEDTOKEN}"
12:12PM DBG set log level new level=debug
12:12PM DBG extracted response dispatch metadata cached=0 dispatch=1
Error: rpc error: code = Unavailable desc = last connection error: connection error: desc = "transport: Error while dialing dial tcp 192.168.9.34:50053: connect: connection refused"
```

The env-vars say: `SPICEDB_DISPATCH_CLUSTER_ENABLED:  false` - is the dispatch actually running?

If I alos port forward the dispatch port:
```
kubectl port-forward deployment/dev-spicedb 50051:50051 50053:50053
Forwarding from 127.0.0.1:50051 -> 50051
Forwarding from [::1]:50051 -> 50051
Forwarding from 127.0.0.1:50053 -> 50053
Forwarding from [::1]:50053 -> 50053
Handling connection for 50053
E1124 12:19:22.623092  859177 portforward.go:406] an error occurred forwarding 50053 -> 50053: error forwarding port 50053 to pod e0565e58875b91a9dfa1ac9858865068e576c4fc7d9fef2ef1af87bd1ae6cf22, uid : failed to execute portforward in network namespace "/var/run/netns/cni-172be34b-ae63-1a8e-2520-77c3c58434a5": failed to connect to localhost:50053 inside namespace "e0565e58875b91a9dfa1ac9858865068e576c4fc7d9fef2ef1af87bd1ae6cf22", IPv4: dial tcp4 127.0.0.1:50053: connect: connection refused IPv6 dial tcp6 [::1]:50053: connect: connection refused 
E1124 12:19:22.623635  859177 portforward.go:234] lost connection to pod
```
and then poke it:
```
nc -zvw1 localhost 50053
Connection to localhost (::1) 50053 port [tcp/*] succeeded!
```
It seems to confirm that dispatch is actually not running

In addition, if I launch a shell into the cluster and probe the ports, it confirms that no dispatch runs
```
kubectl run -it --rm --restart=Never alpine --image=alpine sh
/ # nc -zvw1 dev.default.svc.cluster.local 50051
dev.default.svc.cluster.local (10.104.111.61:50051) open
/ # nc -zvw1 dev.default.svc.cluster.local 50053
/ #
```

ignore me, just saw the PR (https://github.com/authzed/spicedb-operator/pull/112)

(if you don't mind responding in thread, so we can track it)

```
SPICEDB_DISPATCH_CACHE_ENABLED : true
SPICEDB_DISPATCH_CLUSTER_ENABLED : true
SPICEDB_DISPATCH_UPSTREAM_ADDR : kubernetes:///database.spicedb:dispatch
```

TLS is disabled by default because you have to specify the `--dispatch-cluster-tls-key-path` and `--dispatch-cluster-tls-cert-path` flags

do y'all have any interest in a kafka connector that would turn WatchService messages into Kafka events?