844600078504951838 #spicedb

Channels

romil

04/12/2022, 6:11 AM

Hello, Trying to configure the spicedb dispatcher and getting below error from respective pods kuberesolver: watching ended with error='invalid response code 403 for service spicedb Below is the snippet of the configuration:

apiVersion: "v1"
kind: "Service"
metadata:
  name: "spicedb"
  labels:
    app: "spicedb"
spec:
  selector:
    app: "spicedb"
  type: "LoadBalancer"
  ports:
    - name: "grpc"
      port: 50051
      protocol: "TCP"
**      targetPort: 50051
**    - name: "internal"
      port: 50053
      protocol: "TCP"
      targetPort: 50053
    - name: "http"
      port: 8443
      protocol: "TCP"
      targetPort: 8443

-------

  spec:
      dnsPolicy: "ClusterFirst"
      restartPolicy: "Always"
      terminationGracePeriodSeconds: 30
      containers:
        - name: "spicedb"
          image: "quay.io/authzed/spicedb"
          imagePullPolicy: "IfNotPresent"
          command: ["spicedb", "serve"]
          args:
            - grpc-no-tls
            - --http-enabled
            - --dispatch-upstream-addr=kubernetes:///spicedb.authz:50051

------

I assume these are internal port that needs to be configured and not the one expose . Any view why the error still coming ?

user

04/12/2022, 6:12 AM

yeah the

--dispatch-upstream-addr

should be pointing to 50053

user

04/12/2022, 6:12 AM

the 403 I think is because it's attempting to use the 50051 port and not providing the preshared key

romil

04/12/2022, 6:13 AM

Thanks for quick response , I tried configuring 50053 as well, but the error remain same. so is it something to do with the preshared key ? where this key needs to be additionally specified ?

romil

04/12/2022, 6:19 AM

args:
            - grpc-no-tls
            - --http-enabled
            - --dispatch-upstream-addr=kubernetes:///spicedb.authz:50053

user

04/12/2022, 6:20 AM

You shouldn't need to add anything additional for the dispatch wrt the preshared key, but even running at all should require a

--grpc-preshared-key $YOUR_SECRET

. All CLI flags can also be done with environment variables e.g.

SPICEDB_GRPC_PRESHARED_KEY

if you want to mount the value from a secret

user

04/12/2022, 6:21 AM

what version of spicedb are you using? i don't think it even has the

--grpc-no-tls

flag anymore

romil

04/12/2022, 6:21 AM

Correct, I created a secret for same

kubectl -n authz create secret generic spicedb --from-literal=SPICEDB_GRPC_PRESHARED_KEY=graphToken

user

04/12/2022, 6:22 AM

can you try specifying a tag for the image?

quay.io/authzed/spicedb:v1.6.0

romil

04/12/2022, 6:23 AM

ok sure let me try , right now I assume its picking the latest

image: "quay.io/authzed/spicedb"

user

04/12/2022, 6:23 AM

yeah, it could've grabbed latest from a while ago though since your pull policy is

IfNotPresent

user

04/12/2022, 6:26 AM

we need to update the k8s example in the repo, it's an old version of spicedb missing a few useful things (like the http gateway port)

romil

04/12/2022, 6:29 AM

2022/04/12 06:28:35 ERROR: kuberesolver: watching ended with error='invalid response code 403 for service spicedb in namespace authz', will reconnect again
{"level":"info","grpc.component":"server","grpc.method":"Check","grpc.method_type":"unary","grpc.service":"grpc.health.v1.Health","peer.address":"127.0.0.1:51358","protocol":"grpc","requestID":"6c8c5a8ceea2160f692ae2baa4668703","grpc.request.deadline":"2022-04-12T06:28:36Z","grpc.start_time":"2022-04-12T06:28:35Z","grpc.code":"OK","grpc.time_ms":"0.046","time":"2022-04-12T06:28:35Z","message":"started call"}
{"level":"info","grpc.component":"server","grpc.method":"Check","grpc.method_type":"unary","grpc.service":"grpc.health.v1.Health","peer.address":"127.0.0.1:51358","protocol":"grpc","requestID":"6c8c5a8ceea2160f692ae2baa4668703","grpc.request.deadline":"2022-04-12T06:28:36Z","grpc.start_time":"2022-04-12T06:28:35Z","grpc.code":"OK","grpc.time_ms":"0.302","time":"2022-04-12T06:28:35Z","message":"finished call"}
2022/04/12 06:28:36 ERROR: kuberesolver: watching ended with error='invalid response code 403 for service spicedb in namespace authz', will reconnect again

user

04/12/2022, 6:29 AM

oh i wonder if the 403 is actually from kubernetes!

romil

04/12/2022, 6:30 AM

containers:
        - name: "spicedb"
          image: "quay.io/authzed/spicedb:v1.6.0"
          imagePullPolicy: "IfNotPresent"
          command: ["spicedb", "serve"]
          args:
            - grpc-no-tls
            - --http-enabled
            - --dispatch-upstream-addr=kubernetes:///spicedb.authz:50053
          env:
            - name: "SPICEDB_GRPC_SHUTDOWN_GRACE_PERIOD"
              value: "1s"
            - name: "SPICEDB_LOG_LEVEL"
              value: "debug"
            - name: "SPICEDB_GRPC_PRESHARED_KEY"
              valueFrom:
                secretKeyRef:
                  name: "spicedb"
                  key: "SPICEDB_GRPC_PRESHARED_KEY"
          ports:
            - name: "grpc"
              containerPort: 50051
              protocol: "TCP"
            - name: "internal"
              containerPort: 50053
              protocol: "TCP"
          readinessProbe:
            exec:
              command: ["grpc_health_probe", "-v", "-addr=localhost:50051"]
            failureThreshold: 5
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 5

user

04/12/2022, 6:30 AM

yeah i think you need a service account that has get/list/watch on the endpoints resource

romil

04/12/2022, 6:31 AM

Is this what you referring to ? https://discord.com/channels/844600078504951838/900405749405089812/956309021764231198

user

04/12/2022, 6:32 AM

yep!

user

04/12/2022, 6:32 AM

https://github.com/sercand/kuberesolver#rbac

romil

04/12/2022, 6:33 AM

Ok sure, need to go thru this . Thanks

user

04/12/2022, 6:34 AM

🤞Good luck! I'm off to bed for now.

romil

04/12/2022, 8:59 AM

Below is what I have added in the yaml file

apiVersion: v1
kind: ServiceAccount
metadata:
  name: spicedb
  namespace: authz
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: authz
  name: watch-service
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["endpoints"]
  verbs: ["get", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: spicedb-watch-service
  namespace: authz
subjects:
- kind: ServiceAccount
  name: spicedb
roleRef:
  kind: Role
  name: watch-service
  apiGroup: rbac.authorization.k8s.io

I still get below error :

kuberesolver: watching ended with error='invalid response code 403 for service spicedb in namespace authz', will reconnect again

balchu

04/12/2022, 1:21 PM

The role and role binding are in a different namespace as the service account.

balchu

04/12/2022, 1:21 PM

It should be in the same namespace.

romil

04/12/2022, 2:16 PM

all in the above exampel are in authz namespace ? can you point what you found as incorrect plz

balchu

04/12/2022, 2:21 PM

Ooops i thought role was i a diff namespace. By the way, i dont think apiGroup on the role ref is required.

romil

04/12/2022, 2:36 PM

apiGroup is a must

missing required field "apiGroup" in io.k8s.api.rbac.v1.RoleRef; if you choose to ignore these errors, turn validation off with --validate=false

balchu

04/12/2022, 2:39 PM

Ooops. I just did this not so long ago. Wasn't paying to it too much. Your manifest looks ok to me.

balchu

04/12/2022, 2:41 PM

Just assuming that your pods are also in the

authz

namespace

romil

04/12/2022, 2:45 PM

yes