https://authzed.com logo
Join Discord
Powered by
Deploying SpiceDB Operator on Google Cloud (with P...
# spicedb
r
I'm having some trouble deploying SpiceDB using the operator and postgres datastore on Google Cloud. We use Pulumi as our IaC tool, so it may be related to the way I'm trying to translate the config into a pulumi-esque way of doing it. - I have the operator running fine (cloned the spicedb-operator repo, copied the /config dir and loaded each .yaml individually with pulumi). - I have a Postgres instance configured in Google Cloud SQL with a database named 
spicedb-backend
. - I have a User configured with that database on CloudSql (more info coming in thread)
Now I'm trying to create SpiceDBClusters. Here's the config I'm passing:
Copy code
const spiceClusters = new kubernetes.apiextensions.CustomResource(
  'spicedb-clusters',
  {
    apiVersion: 'authzed.com/v1alpha1',
    kind: 'SpiceDBCluster',
    metadata: {
      name: 'spicedb-cluster',
    },
    spec: {
      config: {
        replicas: 2,
        loglevel: 'debug',
        datastoreEngine: 'postgres',
      },
      secretName: 'spicedb-config',
    },
  },
  { provider, dependsOn: [spiceDBConfig] }
);
And here's the 
spiceDBConfig
secret:
Copy code
const spiceDBConfig = new kubernetes.core.v1.Secret(
  'spicedb-config',
  {
    metadata: {
      name: 'spicedb-config',
    },
    stringData: {
      preshared_key: sqlUser.password.apply(p=> p ?? ''),
      datastore_uri: fullDBUri,
    },
  },
  { provider }
);
So the weird stuff seems to be happening with the 
datastore_uri
string. If I make the 
fullDbUri
string look like this:
Copy code
const fullDBUri = pulumi.interpolate`postgresql://${sqlUser.name}:${sqlUser.password}@${postgres.privateIpAddress}:5432/${postgresDb.name}`;
Then I get an error that looks like this in the migration pod that's created:
Copy code
unable to create migration driver for postgres: failed to connect to `host=spicedb-sql-user user=nonroot database=`: hostname resolving error (lookup spicedb-sql-user on 10.8.240.10:53: no such host)
Note that 10.8.240.10 is NOT the internal IP of my postgres instance (weird). Also It's taking the username as the host and can't figure out the database name??
But if I make 
fullDbUri
look like this (with just username):
Copy code
const fullDBUri = pulumi.interpolate`postgresql://${sqlUser.name}@${postgres.privateIpAddress}:5432/${postgresDb.name}`;
Then I see this error in the migration pod:
Copy code
unable to create migration driver for postgres: failed to connect to `host=<CORRECT_IP> user=spicedb-sql-user database=spicedb-backend`: failed SASL auth (FATAL: password authentication failed for user "spicedb-sql-user" (SQLSTATE 28P01))
Note that it seems the connection string is being parsed correctly here: host, user, and database are all correct. I think I'm passing Spice my password via the 
preshared_key
bit in the configs but maybe that's not the case.
How does the operator / spicedbcluster want me to give it the password and username I want it to be using for my postgres instance?
j
@ecordell ^
r
behavior is the same in each case also when I add 
?sslmode=disable
to the uri strings
e
you might need to escape the password?
hard to be sure without seeing it
also just fyi: it looks like you're setting your db password as the spicedb preshared key too, you probably want a different value there
r
> you might need to escape the password? almost certainly this (facepalm) trying it now
> spicedb preshared key what does this do then, if not the pg password?
never mind, i see it's for the clients to connect to spice, not for spice to connect to DB
Yeah I needed to URL encode the password, had special chars. Thanks @ecordell !!
e
while we're at it I'm pretty sure there are pulumi handlers for kustomize that would let you directly install a release like this? https://github.com/authzed/spicedb-operator/releases/tag/v1.12.0 but it's been a while, might be misremembering
r
It could be that I wasn't doing it correctly, but pulumi didn't like it when I tried to do it this way. It couldn't figure out that I wanted to use the same kubernetes provider for all the sub-resources in this kustomization. But I was also pointing it straight to 
https://github.com/authzed/spicedb-operator/tree/main/config
instead of this release. Maybe that was the problem.
In either case, @ecordell , I wonder whether the operator handles auto-scaling the SpiceDB nodes? Or will I need to configure the autoscaler?
Yeah, it's also not working if I point it to 
https://github.com/authzed/spicedb-operator/releases/download/v1.12.0/bundle.yaml
or 
https://github.com/authzed/spicedb-operator/config?ref=v1.12.0
... but that seems like a pulumi problem, not a spice problem.
@ecordell I found the problem with pulumi, it was actually a safeguard that we wrote to keep ourselves from doing something stupid that wasn't allowing me to deploy it this way. so I fixed that and was able to deploy the operator using https://www.pulumi.com/registry/packages/kubernetes/api-docs/kustomize/directory/
Once I get a full deployment up and running, I'd love to write a blog post or something as a guide for doing this... I think we'd also be cool with commiting our code as an example in the operator repo.
8 Views
PreviousNext
Powered by