Managing the Dual Write problem
# spicedb
v
Thinking through the options from the Authzed YouTube video on this, does the following look workable for a reconciliation based system?
Copy code
ts
      //
      // DUAL WRITE PROBLEM
      //
      await dualWriteTransaction({
        ctx,
        repo: repoDid,

        // (DWP/1) writes to the repo database
        actorOps: async (actorTxn: ActorStoreTransactor) => {

          // space (self/profile) record
          actorTxn.space.insertRecord({
            uri: spaceRecord.uri,
            space: spaceRecord.uri.space,
            collection: spaceRecord.uri.collection,
            rkey: spaceRecord.uri.rkey,
            cid: spaceRecord.cid,
            record: spaceRecord.record,
            did: reqDid,
          })

          // space (owner/parent) record
          actorTxn.space.insertRecord({
            uri: spaceRelation.uri,
            space: spaceRelation.uri.space,
            collection: spaceRelation.uri.collection,
            rkey: spaceRelation.uri.rkey,
            cid: spaceRelation.cid,
            record: spaceRelation.record,
            did: reqDid,
          })

        },

        // (DWP/2) writes to the authz service
        authzOps: async (spicedbClient) => {
          await createRelationship(spicedbClient, resource, relation, subject)
        },
      })

      // TODO, (DWP/3) background reconciliation process
Copy code
ts
export async function dualWriteTransaction({
  ctx,
  repo,
  actorOps,
  authzOps,
}: {
  ctx: AppContext,
  repo: string,
  // writes: async (actorTxn) => Promise<void>[],
  actorOps: (actorTxn) => Promise<void>,
  authzOps: (spicedbClient) => Promise<void>,
}) {
  // DUAL WRITE PROBLEM
  // errors could occur anywhere in here
  // and we need to unroll or resolve towards eventual consistency
  // (depending on where there error occurs?)
  //
  // We do the writes in this order
  // (1) actor db (sqlite)
  // (2) spicedb (service)
  // Since we also store the relation in a record in the actor db.
  // we can later ensure that the relation is created in spicedb as well,
  // should the failure happen between the two writes.

  try {
    await ctx.actorStore.transact(repo, async (actorTxn) => {
      // (1) Write our transaction to the database
      await actorOps(actorTxn)
    })

  } catch (err) {
    console.error('error during actor txn', err)
    throw err
  }

  try {
    // (2) Write our relations to the database
    await authzOps(ctx.spicedbClient)

    // we don't roll back because we run a reconciliation process
    // we should support some form of retry, initiated by the user
    // or they do something which starts a reconcilation process for that data
    // with ownership in spice, the account owners should be able to see everything
    // regardless if an operation fails on some operation under their account
    // (i.e. account has god mode over everything under their root space, made at creation time)

    // the reconciliation process is also scoped to an account
    // so it should be very easy to trigger with an xrpc call (yet to be created)

    // additionally, this error handler could notify the reconcilation system
    // that this account / record / operation should be looked at
    // so that we might shorten the loop in eventual consistency
  } catch (err) {
    console.error('error during authz txn', err)
    throw err
  }
}
also, can I wrap multiple writes to SpiceDB in a transaction?
y
no, spicedb doesn't really have transactional semantics. every write call is treated as atomic.
and if that transaction manager is in-process, it'll work for any error that's recoverable within the context of the process, but that doesn't cover things like a server crashing
it's better than not doing the error handling at all, and will deal with most transient errors, but it's not going to be as durable as something like a durable workflow system or event sourcing
v
Yeah, I'm not sure getting event systems or more external api dependencies is something this particular piece of software would likely adopt. It needs to remain accessible to the self-hoster.
Having said that, it seems prudent to expose that choice to the person running the PDS
For someone like Bluesky, running Spice on Spanner for many PDS and using a real event system may be of interest
glad I asked, thanks for bouncing ideas around!
y
sure thing!
7 Views