or are you trying to write tests that ensure that paths in your schema are or are not possible

no, actually provisioning fixtures (relationships) and then doing checks, lookups and expands on top of it

yeah, the latter I'd say

yeah, so some of that behavior is actually possible with the devtools API and not the regular api/testing api

(devtools is what drives the playground)

It doesn't yet have the ability to perform Expand or Lookups, but that could be added

yeah, and it wouldn't too difficult for me to mimic what's in the e2e package and just spin the

test-serve

or

server

outside of the process

yeah, the idea there would be to provide that as a library or testing API

the playground's validation tooling calls that dev server API

Another option is to simply start a server with a YAML config with your schema and relationships already defined and then run queries against that to assert truths -- which is basically how the devtools API works internally

now, let's say one of those paths is not working as I expect

so the schema is defined, but there is a bug in spicedb

and let's also take into consideration that we are not the creators of

spicedb

and dont have an astral connection with the code 😅

what do you think would facilitate us troubleshoot that bug ourselves?

(hence my point of facilitating debugging)

the best bet would be to add the test case to the consistency tests in that scenario

any schema + test data in https://github.com/authzed/spicedb/tree/main/internal/services/testconfigs is tested as fully as we can reasonably do in the consistency test suite

so if you drop a schema + test data in there, and then run

go test ./...

, it'll be tested exhaustively

i.e. check every relationship, expand every one, lookup every one (and verify with checks), etc

that's our recommendation for identifying bugs, partially because it ensures the bug won't occur again (since we'd check the test case in)

I see, I think that's satisfying, would allow me to quickly try to figure out what's going on in the codebase without much overhead

and help us debug in the future as a community 🙂

It's also a very convenient way to create an issue and give developers a repro-case

Rather than having code you've written internally that might be tightly coupled to other systems

no question about it 🙂 i'm coming from the angle of actually ramping up a whole engineering team in a complex codebase we may be committing our architecture to, so anything that facilitates going from a bug manifested in a client application to debugging it in spicedb codebase would definitely help accelerating contributions IMO

I think we are on the same page 😉

Absolutely. We should also document processes like this so that other users can figure out how to reproduce any bugs, too.

Our team heavily uses pprof (profiling) and jaeger (distributed tracing), but not debuggers, so we're also open to hearing about how we can facilitate that workflow better.

When we've found bugs in the past, we've mostly discovered them via the playground and then exporting the YAML from that and dropping it into the consistency tests.

Thanks for the feedback y'all, very insightful 🙇🏻‍♂️ I'll make sure to report findings! I'll check out the devtools API 💯