jcmfernandes10/05/2021, 11:01 PM
hook to trigger a background job that writes to Kafka. We are leaning towards implementing a transactional outbox to keep coherency. That's the solution I worked with in the past and it served my previous company well. We did hit scalability issues at some point but we worked around them by dropping ordering guarantees across some domains (e.g., facts for different tenants could be reordered, but not facts for the same tenant) and started distributing the load across multiple publishers using consistent hashing. We only hit these issues once we started facing significant deviations from our average traffic, years after we put the base infrastructure in place.