long, long term you might want to consider something like CRDB

What is handled differently here that allows cockroach to deal with this better than postgres?

it shards data across nodes if large enough