it is the client's job to deduplicate