I think there could still be a problem if the user is a member of two teams, and the team is the one that's removed access. We wouldn't have a way to know if removing team:1's access to the object, would actually cause that user to lose permissions or not