With in a planner, individual plans should be able to have their own set of members. As of now a member is associated with all or no plans.
Even if a team site has its own Planner, the individual Plans with in it should be customizable with which members you decide should be a part of it.
Sarah Griffin commented
Yes, we have a Department Group/Team, and we have a Planner board for all department tasks, but we want a Planner board just for the Managers/Supervisors within the department, that the rest of the members do not have permission to see.I want to make a Planner board, assign it to the Group, but only invite select Members (not ALL group members)
I'm seeing the same kind of issue, but particularly via Teams, you can make a private channel and only have some team members have access to it, but private channels can't have plans