Only Activity Group Managers can manage Activity Rosters through the group itself.

Activity Leaders can add and drop members under Extracurricular > Activities > Requests & Schedules > Activity Members > Edit