Linguistic concepts can be foreign to students who want to use natural language processing in their work. Few hands-on activities exist for students to learn abstract linguistic constructs. As part of this proposal, we will use a tool that is widely used -- annotation software -- to train students to identify linguistic concepts that they learn in class while also learning about the most critical part of language systems in artificial intelligence -- human feedback and human intelligence. To do so, we propose to use in-class annotation assignments that will be delivered at scale using software that is commonly employed in industry applications (e.g., for annotating medical documents) to create exercises that highlight both practical and theoretical questions. We will assess mastery of linguistic constructs by comparing different groups of students who will perform similar but distinct tasks to annotate the same language data. We will evaluate the learning outcomes at the end of the course using an exit survey and a conceptual quiz.