Transformers: Applications in Language and Communication

Course goals

After completing this course, students can

• assess the practical and functional capacities and limitations of Transformer models when applied to text generation and to other NLP tasks on which they are fine-tuned
• practically apply and fine-tune pre-trained Transformer models
• understand the practical relevance of Transformer models for professions in the creative, language, communications, and media sector;

Assessment
Two assignments, each counting for 50% of the final mark:

• assignment 1, individual:  solving challenges in a take-home exam format
• assignment 2, group: research project (presentation & report)

Elaborate assignment descriptions and assessment rubrics exist for both assignments.

To qualify for a repair of the final result the mark needs to be at least a 4, or “AANV”.

Content

Transformer-based Large Language Models have caught the eye of the wider public recently with ChatGPT.
They are a point in a gradual development of large, data-driven language models (LLMs) that can generate output text that is syntactically smooth and that is semantically coherent with input prompts.

This course takes the current state of the art, Transformer models, as a focus.
Transformer models are related to research in Artificial Intelligence and have applications in professional writing domains (language, communication and media industries) as well as in basic office tools for the wider public (e.g. text editors).

In the first quarter of the course, Transformers are introduced technically and functionally, in view of the recent history of neural-network based language modeling and a longer history of predecessors.
The middle half core of the course then explores the ‘magical’ properties of Transformers: their capacity to be fine-tuned to performing a wide range of natural language processing (NLP) tasks.
We explore several of these fine-tunings, addressing issues such as evaluating performance and scalability.
The final quarter of the course zooms in on ChatGPT, addressing issues such as its particular additional reinforcement learning layer, its multilingual capacities, “prompt science”, and ethics.

Course form
Lectures, seminars.

Literature
As the field of Transformers is in high flux, no text books exist as of yet.
The literature will be composed of various articles that fit each weekly topic.

Additional information

Enrolment in this course is only available by using the pre-enrolment form that will be forwarded to all ADSM students in the first week of November.