Monash University.

CE/Lecturer, Natural Language Processing (FIT5217, 2022, 2023, 2024). Syllabus:

• Week 1 Introduction to Natural Language Processing
• Week 2 Language Modelling
  • What is a language model?
  • Context Length
  • The Chain Rule of probability
  • n-gram language models
  • Data sparsity issues
  • Smoothed n-grams
    • Add-k
    • Kneser-Ney (not examinable)
    • Stupid Backoff
  • Evaluating model performance
• Week 3 Sequence Labelling
  • Word categories
  • Part-of-Speech (POS) Tagging
  • Other Sequence Labelling Problems
  • Hidden Markov Model (HMM)
  • Observation Likelihood
  • Most Likely State Sequence
  • Supervised Learning of HMM
  • Evaluation
• Week 4 Syntactic Parsing
  • Syntax
  • Syntactic Parsing
  • CKY Parsing
  • Limitations of Context Free Grammars
    • Statistical Parsing
    • Probabilistic CKY Parsing
  • PCFG Training
  • Limitations of PCFGs
  • Treebanks
  • Evaluating model performance
  • Alternative Formalisms
• Week 5 Lexical Semantics 
  • Word Sense disambiguation
    • Word Senses
    • WordNet
    • Learn WSD
    • Evaluation
  • Coreference resolution
    • Coreference
    • Coreference Models
    • Evaluation
• Week 6 Linear Text Classification
  • Text classification
  • Classification methods
  • Naïve Bayes Model
  • Logistic Regression Model
  • Evaluation
• Week 7 Neural Networks and Neural Language Models 
  • Introduction to Neural Networks
  • The challenge of statistical language modelling
  • Neural n-gram language models
  • Recurrent language models
  • A few key papers (not examinable)
• Week 8 Neural Machine Translation
  • Machine Translation
  • Decoding Algorithms
  • Sequence-to-Sequence Models
  • Attention Mechanism
  • Evaluation of MT systems
  • Examples of other Encoder-Decoder tasks
• Week 9 Distributional Representations
  • Meaning and Lexical Semantics
  • Vector Semantics
  • Count-based Distributed Representations (tf-id, PMI)
  • Sparse vs. Dense Representation (SVD/PCA)
  • Word Embeddings (Word2Vec, GloVe, fastText)
  • Contextualized Word Embeddings (ELMo, BERT)
• Week 10 Transformers and Pretrained Models
  • Transformers (in details)
  • Pretrained Large Language Models
    • Encoders
    • Decoders
    • Encoder-Decoder
  • Parameter-efficient Finetuning Methods
    • Adapters
    • Prefix-Tuning
    • LoRA
  • Limitations, ethics, biases, environment
• Week 11 Speech Recognition and Translation
  • Speech Processing Tasks
  • Automatic Speech Recognition
    • Model Design
    • Evaluation Metrics
  • Speech Translation
    • Model Design
    • Evaluation Metrics
  • Pre-trained Speech Transformers
    • Wav2vec 2
    • Whisper Speech Encoder
  • SUPERB Evaluation Benchmark
• Week 12  Advanced Topics in Large Language Models
  • Prompting
  • In-context zero- or few-shot prompting
    • Emergent in-context abilities
    • Chain-of-thought Prompting
  • Instruction-Tuning Large Language Models
  • Reinforcement Learning with Human Feedback for LLMs
    • Policy Gradient
    • Reward Model to Rank
  • A systematic limitation of LLMs
    • System 1 and System 2 Modes of Reasoning
  • Latest Trends
    • Alpaca family of models
    • LangChain
    • Multimodal Instruction Tuned LM

CE/Lecturer, Data Analysis for Semi-structured Data (FIT5212, 2022, 2023, 2024). Syllabus:

• Week 1 Intro to semi-structured data
• Week 2 Text representation
  • Word and Document Representations (Count Vector, TF-IDF, LSA)
  • Basic Embedding Models (Word2vec, GLoVe)
  • Contextualized Embedding Models (ELMo, BERT)
• Week 3 Text classification
  • Data Preparation
  • Basic Classifier 
    • Logistic Regression Model and Cross-Entropy Loss Function
  • Neural Network-based Classifier
  • BERT-style Classification
• Week 4 Text clustering and matrix factorisation
  • Clustering Examples
  • Mixture Models
  • Topic Models (LDA)
  • Matrix Factorisation
• Week 5 Text generation
  • ​​​​​​​Neural Language Models
  • Sequence-to-Sequence models
  • Neural Machine Translation
  • Attention mechanism
  • Other language generation tasks
  • Evaluation
• Week 6 Pretrained language models
  • ​​​​​​​Introduction to Transformers
  • Pretrained Language Models
    • Pre-trained Encoder-Decoders (T5 and BART)
    • Pre-trained Encoders (BERT family)
    • Pre-trained Decoders (GPT family)
  • Finetuning Pretrained Language Models
• Week 7 Recommender systems (1) 
  • ​​​​​​​Introduction to recommender systems
  • Various Recommender Systems in Real-Life Applications
  • Content Based Recommendation Systems
  • Collaborative Filtering Algorithms
• Week 8 Recommender systems (2)
  • ​​​​​​​Matrix Factorisation for Recommender Systems
  • Neural Collaborative Filtering
  • Neural Matrix Factorisation
• Week 9 Graph clustering 
  • ​​​​​​​Graph Basics
  • K-means Graph Clustering
  • Spectral Clustering
  • Graph Clustering Evaluation
• Week 10 Graph representation learning
  • ​​​​​​​Network Embedding Principle
  • Random Walk Based Approaches
  • Matrix Factorization Approaches
  • Evaluation
• Week 11 Graph Neural Networks
  • ​​​​​​​Disadvantages of shallow embedding for graph data
  • Connections of graph neural networks to traditional deep learning
  • Basic idea of graph neural networks
  • Graph Convolutional Network (GCN)
  • GraphSage Network (GraphSage)
  • Graph Attention Network (GAT)
  • Application of graph neural networks
• Week 12 Knowledge Graphs and Text
  • ​​​​​​​Knowledge Graph construction
  • Examples of Knowledge Graphs
  • Knowledge-intensive tasks
  • Infusing external knowledge into text models

UCL.
Co-lecturer, Applied Machine Learning II (ELEC0135, 2020).
Module Lead, Emerging Topics in Integrated Machine Learning Systems (ELEC0139, 2020).

University of Cambridge.
Lecturer, Quantitative Methods in Analyzing Linguistic Data (QMALD, 2019).
Guest Lecturer, Computational Linguistics (LI18, 2019).