Monash University.

CE/Lecturer, Natural Language Processing (FIT5217, 2022, 2023, 2024). Syllabus:

• Week 1 Introduction to Natural Language Processing
• Week 2 Language Modelling
  • What is a language model?
  • Context Length
  • The Chain Rule of probability
  • n-gram language models
  • Data sparsity issues
  • Smoothed n-grams
    • Add-k
    • Kneser-Ney (not examinable)
    • Stupid Backoff
  • Evaluating model performance
• Week 3 Sequence Labelling
  • Word categories
  • Part-of-Speech (POS) Tagging
  • Other Sequence Labelling Problems
  • Hidden Markov Model (HMM)
  • Observation Likelihood
  • Most Likely State Sequence
  • Supervised Learning of HMM
  • Evaluation
• Week 4 Syntactic Parsing
  • Syntax
  • Syntactic Parsing
  • CKY Parsing
  • Limitations of Context Free Grammars
    • Statistical Parsing
    • Probabilistic CKY Parsing
  • PCFG Training
  • Limitations of PCFGs
  • Treebanks
  • Evaluating model performance
  • Alternative Formalisms
• Week 5 Linear Text Classification
  • Text classification
  • Classification methods
  • Naïve Bayes Model
  • Logistic Regression Model
  • Evaluation
• Week 6 Neural Networks and Neural Language Models 
  • Introduction to Neural Networks
  • The challenge of statistical language modelling
  • Neural n-gram language models
  • Recurrent language models
  • A few key papers (not examinable)
• Week 7 Neural Machine Translation
  • Machine Translation
  • Decoding Algorithms
  • Sequence-to-Sequence Models
  • Attention Mechanism
  • Evaluation of MT systems
  • Examples of other Encoder-Decoder tasks
• Week 8 Distributional Semantics
  • Meaning and Lexical Semantics
  • Vector Semantics
  • Count-based Distributed Representations (tf-id, PMI)
  • Sparse vs. Dense Representation (SVD/PCA)
  • Word Embeddings (Word2Vec, GloVe, fastText)
  • Contextualized Word Embeddings (ELMo, BERT)
• Week 9 Transformers and Pretrained Models
  • Transformers (in details)
  • Pretrained Large Language Models
    • Encoders
    • Decoders
    • Encoder-Decoder
  • Parameter-efficient Finetuning Methods
    • Adapters
    • Prefix-Tuning
    • LoRA
  • Limitations, ethics, biases, environment
• Week 10 Neural Speech Recognition and Translation
  • Speech Processing Tasks
  • Automatic Speech Recognition
    • Model Design
    • Evaluation Metrics
  • Speech Translation
    • Model Design
    • Evaluation Metrics
  • Pre-trained Speech Transformers
    • Wav2vec 2
    • Whisper Speech Encoder
  • SUPERB Evaluation Benchmark
• Week 11 Advanced Topics in Large Language Models (I)
  • Prompting
  • In-context zero- or few-shot prompting
    • Emergent in-context abilities
    • Chain-of-thought Prompting
  • Instruction-Tuning Large Language Models
  • Alignment with Human Feedback (PPO and DPO)
    • Policy Gradient
    • Reward Model to Rank
  • A systematic limitation of LLMs
    • System 1 and System 2 Modes of Reasoning
  • Open-source LLMs
  • Evaluation Challenges
• Week 12 Advanced Topics in Large Language Models (II)
  • LLM Augmentations
    • Retrieval Augmented Generation (RAG)
    • Tool-Augmentation
    • Self-Correction
  • Language Agents
    • Training-free: ReAct, Reflexion, Critic, LATS
    • Fine-tuned: FireAct, TORA, Eurus
  • Latest Trends

CE/Lecturer, Data Analysis for Semi-structured Data (FIT5212, 2022, 2023, 2024). Syllabus:

• Week 1 Intro to semi-structured data
• Week 2 Text representation
  • Word and Document Representations (Count Vector, TF-IDF, LSA)
  • Basic Embedding Models (Word2vec, GLoVe)
  • Contextualized Embedding Models (ELMo, BERT)
• Week 3 Text classification
  • Data Preparation
  • Basic Classifier 
    • Logistic Regression Model and Cross-Entropy Loss Function
  • Neural Network-based Classifier
  • BERT-style Classification
• Week 4 Text clustering and matrix factorisation
  • Clustering Examples
  • Mixture Models
  • Topic Models (LDA)
  • Matrix Factorisation
• Week 5 Text generation
  • ​​​​​​​Neural Language Models
  • Sequence-to-Sequence models
  • Neural Machine Translation
  • Attention mechanism
  • Other language generation tasks
  • Evaluation
• Week 6 Pretrained language models
  • ​​​​​​​Introduction to Transformers
  • Pretrained Language Models
    • Pre-trained Encoder-Decoders (T5 and BART)
    • Pre-trained Encoders (BERT family)
    • Pre-trained Decoders (GPT family)
  • Finetuning Pretrained Language Models
• Week 7 Recommender systems (1) 
  • ​​​​​​​Introduction to recommender systems
  • Various Recommender Systems in Real-Life Applications
  • Content Based Recommendation Systems
  • Collaborative Filtering Algorithms
• Week 8 Recommender systems (2)
  • ​​​​​​​Matrix Factorisation for Recommender Systems
  • Neural Collaborative Filtering
  • Neural Matrix Factorisation
• Week 9 Graph clustering 
  • ​​​​​​​Graph Basics
  • K-means Graph Clustering
  • Spectral Clustering
  • Graph Clustering Evaluation
• Week 10 Graph representation learning
  • ​​​​​​​Network Embedding Principle
  • Random Walk Based Approaches
  • Matrix Factorization Approaches
  • Evaluation
• Week 11 Graph Neural Networks
  • ​​​​​​​Disadvantages of shallow embedding for graph data
  • Connections of graph neural networks to traditional deep learning
  • Basic idea of graph neural networks
  • Graph Convolutional Network (GCN)
  • GraphSage Network (GraphSage)
  • Graph Attention Network (GAT)
  • Application of graph neural networks
• Week 12 Knowledge Graphs and Text
  • ​​​​​​​Knowledge Graph construction
  • Examples of Knowledge Graphs
  • Knowledge-intensive tasks
  • Infusing external knowledge into text models

UCL.
Co-lecturer, Applied Machine Learning II (ELEC0135, 2020).
Module Lead, Emerging Topics in Integrated Machine Learning Systems (ELEC0139, 2020).

University of Cambridge.
Lecturer, Quantitative Methods in Analyzing Linguistic Data (QMALD, 2019).
Guest Lecturer, Computational Linguistics (LI18, 2019).