Language Modeling (LM)


A statistical language model is a probability distribution over sequences of words. Given such a sequence, say of length m, it assigns a probability to the whole sequence. The language model provides context to distinguish between words and phrases that sound similar.

Overview

Generalized Language Models
Trend in large unsupervised pre-trained language models which have achieved amazing SOTA results on a variety of language tasks.
transformers attention bert elmo
Evaluation Metrics for Language Modeling
In this article, we will focus on traditional intrinsic metrics that are extremely useful during the process of training the language model itself.
language-modeling metrics bayesian-inference gaussian-processes

Tutorials

Controlling Text Generation with Plug and Play Language Models
This article discusses an alternative approach to controlled text generation, titled the Plug and Play Language Model (PPLM).
language-modeling huggingface uber-ai text-generation
Custom Classifier on Top of Bert-like Language Model
Take pre-trained language model and build custom classifier on top of it.
bert language-modeling pytorch pytorch-lightning
Transfer Learning with T5: the Text-To-Text Transfer Transformer
In the paper, we demonstrate how to achieve state-of-the-art results on multiple NLP tasks using a text-to-text transformer pre-trained on a large text ...
transformers t5 question-answering reading-comprehension
How to Train a New Language Model From Scratch Using Transformers
In this post we’ll demo how to train a “small” model (84 M parameters = 6 layers, 768 hidden size, 12 attention heads).
language-modeling transformers tokenizers huggingface
Evaluation of Text Generation: A Survey
Evaluation methods of natural language generation (NLG) and language modeling.
text-generation language-modeling natural-language-generation survey

Libraries

General
Transformers - Hugging Face
🤗 Transformers: State-of-the-art Natural Language Processing for TensorFlow 2.0 and PyTorch.
transformers huggingface attention bert
AllenNLP Interpret
A Framework for Explaining Predictions of NLP Models
interpretability explainability natural-language-processing api
Finetune: Scikit-learn Style Model Finetuning for NLP
Finetune is a library that allows users to leverage state-of-the-art pretrained NLP models for a wide variety of downstream tasks.
natural-language-processing finetuning pretraining transformers
How Big Should My Language Model Be?
Tool to explore language model training and optimize the compute costs.
language-modeling compute calculator huggingface
Table of Contents
Share a project
Share something you or the community has made with ML.
Topic experts
Share