BERT is a well-known and powerful pre-trained “encoder” model. Let’s see how we can use it as a “decoder” to form an encoder-decoder architecture. The Transformer architecture consists of two main building blocks — encoder and decoder components — which we stack on top of each other to form a…

Is it possible to train a model with transformer architecture to learn generating summaries with different styles? While it’s true that deep learning (specifically transformer architecture) keeps pushing the SOTA scores, they have a kind of significant shortcoming. No, I am not talking about their memory usage! We know how…

A new state-of-the-art result for the text summarization task on both CNN/DM and XSum datasets. It uses contrastive learning to rank the candidates. A new state-of-the-art (SOTA) result for the abstractive text summarization task was published [1] shortly after I wrote about the SimCLS [2] (previous SOTA) paper. Interestingly, they…

Explain the paper that currently holds the state-of-the-art result in the abstractive text summarization. — This paper was published in June 2021, not relatively new research. (at least in the fast-paced machine learning universe) But, it has been on top of the abstractive summarization leaderboard that I follow for nine months, which is impressive by the same machine learning universe standards! They used a combination…

A handy library to load the datasets, easily manipulate them, and evaluate your results using implementations of well-known metrics. I’ve been using the Huggingface Datasets library for a while now. Not going to lie, I was skeptical at first, given my experience of using other libraries’ built-in dataset loading functionality…

This library lets you easily see which words contribute the most in terms of self-attention score in each layer and head to the output using the pre-trained RoBERTa model. The self-attention mechanism assigns a score to each token with respect to all other tokens in the context. The scoring tensor…

The family of GPT models keeps improving the quality of the generated text to the point that they are now being used to write blog posts. Artificial Intelligence (AI) advancements keep making many tasks easier and more manageable for us. Maybe it is time to use AI to take care…

Part 2 of the introductory series about training a Text Summarization model (or any Seq2seq/Encoder-Decoder Architecture) with sample codes using HuggingFace. This piece aims to give you a deeper understanding of the sequence-to-sequence (seq2seq) networks and how it is possible to train them for automatic text summarization. I will use…

An introductory story about the inference process in the Abstractive Text Summarization task (Seq2seq/Encoder-Decoder Architecture) with sample codes from HuggingFace. The first of a series! Update “December 14, 2021”: I published the 2nd part of the series that explains the training loop for a transformer-based encoder-decoder model. What is Automatic Text Summarization? It is a…