Month in 4 Papers (March 2024)

This series of posts is designed to bring you the newest findings and developments in the NLP field. I’ll delve into four significant research papers each month, offering a comprehensive summary. Be sure to visit my blog regularly or subscribe to my newsletter for monthly updates. Let’s dive in!

Extend Context Size

📝 Soaring from 4K to 400K: Extending LLM’s Context with Activation Beacon [paper] [code]

This paper presents the concept of employing beacon tokens to reduce the size of activations. Their method increases the context length by 100 without compromising the model’s performance. The approach was implemented by segmenting the input and placing a beacon [bcn] token at the end of each segment. This token is designed to learn to compress the activation from preceding tokens into a more compact form, independent of length.

Then, the model will employ these beacon tokens during the inference phase to generate key/value vectors. While it’s accurate that inference time will grow linearly, the critical point is the constant memory usage and perplexity in relation to expanding the context length. The findings indicate that by utilizing…