DeepSeek AI Forgetting: Fast Text Compression & Data Generation

The traditional method for AI to process information, particularly text, involves converting every word into tokens. This process is resource-intensive and becomes increasingly expensive with longer texts, leading to what is described as the AI "choking" on the information. Humans, however, do not perfectly recall every detail; some information fades over time. This observation led scientists at DeepSeek to explore the concept of "artificial forgetting" for AI systems.

The Challenge of AI Memory

As AI systems engage in longer conversations or process more documents, the cost of operation escalates because the AI must retain all information. DeepSeek OCR proposes a solution: instead of storing raw text, convert the information into an image, then compress that image. This is analogous to JPEG compression for visual data but is specifically designed for learned, token-based data. This method drastically reduces the data size.

The Paradox of Compression and Information Loss

A natural concern with such high compression is the potential loss of information. However, DeepSeek OCR demonstrates surprising efficiency. At a 10x compression level, which means using 90% less space, over 96% of the original text can still be recovered. Pushing compression further, to nearly 20x, results in OCR accuracy dropping to 60%. While this represents a significant loss, it is still remarkably high given the extreme compression.

Beyond Traditional OCR: Compressing Memories

Initially, this work might seem like a sophisticated OCR system, taking handwritten notes or documents and converting them into text. It takes a small number of compressed vision tokens and expands them into a larger number of text tokens. However, this is not merely an application of old JPEG technology, which is over 30 years old. JPEG compresses pixels; DeepSeek's method compresses "memories." These memories are first transformed into a picture, and then that picture is compressed.

The Engineering Innovation

The core innovation lies in an encoder designed to handle very high-resolution documents without incurring exorbitant costs. This is achieved through a three-pronged approach:

1. Local Window Attention: The system first uses local window attention, meaning it doesn't scrutinize every single pixel with full focus.
2. 16x Convolutional Compressor: A 16x convolutional compressor further processes the data.
3. Global Visual Processing: Finally, it performs global visual processing.

In essence, the system "skims," "squeezes," and then "reads."

Generating New Data at Unprecedented Speeds

A significant implication of this cost-effective compression is its ability to generate new data, not just remember old data. The paper claims that this system can generate training data for Large Language Models (LLMs) and Vision-Language Models (VLMs) at a rate exceeding 200,000 pages per day on a single A100 GPU, demonstrating "insane speeds."

Limitations and Future Considerations

Despite its impressive capabilities, the paper acknowledges certain limitations. OCR alone is not sufficient to prove true text compression. A stress test suggests that the model might heavily rely on language priors rather than solely visual memory. This means that when an image is highly compressed, the system might not genuinely "see" the missing text but instead "guesses" the most probable words based on context, similar to an advanced autocomplete function.

An Open and Impactful System

DeepSeek's system is open-source and freely available, fostering the creation of training data that can benefit other open models. This development is seen as a significant step forward in AI research, pushing the boundaries of efficient information processing and data generation.