Customize Consent Preferences

We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.

The cookies that are categorized as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... 

Always Active

Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

No cookies to display.

Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

No cookies to display.

Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

No cookies to display.

Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.

No cookies to display.

Advertisement cookies are used to provide visitors with customized advertisements based on the pages you visited previously and to analyze the effectiveness of the ad campaigns.

No cookies to display.

Zyphra’s Zyda: A Massive 1.3T Language Model Dataset Competing with Pile, C4, and arXiv

Reading Time: < 1 minute

Zyphra Technologies has just unveiled Zyda, a groundbreaking dataset designed to revolutionize language model training. With an impressive 1.3 trillion tokens, Zyda is a carefully curated compilation of premium open datasets such as RefinedWeb, Starcoder, C4, Pile, Slimpajama, pe2so, and arxiv. According to Zyphra, their ablation studies have shown that Zyda outperforms the datasets it was built upon.

The company’s CEO, Krithik Puthalath, explained that Zyda was created to provide a high-quality dataset for training language models at an unprecedented scale. By combining and cleaning up existing datasets, Zyphra ensured that Zyda offers a unique and top-notch collection of tokens. The dataset underwent rigorous syntactic filtering and deduplication efforts to eliminate low-quality documents and remove duplicates, resulting in a dataset that stands out in the field.

RefinedWeb, Slimpajama, and StarCoder are the largest contributors to Zyda, with RefinedWeb accounting for 43.6% of the dataset. The meticulous curation process led to the discarding of approximately 40% of the initial dataset, reducing the token count to 1.3 trillion.

Developers can now access Zyda on Zyphra’s Hugging Face page, opening up new possibilities for smarter AI applications. With improved word predictions, text generation, and language translation capabilities, Zyda promises to streamline the development process and enhance the performance of language models.

If you’re eager to explore the potential of Zyda and delve into the world of advanced language modeling, Zyphra’s latest innovation is definitely worth checking out.

Taylor Swifts New Album Release Health issues from using ACs Boston Marathon 2024 15 Practical Ways To Save Money