GitHub Typo Corpus collects misspellings and grammatical corrections from GitHub commits to create a comprehensive dataset for Natural Language Processing research. It aims to provide a resource for training and evaluating models designed to detect and correct typos, a common challenge in real-world text. The dataset includes over 350k edits across more than 15 languages, making it the largest of its kind.
The dataset is large-scale and multilingual, covering a significant number of languages beyond English. It provides detailed information about each edit, including the probability of it being a typo and the perplexity of the corrected text. The dataset's structured format facilitates easy analysis and integration with various NLP models and tools.
- Multilingual Support: Includes data from over 15 languages, with detailed breakdowns for English, Simplified Chinese, and Japanese.
- Detailed Edits: Provides information on typo probability, perplexity, and edit context (source and target text, file path, language).
- Structured Format: Data is stored in JSONL format, enabling easy parsing and integration with NLP pipelines.
- Source Code Included: Contains Python code for collecting repositories, commits, and edits from GitHub.
- Community Resource: Supports research in typo detection, correction, and related NLP tasks.
The project is actively maintained, with recent commits indicating ongoing development and updates. The accompanying documentation and source code are readily available, promoting transparency and community contribution. The last commit was made in December 2019, suggesting recent activity. The dataset is a valuable resource for researchers in the field of NLP.
Researchers and developers interested in typo detection, error correction, and natural language processing can benefit from this dataset. It addresses the need for a large, multilingual dataset of real-world misspellings, enabling the development of more robust and accurate typo correction models. The dataset offers a valuable resource for improving text quality and user experience in various applications.