Novel Language Resources for Hindi: An Aesthetics Text Corpus and a Comprehensive Stop Lemma List

This paper is an effort to complement the contributions made by researchers working toward the inclusion of non-English languages in natural language processing studies. Two novel Hindi language resources have been created and released for public consumption. The first resource is a corpus consisting of nearly thousand pre-processed fictional and nonfictional texts spanning over hundred years. The second resource is an exhaustive list of stop lemmas created from 12 corpora across multiple domains, consisting of over 13 million words, from which more than 200,000 lemmas were generated, and 11 publicly available stop word lists comprising over 1000 words, from which nearly 400 unique lemmas were generated. This research lays emphasis on the use of stop lemmas instead of stop words owing to the presence of various, but not all morphological forms of a word in stop word lists, as opposed to the presence of only the root form of the word, from which variations could be derived if required. It was also observed that stop lemmas were more consistent across multiple sources as compared to stop words. In order to generate a stop lemma list, the parts of speech of the lemmas were investigated but rejected as it was found that there was no significant correlation between the rank of a word in the frequency list and its part of speech. The stop lemma list was assessed using a comparative method. A formal evaluation method is suggested as future work arising from this study.


page 1

page 3


Accuracy of the Uzbek stop words detection: a case study on "School corpus"

Stop words are very important for information retrieval and text analysi...

LEXpander: applying colexification networks to automated lexicon expansion

Recent approaches to text analysis from social media and other corpora r...

LScDC-new large scientific dictionary

In this paper, we present a scientific corpus of abstracts of academic p...

Automatic Extraction of the Romanian Academic Word List: Data and Methods

This paper presents the methodology and data used for the automatic extr...

The Open corpus of the Veps and Karelian languages: overview and applications

A growing priority in the study of Baltic-Finnic languages of the Republ...

Frozen Binomials on the Web: Word Ordering and Language Conventions in Online Text

There is inherent information captured in the order in which we write wo...

Word Familiarity and Frequency

Word frequency is assumed to correlate with word familiarity, but the st...

Please sign up or login with your details

Forgot password? Click here to reset