Category: Tools

  • NLP pipeline for Croatian and Serbian

    A Python module comprising of a tokeniser, a part-of-speech/MSD tagger, a lemmatiser, a dependency parser, and a named entity recognizer for most South Slavic languages. For Croatian and Serbian there are models for processing standard and Internet non-standard texts. The estimated accuracy of morphosyntactic tagging for this tool is ~94%, while for lemmatisation the accuracy is ~99%. Dependency parsing has an labeled attachment score of ~0.9, while named entity recognition achieves a micro-F1 of ~0.9.

    Author
    Nikola Ljubešić
    Publications
    The experiments yielding this pipeline have been described in the following paper: Nikola Ljubešić and Kaja Dobrovoljc (2019). What Does Neural Bring? Analysing Improvements in Morphosyntactic Annotation and Lemmatisation of Slovenian, Croatian and Serbian. Proceedings of the 7th Workshop on Balto-Slavic Natural Language Processing. Florence, Italy. pp. 29-34. [Link] [.bib]

  • Stemmers for Serbian and Croatian: SCStemmers

    This package is a Java reimplementation of four previously published stemming algorithms for Serbian and Croatian:

    • The greedy and the optimal subsumption-based stemmer for Serbian, by Vlado Kešelj and Danko Šipka
    • A refinement of the greedy subsumption-based stemmer, by Nikola Milošević
    • A “Simple stemmer for Croatian v0.1”, by Nikola Ljubešić and Ivan Pandžić

    All the stemmers expect the input text to be formatted in UTF-8. Their outputs are also UTF-8 encoded.

    Author
    Vuk Batanović
    Availability
    The package and a more extensive documentation can be downloaded from the SCStemmers GitHub repository.
    Publications

    The SCStemmers package was introduced in:

    Vuk Batanović, Boško Nikolić, Milan Milosavljević (2016). Reliable Baselines for Sentiment Analysis in Resource-Limited Languages: The Serbian Movie Review Dataset. Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016), pp. 2688-2696, Portorož, Slovenia. [Link] [.bib]

    The original papers describing each implemented stemming algorithm are:

    • For the greedy and the optimal subsumption-based stemmer for Serbian: Vlado Kešelj, Danko Šipka (2008). A Suffix Subsumption-Based Approach to Building Stemmers and Lemmatizers for Highly Inflectional Languages with Sparse Resources. Infotheca 9(1-2), pp. 23a-33a. [Link]
    • For the refinement of the greedy subsumption-based stemmer: Nikola Milošević (2012). Stemmer for Serbian language. arXiv preprint arXiv:1209.4471. [Link]
    • For the “Simple stemmer for Croatian v0.1”: Nikola Ljubešić, Damir Boras, Ozren Kubelka (2007). Retrieving Information in Croatian: Building a Simple and Efficient Rule-Based Stemmer. Digital Information and Heritage, pp. 313–320. [Link]

  • Croatian and Serbian lemmatiser [legacy]

    This tool is considered a legacy tool as the NLP pipeline achieves better results on the same task, but is not available as a web service yet.

    A tool for automatic lemmatisation (returning the base or dictionary form of an inflected word). The tool looks up the hrLex/srLex lexicons and uses a predictive model for lemmatising OOVs (out of vocabulary words) which was trained on available corpora and lexicons.

    Author
    Nikola Ljubešić
    Availability
    The lemmatiser is freely available in three forms:
    1. For local use, the code and models of the lemmatiser can be downloaded from this GitHub repository.
    2. The lemmatiser web service can be used online, via our web interface that can be found here.
    3. Our web service can be accessed from of our Python library, which can also be downloaded from the CLARIN.SI GitHub repository. Instructions on how to install the ReLDI library from GitHub can be found here (in Serbian). Alternatively, the easiest way to install it is through PyPI from the command line interface. (Detailed instructions also on GitHub.)

    The third option, i.e. using the ReLDI Python library, is most recommended for handling larger amounts of data.

  • Diacritic restoration tool

    A tool for automatic diacritic restoration on text with potentially missing diacritics (e.g. it turns kuca into kuća if necessary). Reported accuracy of the tool: 99.5% on standard language and 99.2% on non-standard language.

    Authors
    Nikola Ljubešić, Tomaž Erjavec, Darja Fišer
    Availability
    The tool is freely available in two forms:
    1. The code and models of the tool can be downloaded from this GitHub repository.
    2. Our web service can be accessed from of our Python library, which can also be downloaded from the CLARIN.SI GitHub repository. Instructions on how to install the ReLDI library from GitHub can be found here (in Serbian). Alternatively, the easiest way to install it is through PyPI from the command line interface. (Detailed instructions also on GitHub.)

    The second option, i.e. using the ReLDI Python library, is most recommended for handling larger amounts of data.

    Publications
    Nikola Ljubešić, Tomaž Erjavec, and Darja Fišer (2016). Corpus-based diacritic restoration for south slavic languages. Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16). Portorož, Slovenia. [Link] [.bib]

  • Croatian and Serbian tokeniser [legacy]

    This tool is considered a legacy tool as the NLP pipeline achieves better results on the same task, but is not available as a web service yet.

    A tool for automatic tokenisation (dividing text into words and sentences). It was engineered through iterative runs on representative datasets and features modes for both standard and non-standard language.

    Authors
    Nikola Ljubešić, Tomaž Erjavec
    Availability
    The tokeniser is freely available in three forms:
    1. For local use, the tokeniser can be downloaded from this GitHub repository.
    2. The tokeniser can be used online, via our web interface that can be found here.
    3. Our web service can be accessed from of our Python library, which can also be downloaded from the CLARIN.SI GitHub repository. Instructions on how to install the ReLDI library from GitHub can be found here (in Serbian). Alternatively, the easiest way to install it is through PyPI from the command line interface. (Detailed instructions also on GitHub.)

    The third option, i.e. using the ReLDI Python library, is most recommended for handling larger amounts of data.

  • Croatian and Serbian part of speech (POS) and morphosyntactic (MSD) tagger [legacy]

    This tool is considered a legacy tool as the NLP pipeline achieves better results on the same task, but is not available as a web service yet.

    A tool for automatic annotation on the morphosyntactic level. It is capable of tagging both Croatian and Serbian as models for both languages are present in the tool.
    The tagger is based on the CRF algorithm trained on a 500,000-token Croatian training corpus and the hrLex/srLex lexicons for each respective language.

    The set of morphosyntactic tags used in the corpus follows the revised MULTEXT-East V5 tagset for Croatian and Serbian, available here.

    Accuracies calculated on test sets for each language:
    • Croatian: 92.53%
    • Serbian: 92.33%
    Author
    Nikola Ljubešić
    Availability
    The tagger is freely available in three forms:
    1. For local use, the code and models of the tagger can be downloaded from this GitHub repository.
    2. The tagger web service can be used online, via our web interface that can be found here.
    3. Our web service can be accessed from of our Python library, which can also be downloaded from the CLARIN.SI GitHub repository. Instructions on how to install the ReLDI library from GitHub can be found here (in Serbian). Alternatively, the easiest way to install it is through PyPI from the command line interface. (Detailed instructions also on GitHub.)

    The third option, i.e. using the ReLDI Python library, is most recommended for handling larger amounts of data.

    Publications
    The tagger and its construction process have been described in detail in the following paper:
    Nikola Ljubešić, Filip Klubička, Željko Agić, Ivo-Pavao Jazbec (2016). New Inflectional Lexicons and Training Corpora for Improved Morphosyntactic Annotation of Croatian and Serbian. Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16). Portorož, Slovenia. [Link] [.bib]