Category: Datasets

Modern-Botok. Custom dictionary for modern Tibetan

Posted on by Xaver in Allgemein, Datasets, Outputs

Tsikchen.tsv is a customised dictionary to be integrated into the Tibetan tokenizer BoTok. BoTok can tokenize classical Tibetan text or traditional genres out of the box. However, since it depends on a dictionary for tokenization, it lacks capabilities for modern Tibetan, in particular, the language of modern newspapers published in the PRC or on the subcontinent. Adding this customised dictionary adds functionality for modern Tibetan to BoTok.

Erhard, F. Xaver & Kyogoku, Yuki. (2024). Modern-Botok. Custom dictionary for modern Tibetan (v0.1) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.14034747

The custom dictionary tsikchen was compiled from Christian Steinert’s collection and contains the following dictionaries:

  1. Grand Monlam Dictionary (default dictionary of Botok)
  2. Jim Valby
  3. Ives Waldo
  4. Dan Martin
  5. Tshig mdzod chen mo
  6. Dung dkar
  7. Tibetan Terminology Project

The Divergent Discourses project cleaned up and edited the resulting dictionary to the project’s requirements (removal of double entries, phraseologisms, ungrammatical entries, etc; addition of ca. 1000 personal and place names).

For its installation, see the Divergent Discourses’ modern-botok repository on github.

Tibetan Modern U-chen Print 0.1

Posted on by Xaver in Datasets, Models

Tibetan Modern U-chen Print 0.1 (TMUP 0.1) is the first Transkribus HTR model for printed Tibetan language publications in Uchen (དབུ་ཅན་ dbu can) script. It has been trained on texts that were published in the PRC between the 1950s and 1980s. The model was trained on 522 pages in 20 documents. The training set consists of 470 pages; the validation set consists of 52 (10%) automatically selected pages. No base model was used. The model was developed by Franz Xaver Erhard (Leipzig University) and Xiaoying 笑影 (Leipzig University) for the Divergent Discourses project (DFG/AHRC).

The model is publicly available within the Transkribus environment. You can view and test the model at

https://readcoop.eu/model/tibetan-modern-u-chen-print/

Details on the model and the Ground Truth of the training set can be viewed on the Transkribus-Site of the Divergent Discourses project:

https://app.transkribus.org/sites/uchan

The training set to the model – consisting of the image files (jpg) and the corresponding Transkribus pageXML files – is available for download from:

Erhard, Franz Xaver, Xiaoying 笑影, Barnett, Robert, Hill, Nathan W., 2024. Tibetan Modern U-chen Print (TMUP) 0.1: Training Data for a Transkribus HTR Model for Modern Tibetan Printed Texts. https://doi.org/10.48796/20240313-000