Languedoc

=========

Language identification library based on ["N-Gram-Based Text Categorization"](https://www.cis.lmu.de/~stef/seminare/sprachenidentifizierung/cavnar_trenkle.pdf) by Cavnar and Trenkle.

## Usage

1. Create a directory `data`, with a subdirectory for each target language. Fill with your training data.

2. Run `PYTHONPATH=src/ python3 train.py`. It will create a `models.json.gz` file.

```python

import languedoc

language = languedoc.identify("A text you want to identify.")

```

It will output the identifier that you used as the subdirectory name in step 1, based on the closest match between n-gram frequencies.

## Accuracy

Below is the training script output from my training data, from seven major European languages. It is worth noting that the crossvalidation iterates through all languages, for each creates five models and for each creates ten tests of every length of 8, 16, 32 and 64 characters. If we count the misidentified samples, we can see that the tiny 8 character samples have 84% success rate, 16 chars rise to 96% and for 32 and longer there are no errors at all.

```