# Adding a Language to Goruut

## Prerequisites

Before you begin, ensure you have the following:

- **Go** installed on your system. You can install it using:
  - Linux: `sudo apt-get install golang`
  - Windows: Download and install from [Golang's official website](https://go.dev/).
- **Git Bash** installed on Windows (if you're using Windows). Download it from [Git's official website](https://git-scm.com/downloads).
- A dictionary of the language you want to add, preferably in a format that can be converted to a TSV file.

## Roadmap

Adding a language to Goruut involves several steps, as outlined below.

![Adding a langauge to Goruut](adding_a_language.drawio.svg)

- Step 1: Create the Language Folder
- Step 2: Prepare the `lexicon.tsv` File, which contains your language's words and sentences
- Step 3: Create the `language.json` File, inside which the Map is automatically generated by the system
- Step 4: Run `study_language.sh` to study the prefix grammar (the `Map` of the tokenizer)
- Step 5: Run `clean_language.sh` to align words and generate the longest prefix form grammar
- Step 6: Run `train_language.sh` to train the G2P engine model
- Step 7: Add Glue Code (`language.go`) to embed your language models
- Step 8: Modify `dicts.go` to integrate your new language
- Step 9: Backtest the Model to see how it performs, or test it using the user interface

## Step 1: Create the Language Folder

Create a folder in `dicts/` according to the name of your language. This folder will be referred to as the language folder.

If your language does not use spaces as separators (e.g., Chinese, Japanese, Tibetan), it is considered a **padspace** language.

## Step 2: Prepare the `lexicon.tsv` File

Create a `lexicon.tsv` text file in the language folder. This is a TSV file that serves as the dictionary, mapping language words (or sentences, for padspace languages) to their IPA pronunciations.

### Example `lexicon.tsv` Content (Romanian Language):

```tsv
frumos	fruˈmos
mâncare	mɨnˈkare
apă	ˈapə
om	om
femeie	feˈmeje
dragoste	ˈdraɡoste
copil	koˈpil
floare	ˈfloare
pădure	pəˈdure
soare	ˈsoare
```

For multiple IPA pronunciations of the same word (allowed by the system), use multiple rows:

```tsv
înțelele	ɨntseˈledʒe
înțelele	ɨntseˈleʒe
```

**Padspace languages** must include both whole sentences and individual words in their `lexicon.tsv` file.

## Step 3: Create the `language.json` File

Use this default template and modify it according to your language:

```json
{
  "Map": {
    "a": ["a"],
    "j": ["ʒ"]
  },
  "SrcMulti": null,
  "DstMulti": null,
  "SrcMultiSuffix": null,
  "DstMultiSuffix": ["ː"],
  "DropLast": null,
  "PrePhonWordSteps": [
    { "ToLower": true }
  ]
}
```

## Step 4: Run `study_language.sh`

1. Navigate to `cmd/analysis2`.
2. Run `go build` to compile the `analysis2` program.
3. Run `./study_language.sh <language>` in Git Bash.
4. If using a padspace language, add the `-padspace` parameter.
5. Adjust the `--rowlossimportance` hyper-parameter to tweak rule learning (default: 5).

## Step 5: Run `clean_language.sh`

1. Compile the `dicttomap` executable: `go build` in `cmd/dicttomap`.
2. Run `./clean_language.sh <language>` in Git Bash.
3. For padspace languages, use the `-padspace` parameter.

## Step 6: Run `train_language.sh`

1. Clone `https://github.com/neurlang/classifier` next to the `goruut` repo folder uing git.
2. Compile the programs in `cmd/train_phonemizer` and `cmd/backtest` (`go build`).
3. Run `train_language.sh <language> -maxpremodulo <value>` (recommended: 5 times the cleaning complexity).
4. If starting a new training (not finetuning), use `-overwrite`. **With** `-overwrite`, the old model will be **deleted**.
5. For padspace languages, add `-padspace`.

## Step 7: Add Glue Code (`language.go`)

1. Copy `language.go` from another language and place it in your folder.
2. Modify `package otherlanguage` to `package <yourfoldername>`.
3. Ensure `weights6.json.lzw` is embedded.
4. Create blank files for the other files (`missing.tsv`) which are embedded to bypass `go build`.

## Step 8: Modify `dicts.go`

1. In `dicts.go`, add an import statement referring to your language folder.
2. In `dicts.go`, modify the switch statement:

```go
case "UserFriendlyLanguageName":
    return yourlanguage.Language.ReadFile(lzw(filename))
```

3. In `dicts.go`, update `LangName(...)` to include your language folder.

## Step 9: Backtest the Model

1. Navigate to `cmd/backtest`.
2. Compile the program: `go build`.
3. Run `./backtest -langname <language>`.
4. Review success rate and adjust hyperparameters as needed.

## Troubleshooting

### Issue: `./analysis2: No such file or directory`

**Solution:** Ensure `go build` was executed in `cmd/analysis2`. The same solution applies for missing `backtest` / `dicttomap` in their respective directories.

### Issue: `language.json`'s `Map` (e.g. the grammar) is not being created

**Solution:** Check if `lexicon.tsv` uses tab separators and contains sample mappings, check that you have at least two letters from the language in `Map` to kick-start the training.

## Glossary

- **Padspace language:** A language that does not use spaces in it's written form (e.g., Chinese, Japanese).
- **TSV file:** A tab-separated values UTF-8 file for data storage.
- **IPA:** International Phonetic Alphabet for pronunciation.

## External Resources

- [Golang Installation Guide](https://go.dev/doc/install)
- [Git Bash Installation Guide](https://git-scm.com/downloads)
- [IPA Chart](https://www.internationalphoneticassociation.org/content/ipa-chart)

## Feedback

If you encounter any issues, please open an issue on [GitHub](https://github.com/neurlang/goruut/issues) or reach out to the community for support.


