# Goruut Phonemization System - Documentation

## Overview

Goruut is an advanced open-source phonemization system that provides bidirectional conversion between written text and IPA phonetic representations. Designed for both research and production use, it offers:

- High-accuracy phonemization for 140 languages
- Compact, efficient models that run on CPU
- Self-hostable architecture with no external dependencies
- Support for both word-level and sentence-level processing

## Core Capabilities

### 1. Phonemization (G2P)

Converts orthographic text to International Phonetic Alphabet (IPA) representations. Handles:

- Regular vocabulary words
- Out-of-vocabulary words through learned patterns
- Language-specific phonetic rules
- Homograph disambiguation (in supported languages only)
- Tonal languages

### 2. Dephonemization (P2G)

Reverse process that converts IPA back to written text. Features:

- Bidirectional consistency with phonemization
- Handling of multiple possible orthographic representations
- Context-aware disambiguation

### 3. Advanced Word Analysis

- Pronunciation explanation for OOV words
- Word decomposition explaning applied phonetic rules

## Language Support

Goruut provides specialized models for:

### Major Language Families

- Indo-European (English, Spanish, Hindi, Russian, ...)
- Sino-Tibetan (Mandarin, ...)
- Afro-Asiatic (Arabic, Hebrew, ...)
- Uralic (Finnish, Hungarian, ...)
- And 20+ other language families

### Special Features

- **English**: Full homograph support (e.g., "read" → /ɹiːd/ or /ɹɛd/)
- **Chinese**: Handles character input
- **Arabic**: Supports diacritic restoration
- **Dialects**: Separate models for major dialect groups

## Technical Architecture

### Learning System

Goruut uses a hybrid approach:

1. **Statistical Grammar Induction**
    - Automatically learns language-specific G2P/P2G rules
    - Builds longest prefix finite-state-transducer grammar from training data
    - Identifies productive morphological patterns
    - Generates an alignment of words and the phonetic representation for many words in the lexicon

2. **Hashtron Transformers**
    - Compact weightless networks using integer arithmetic
    - Specialized architectures for different tasks:
        - Phonemization (8-phoneme window)
        - Dephonemization (8-phoneme window)
        - Homograph disambiguation (8-word window)

### Model Specifications

| Component               | Layers | Attention Heads | Context Window      | Model Size |
|-------------------------|--------|-----------------|---------------------|------------|
| Phonemization           | 6      | 4               | 8 phonemes          | ~1.2MB     |
| Dephonemization         | 6      | 4               | 8 phonemes          | ~1.2MB     |
| Homograph Resolver      | 6      | 4               | 8 words             | ~100.0MB (dataset size specific) |

## Training System

### Data Requirements

For each language, Goruut needs:

1. **Core Lexicon (lexicon.tsv)**
    - Raw word-to-IPA mappings
    - Minimum: 10,000 entries recommended
    - Format: `word<tab>pronunciation`

2. **Aligned Subset (clean.tsv)**
    - High-quality aligned pairs 
    - Automatically generated by the system from `lexicon.tsv`
    - Used for model training
    - Typically 20%-80% of core lexicon

3. **Homograph Data (multi.tsv) [Optional]**
    - Sentence-level examples
    - Format: `sentence<tab>pronunciation`

### Training Process

1. **Grammar Induction**  
   `./study_language.sh french` (in `cmd/analysis2` folder)

2. **Data Preparation and Alignment**  
   `./clean_language.sh french` (in `cmd/analysis2` folder)
   
3. **Coverage Evaluation**
   `go build && ./coverage.sh french` (in `cmd/backtest` folder)
   
4. **Transformer Training**  
   `./train_language.sh french --maxpremodulo [NUMBER]` (in `cmd/analysis2` folder, NUMBER is set to 5 times the cleaning complexity)

5. **Homograph Transformer Training (optional)**  
   `./train_homograph.sh french` (in `cmd/backtest` folder)

## API Reference

### HTTP Endpoints

1. **Phonemize Sentence**  
   POST /tts/phonemize/sentence  
   Content-Type: application/json
   
## Deployment Options

### Docker Quickstart

1. Clone repository  
`git clone https://github.com/neurlang/goruut.git`  
`cd goruut`

2. Build and run  
`docker compose up -d --build`

3. Verify  
`curl -X POST http://localhost:18080/tts/phonemize/sentence -H "Content-Type: application/json" -d '{"Language":"English","Sentence":"Test sentence"}'`

### Bare Metal Installation

1. **Requirements:**
 - Go 1.22+
 - Python 3.8+ (for inference only)
 - 2GB RAM minimum

2. **Installation:**  

 - `cd ./cmd/goruut`  
 - `go build`  
 - `./goruut --configfile ../../configs/config.json`

## Performance Characteristics

| Operation               | Initial Latency | Average latency | Memory Usage |
|-------------------------|-----------------|-----------------|--------------|
| Phonemization           | 236 ms/sentence | 2 ms/sentence   | 100MB        |
| Dephonemization         | 228 ms/sentence | 1 ms/sentence   | 100MB        |

## Contribution Guide

### Adding New Languages

1. Create language directory in [neurlang/dataset](https://github.com/neurlang/dataset) same as the ones in `dicts/` used for training
2. Add training data:
 - lexicon.tsv (required)
 - clean.tsv (recommended)
 - multi.tsv (optional for homographs)
3. Submit pull request to [neurlang/dataset](https://github.com/neurlang/dataset)

### Improving Existing Models

1. Fork repository
2. Train improved models using:  
`./cmd/analysis2/train_language.sh [LANGUAGE] --maxpremodulo [NUMBER]`
3. Include validation results
4. Submit PR with new weight files to [neurlang/goruut](https://github.com/neurlang/goruut)

## Limitations and Roadmap

### Current Limitations

- No built-in number pronunciation
- Limited handling of abbreviations
- No automatic language detection

### Planned Features

- Numerics expression handling
- Abbreviations
- Improved OOV learning (reinforcement learning)
- Enhanced dialect blending

## Frequently Asked Questions

**Q:** How accurate is the English homograph handling?  
**A:** Current models achieve 85% accuracy on the Google Homograph Benchmark.

**Q:** Can I use this commercially?  
**A:** Yes, Goruut is in general licensed under MIT. However keep in mind that some datasets (Tibetan) that we train permit only non commercial use. Consult a lawyer if needing the model trained on those languages.

**Q:** What's the smallest language model?  
**A:** Most models are 0.8-1.5MB. The smallest is probably Esperanto at ~60KB.

**Q:** How often are models updated?  
**A:** We release quarterly updates with improved models.


### Technical Implementation

**Q: What algorithms/models power the phonemization?**  
A: Goruut uses a hybrid system combining Statistical Grammar Induction and Hashtron Transformers (compact weightless neural networks). The architecture includes specialized 6-layer transformers for phonemization, dephonemization, and homograph disambiguation.

**Q: How are language models structured?**  
A: Each language has forward (`weights6.json.zlib`) and reverse (`weights6_reverse.json.zlib`) model files containing transformer weights. These are trained separately for G2P and P2G tasks.

### Data & Training

**Q: How are new languages added?**  
A: By submitting to `neurlang/dataset` with:
- `lexicon.tsv` (raw word-IPA pairs)
- `clean.tsv` (aligned high-quality subset)
- Optional `multi.tsv` for homographs

**Q: What's the training workflow?**  
A: Four-stage pipeline:
1. Grammar induction (`study_language.sh`)
2. Data alignment (`clean_language.sh`) 
3. Coverage evaluation (`coverage.sh`)
4. Model training (`train_language.sh`)

### Usage

**Q: How to use as a library?**  
A: Import `github.com/neurlang/goruut/lib` in Go code. The library exposes phonemization/dephonemization functions with language-specific contexts. In python, use `pygoruut`.

**Q: HTTP service setup?**  
A: Run the binary with a config file:
```bash
./goruut --configfile configs/config.json
```
Then query endpoints like POST /tts/phonemize/sentence

### Performance

**Q: What accuracy metrics are available?**  
A: The system generates:
- `coverage_*.txt` files showing lexicon coverage percentages
- Validation reports comparing model performance on test sets
- Homograph disambiguation accuracy (e.g., 85% for English)

**Q: How are models evaluated during training?**  
A: Through:
1. Automatic generation of `weights*.best` files (best-performing snapshots)
2. Alignment quality scores in cleaning reports
3. Out-of-vocabulary word handling tests

### Contributing

**Q: How can I improve existing language models?**  
A: Three-step process:
1. Fork the repository
2. Retrain models using:  
   `./train_language.sh [LANGUAGE] --maxpremodulo [NUMBER]`
3. Submit pull request with new weight files

**Q: What's needed to add a new language?**  
A: Minimum requirements:
- `lexicon.tsv` (10,000+ word-IPA pairs)
- `clean.tsv` (aligned subset)
- Optional `multi.tsv` for homographs
- Must follow existing directory structure

**Q: How are contributions validated?**  
A: All submissions are tested via:
- Coverage verification scripts
- Alignment quality checks
- Benchmark comparisons against existing models

### Customization

**Q: Can I adjust phoneme representations for specific use cases?**  
A: IPA Flavors mechanism exists (via configuraton) to translate to a custom IPA-like phone set. The default config file contains an example how to output espeak phones using the system.

### Limitations

**Q: Can it process mixed-language text?**  
A: Yes, but not automatically - input must be specific to the user-selected languages in language priority order. Automatic language detection is a planned feature.

**Q: How does it handle rare/unseen words?**  
A: Uses learned phonetic patterns, but accuracy declines for:
- Highly irregular pronunciations
- Words with no similar training examples
- Obscure proper nouns

### Comparison

**Q: How does this compare to eSpeak/Festival?**  
A: Key differentiators:

| Feature          | Goruut                     | eSpeak/Festival          |
|------------------|----------------------------|--------------------------|
| Architecture     | Hashtron Transformers      | Rule-based               |
| Bidirectional    | Yes (G2P + P2G)            | G2P only                 |
| Model Size       | 0.8-1.5MB per language     | Larger                   |
| Homographs       | Context-aware (85% acc.)   | Limited handling         |
| Tonal Languages  | Full support               | Basic support            |
| Self-hostable    | Yes, no external deps      | Depends on system        |


## Support and Community

- GitHub Discussions: [https://github.com/neurlang/goruut/discussions](https://github.com/neurlang/goruut/discussions)
- Issue Tracker: [https://github.com/neurlang/goruut/issues](https://github.com/neurlang/goruut/issues)
- Contributor Chat: Discord invite in README.md

## License

MIT - See LICENSE file for complete terms.
See [neurlang/dataset](https://github.com/neurlang/dataset) regarding the licenses for the training data used.
