Text To Speech
Dong Wang, Zhiyong Zhang
We are interested in a flexible syntehsis based on neural model . The basic idea is that since the neural model can be traind with multiple conditions, we can treat speaker and emotion as the conditional factors. We use the speaker vector and emotion vector as addiiontal input to the model, and then train a single model that can produce sound of different speakers and different emotions.
In the following experiments, we use a simple DNN architecture to implement the training. The vocoder is WORD.
The first step is mono-speaker systems. We trained three systems: a female, a male and a child, each with a single network. The performance is like the ofllowing.
Now we combine all the data from male, female and child to train a single model.
The first experiment is that the data are blindly combined, without any indicator of speakers.
- Female & Male
- Female & Child
- Male & Child
Now we use speaker vector as an indicator of the speaker trait.
- Specific person
Firstly, use the speaker fector to specifiy a particular person:
- Interpolate of different person
Now let's produce interpolated voice by interpolating two speakers: female and amle.
- Female & Male with different ratio
- (1) 0.0:1.0
- (2) 0.1:0.9
- (3) 0.2:0.8
- (4) 0.3:0.7
- (5) 0.4:0.6
- (6) 0.5:0.5
- (7) 0.6:0.4
- (8) 0.7:0.3
- (9) 0.8:0.2
- (10) 0.9:0.1
- (11) 1.0:0.0
Using emotion vectors can specify which emotio to use, and the emotion can be also interpolated.
- Specific emotion
- Interpolation emotion
- Angry & neutral with different ratio
Finally, all the data (different speakers and different emotions) are combined together. Note that only the child voice has different emotions of training data. We hope that this emotion can be learned so that we can generate voice of other speakers with emotion, although they do not have any training data with emtoions.
We compare the different implementation of mlpg AS merlin does(mlpg.py and fast_mlpg.py). There are three implementations:
- mlpg: As mlpg.py while compute all the dimension of delta features(including lf0/bap/mgc, the dim is 1/5/60 respectively)
- mlpg-lossy: Wrong implementation of mlpg.py by only considering the first dimension of global co-variance.
- fast-mlpg: As fast_mlpg.py in merlin.
- Computation Time(Estimation)
alg. | lf0(dim=1) | bap(dim=5) | mgc(dim=60) mlpg-lossy | 100000 | 130000 | 160000 mlpg | 130000 | 500000 | 6200000 fast-mlpg | 60000 | 300000 | 3580000 avg-rate | 1:1.3:0.6 | 1:4:2+ | 1:40:20+
- Synthesis waves