nnsvs.acoustic_models
Sinsy-based models
ResSkipF0FFConvLSTM
- class nnsvs.acoustic_models.ResSkipF0FFConvLSTM(in_dim, ff_hidden_dim=2048, conv_hidden_dim=1024, lstm_hidden_dim=256, out_dim=199, dropout=0.0, num_lstm_layers=2, bidirectional=True, in_lf0_idx=300, in_lf0_min=5.3936276, in_lf0_max=6.491111, out_lf0_idx=180, out_lf0_mean=5.953093881972361, out_lf0_scale=0.23435173188961034, skip_inputs=False, init_type='none', use_mdn=False, num_gaussians=8, dim_wise=False)[source]
FFN + Conv1d + LSTM + residual/skip connections
A model proposed in Hono et al. [HHO+21].
- Parameters:
in_dim (int) – input dimension
ff_hidden_dim (int) – hidden dimension of feed-forward layer
conv_hidden_dim (int) – hidden dimension of convolutional layer
lstm_hidden_dim (int) – hidden dimension of LSTM layer
out_dim (int) – output dimension
dropout (float) – dropout rate
num_ls (int) – number of layers of LSTM
bidirectional (bool) – whether to use bidirectional LSTM or not
in_lf0_idx (int) – index of lf0 in input features
in_lf0_min (float) – minimum of lf0 in the training data of input features
in_lf0_max (float) – maximum of lf0 in the training data of input features
out_lf0_idx (int) – index of lf0 in output features
out_lf0_mean (float) – mean of lf0 in the training data of output features
out_lf0_scale (float) – scale of lf0 in the training data of output features
skip_inputs (bool) – whether to use skip connection for the input features
init_type (str) – initialization type
use_mdn (bool) – whether to use MDN or not
num_gaussians (int) – number of gaussians in MDN
dim_wise (bool) – whether to use MDN with dim-wise or not
Tacotron-based models
Duration-informed Tacotron (Okamoto et al. [OTSK19]) based models.
NonAttentiveDecoder
- class nnsvs.acoustic_models.NonAttentiveDecoder(in_dim=512, out_dim=80, layers=2, hidden_dim=1024, prenet_layers=2, prenet_hidden_dim=256, prenet_dropout=0.5, zoneout=0.1, reduction_factor=1, downsample_by_conv=False, postnet_layers=0, postnet_channels=512, postnet_kernel_size=5, postnet_dropout=0.0, init_type='none', eval_dropout=True, prenet_noise_std=0.0, initial_value=0.0)[source]
Non-attentive autoregresive model based on the duration-informed Tacotron
Duration-informed Tacotron Okamoto et al. [OTSK19].
Note
if the target features of the decoder is normalized to N(0, 1), consider setting the initial value carefully so that it roughly matches the value of silence. e.g., -4 to -10.
initial_value=0
works okay for large databases but I found that -4 or lower worked better for smaller databases such as nit-song070.- Parameters:
in_dim (int) – Input dimension.
out_dim (int) – Output dimension.
layers (int) – Number of LSTM layers.
hidden_dim (int) – Hidden dimension of LSTM.
prenet_layers (int) – Number of prenet layers.
prenet_hidden_dim (int) – Hidden dimension of prenet.
prenet_dropout (float) – Dropout rate of prenet.
zoneout (float) – Zoneout rate.
reduction_factor (int) – Reduction factor.
downsample_by_conv (bool) – If True, downsampling is performed by convolution.
postnet_layers (int) – Number of postnet layers.
postnet_channels (int) – Number of postnet channels.
postnet_kernel_size (int) – Kernel size of postnet.
postnet_dropout (float) – Dropout rate of postnet.
init_type (str) – Initialization type.
eval_dropout (bool) – If True, dropout is applied in evaluation.
initial_value (float) – initial value for the autoregressive decoder.
MDNNonAttentiveDecoder
- class nnsvs.acoustic_models.MDNNonAttentiveDecoder(in_dim=512, out_dim=80, layers=2, hidden_dim=1024, prenet_layers=2, prenet_hidden_dim=256, prenet_dropout=0.5, zoneout=0.1, reduction_factor=1, downsample_by_conv=False, num_gaussians=8, sampling_mode='mean', init_type='none', eval_dropout=True, prenet_noise_std=0.0, initial_value=0.0)[source]
Non-atteive decoder with MDN
Each decoder step outputs the parameters of MDN.
- Parameters:
in_dim (int) – input dimension
out_dim (int) – output dimension
layers (int) – number of LSTM layers
hidden_dim (int) – hidden dimension
prenet_layers (int) – number of prenet layers
prenet_hidden_dim (int) – prenet hidden dimension
prenet_dropout (float) – prenet dropout rate
zoneout (float) – zoneout rate
reduction_factor (int) – reduction factor
downsample_by_conv (bool) – if True, use conv1d to downsample the input
num_gaussians (int) – number of Gaussians
sampling_mode (str) – sampling mode
init_type (str) – initialization type
eval_dropout (bool) – if True, use dropout in evaluation
initial_value (float) – initial value for the autoregressive decoder.
BiLSTMNonAttentiveDecoder
- class nnsvs.acoustic_models.BiLSTMNonAttentiveDecoder(in_dim=512, ff_hidden_dim=2048, conv_hidden_dim=1024, lstm_hidden_dim=256, num_lstm_layers=2, out_dim=80, decoder_layers=2, decoder_hidden_dim=1024, prenet_layers=2, prenet_hidden_dim=256, prenet_dropout=0.5, zoneout=0.1, reduction_factor=1, downsample_by_conv=False, postnet_layers=0, postnet_channels=512, postnet_kernel_size=5, postnet_dropout=0.0, in_ph_start_idx: int = 1, in_ph_end_idx: int = 50, embed_dim=None, init_type='none', eval_dropout=True, prenet_noise_std=0.0, initial_value=0.0)[source]
BiLSTM-based encoder + NonAttentiveDecoder
The encoder is based on the arthitecture of the Sinsy acoustic model.
- Parameters:
in_dim (int) – Input dimension.
ff_hidden_dim (int) – Hidden dimension of feed-forward layers in the encoder.
conv_hidden_dim (int) – Hidden dimension of convolution layers in the encoder.
lstm_hidden_dim (int) – Hidden dimension of LSTM layers in the encoder.
num_lstm_layers (int) – Number of LSTM layers in the encoder.
out_dim (int) – Output dimension.
layers (int) – Number of LSTM layers.
hidden_dim (int) – Hidden dimension of LSTM.
prenet_layers (int) – Number of prenet layers.
prenet_hidden_dim (int) – Hidden dimension of prenet.
prenet_dropout (float) – Dropout rate of prenet.
zoneout (float) – Zoneout rate.
reduction_factor (int) – Reduction factor.
downsample_by_conv (bool) – If True, downsampling is performed by convolution.
postnet_layers (int) – Number of postnet layers.
postnet_channels (int) – Number of postnet channels.
postnet_kernel_size (int) – Kernel size of postnet.
postnet_dropout (float) – Dropout rate of postnet.
in_ph_start_idx (int) – Start index of phoneme features.
in_ph_end_idx (int) – End index of phoneme features.
embed_dim (int) – Embedding dimension.
init_type (str) – Initialization type.
eval_dropout (bool) – If True, dropout is applied in evaluation.
initial_value (float) – initial value for the autoregressive decoder.
BiLSTMMDNNonAttentiveDecoder
- class nnsvs.acoustic_models.BiLSTMMDNNonAttentiveDecoder(in_dim=512, ff_hidden_dim=2048, conv_hidden_dim=1024, lstm_hidden_dim=256, num_lstm_layers=2, out_dim=80, decoder_layers=2, decoder_hidden_dim=1024, prenet_layers=2, prenet_hidden_dim=256, prenet_dropout=0.5, zoneout=0.1, reduction_factor=1, downsample_by_conv=False, num_gaussians=8, sampling_mode='mean', in_ph_start_idx: int = 1, in_ph_end_idx: int = 50, embed_dim=None, init_type='none', eval_dropout=True, prenet_noise_std=0, initial_value=0.0)[source]
BiLSTM-based encoder + NonAttentiveDecoder (MDN version)
The encoder is based on the arthitecture of the Sinsy acoustic model.
- Parameters:
in_dim (int) – Input dimension.
ff_hidden_dim (int) – Hidden dimension of feed-forward layers in the encoder.
conv_hidden_dim (int) – Hidden dimension of convolution layers in the encoder.
lstm_hidden_dim (int) – Hidden dimension of LSTM layers in the encoder.
num_lstm_layers (int) – Number of LSTM layers in the encoder.
out_dim (int) – Output dimension.
layers (int) – Number of LSTM layers.
hidden_dim (int) – Hidden dimension of LSTM.
prenet_layers (int) – Number of prenet layers.
prenet_hidden_dim (int) – Hidden dimension of prenet.
prenet_dropout (float) – Dropout rate of prenet.
zoneout (float) – Zoneout rate.
reduction_factor (int) – Reduction factor.
downsample_by_conv (bool) – If True, downsampling is performed by convolution.
num_gaussians (int) – Number of Gaussians.
sampling_mode (str) – Sampling mode.
postnet_layers (int) – Number of postnet layers.
postnet_channels (int) – Number of postnet channels.
postnet_kernel_size (int) – Kernel size of postnet.
postnet_dropout (float) – Dropout rate of postnet.
in_ph_start_idx (int) – Start index of phoneme features.
in_ph_end_idx (int) – End index of phoneme features.
embed_dim (int) – Embedding dimension.
init_type (str) – Initialization type.
eval_dropout (bool) – If True, dropout is applied in evaluation.
initial_value (float) – initial value for the autoregressive decoder.
Tacotron-based F0 models
ResF0NonAttentiveDecoder
- class nnsvs.acoustic_models.ResF0NonAttentiveDecoder(in_dim=512, out_dim=1, layers=2, hidden_dim=1024, prenet_layers=2, prenet_hidden_dim=256, prenet_dropout=0.5, zoneout=0.1, reduction_factor=1, downsample_by_conv=False, scaled_tanh=True, in_lf0_idx=300, in_lf0_min=5.3936276, in_lf0_max=6.491111, out_lf0_idx=180, out_lf0_mean=5.953093881972361, out_lf0_scale=0.23435173188961034, init_type='none', eval_dropout=True)[source]
Duration-informed Tacotron with residual F0 prediction.
- Parameters:
in_dim (int) – dimension of encoder hidden layer
out_dim (int) – dimension of output
layers (int) – number of LSTM layers
hidden_dim (int) – dimension of hidden layer
prenet_layers (int) – number of pre-net layers
prenet_hidden_dim (int) – dimension of pre-net hidden layer
prenet_dropout (float) – dropout rate of pre-net
zoneout (float) – zoneout rate
reduction_factor (int) – reduction factor
downsample_by_conv (bool) – if True, downsample by convolution
scaled_tanh (bool) – if True, use scaled tanh for residual F0 prediction
in_lf0_idx (int) – index of lf0 in input features
in_lf0_min (float) – minimum of lf0 in the training data of input features
in_lf0_max (float) – maximum of lf0 in the training data of input features
out_lf0_idx (int) – index of lf0 in output features
out_lf0_mean (float) – mean of lf0 in the training data of output features
out_lf0_scale (float) – scale of lf0 in the training data of output features
init_type (str) – initialization type
eval_dropout (bool) – if True, use dropout in evaluation
MDNResF0NonAttentiveDecoder
- class nnsvs.acoustic_models.MDNResF0NonAttentiveDecoder(in_dim=512, out_dim=80, layers=2, hidden_dim=1024, prenet_layers=2, prenet_hidden_dim=256, prenet_dropout=0.5, zoneout=0.1, reduction_factor=1, downsample_by_conv=False, scaled_tanh=True, num_gaussians=4, sampling_mode='mean', in_lf0_idx=300, in_lf0_min=5.3936276, in_lf0_max=6.491111, out_lf0_idx=180, out_lf0_mean=5.953093881972361, out_lf0_scale=0.23435173188961034, init_type='none', eval_dropout=True)[source]
Duration-informed Tacotron with residual F0 prediction (MDN-version)
- Parameters:
in_dim (int) – dimension of encoder hidden layer
out_dim (int) – dimension of output
layers (int) – number of LSTM layers
hidden_dim (int) – dimension of hidden layer
prenet_layers (int) – number of pre-net layers
prenet_hidden_dim (int) – dimension of pre-net hidden layer
prenet_dropout (float) – dropout rate of pre-net
zoneout (float) – zoneout rate
reduction_factor (int) – reduction factor
downsample_by_conv (bool) – if True, downsample by convolution
scaled_tanh (bool) – if True, use scaled tanh for residual F0 prediction
num_gaussians (int) – number of Gaussian
sampling_mode (str) – sampling mode
in_lf0_idx (int) – index of lf0 in input features
in_lf0_min (float) – minimum of lf0 in the training data of input features
in_lf0_max (float) – maximum of lf0 in the training data of input features
out_lf0_idx (int) – index of lf0 in output features
out_lf0_mean (float) – mean of lf0 in the training data of output features
out_lf0_scale (float) – scale of lf0 in the training data of output features
init_type (str) – initialization type
eval_dropout (bool) – if True, use dropout in evaluation
BiLSTMResF0NonAttentiveDecoder
- class nnsvs.acoustic_models.BiLSTMResF0NonAttentiveDecoder(in_dim=512, ff_hidden_dim=2048, conv_hidden_dim=1024, lstm_hidden_dim=256, num_lstm_layers=2, dropout=0.0, out_dim=80, decoder_layers=2, decoder_hidden_dim=1024, prenet_layers=2, prenet_hidden_dim=256, prenet_dropout=0.5, zoneout=0.1, reduction_factor=1, downsample_by_conv=False, scaled_tanh=True, in_lf0_idx=300, in_lf0_min=5.3936276, in_lf0_max=6.491111, out_lf0_idx=180, out_lf0_mean=5.953093881972361, out_lf0_scale=0.23435173188961034, use_mdn=False, num_gaussians=4, sampling_mode='mean', in_ph_start_idx: int = 1, in_ph_end_idx: int = 50, embed_dim=None, init_type='none')[source]
BiLSTM-based encoder + duration-informed Tacotron with residual F0 prediction.
- Parameters:
in_dim (int) – dimension of encoder hidden layer
ff_hidden_dim (int) – Hidden dimension of feed-forward layers in the encoder.
conv_hidden_dim (int) – Hidden dimension of convolution layers in the encoder.
lstm_hidden_dim (int) – Hidden dimension of LSTM layers in the encoder.
num_lstm_layers (int) – Number of LSTM layers in the encoder.
out_dim (int) – dimension of output
layers (int) – number of LSTM layers
hidden_dim (int) – dimension of hidden layer
prenet_layers (int) – number of pre-net layers
prenet_hidden_dim (int) – dimension of pre-net hidden layer
prenet_dropout (float) – dropout rate of pre-net
zoneout (float) – zoneout rate
reduction_factor (int) – reduction factor
downsample_by_conv (bool) – if True, downsample by convolution
scaled_tanh (bool) – if True, use scaled tanh for residual F0 prediction
in_lf0_idx (int) – index of lf0 in input features
in_lf0_min (float) – minimum of lf0 in the training data of input features
in_lf0_max (float) – maximum of lf0 in the training data of input features
out_lf0_idx (int) – index of lf0 in output features
out_lf0_mean (float) – mean of lf0 in the training data of output features
out_lf0_scale (float) – scale of lf0 in the training data of output features
use_mdn (bool) – if True, use mixture density network for F0 prediction
num_gaussians (int) – number of gaussians in MDN
sampling_mode (str) – sampling mode in inference. “mean” or “random”
in_ph_start_idx (int) – Start index of phoneme features.
in_ph_end_idx (int) – End index of phoneme features.
embed_dim (int) – Embedding dimension.
init_type (str) – initialization type
Multi-stream models
MultistreamSeparateF0ParametricModel
- class nnsvs.acoustic_models.MultistreamSeparateF0ParametricModel(in_dim: int, out_dim: int, stream_sizes: list, reduction_factor: int, encoder: Module, mgc_model: Module, lf0_model: Module, vuv_model: Module, bap_model: Module, vib_model: Module | None = None, vib_flags_model: Module | None = None, in_rest_idx=1, in_lf0_idx=300, in_lf0_min=5.3936276, in_lf0_max=6.491111, out_lf0_idx=180, out_lf0_mean=5.953093881972361, out_lf0_scale=0.23435173188961034, lf0_teacher_forcing=True)[source]
Multi-stream model with a separate F0 prediction model
acoustic features: [MGC, LF0, VUV, BAP]
vib_model and vib_flags_model are optional and will be likely to be removed.
Conditional dependency: p(MGC, LF0, VUV, BAP |C) = p(LF0|C) p(MGC|LF0, C) p(BAP|LF0, C) p(VUV|LF0, C)
- Parameters:
in_dim (int) – Input dimension.
out_dim (int) – Output dimension.
stream_sizes (list) – List of stream sizes.
reduction_factor (int) – Reduction factor.
encoder (nn.Module) – A shared encoder.
mgc_model (nn.Module) – MGC prediction model.
lf0_model (nn.Module) – log-F0 prediction model.
vuv_model (nn.Module) – V/UV prediction model.
bap_model (nn.Module) – BAP prediction model.
in_rest_idx (int) – Index of the rest symbol in the input features.
in_lf0_idx (int) – index of lf0 in input features
in_lf0_min (float) – minimum value of lf0 in the training data of input features
in_lf0_max (float) – maximum value of lf0 in the training data of input features
out_lf0_idx (int) – index of lf0 in output features. Typically 180.
out_lf0_mean (float) – mean of lf0 in the training data of output features
out_lf0_scale (float) – scale of lf0 in the training data of output features
lf0_teacher_forcing (bool) – Whether to use teacher forcing for F0 prediction.
NPSSMultistreamParametricModel
- class nnsvs.acoustic_models.NPSSMultistreamParametricModel(in_dim: int, out_dim: int, stream_sizes: list, reduction_factor: int, lf0_model: Module, mgc_model: Module, bap_model: Module, vuv_model: Module, in_rest_idx=0, in_lf0_idx=51, in_lf0_min=5.3936276, in_lf0_max=6.491111, out_lf0_idx=60, out_lf0_mean=5.953093881972361, out_lf0_scale=0.23435173188961034, npss_style_conditioning=False, vuv_model_bap_conditioning=True, vuv_model_bap0_conditioning=False, vuv_model_lf0_conditioning=True, vuv_model_mgc_conditioning=False)[source]
NPSS-like cascaded multi-stream model with no mixture density networks.
NPSS: Blaauw and Bonada [BB17]
Different from the original NPSS, we don’t use spectral parameters for the inputs of aperiodicity and V/UV prediction models. This is because (1) D4C does not use spectral parameters as input for aperiodicity estimation. (2) V/UV detection is done from aperiodicity at 0-3 kHz in WORLD. In addition, f0 and VUV models dont use MDNs.
Empirically, we found the above configuration works better than the original one.
Conditional dependency: p(MGC, LF0, VUV, BAP |C) = p(LF0|C) p(MGC|LF0, C) p(BAP|LF0, C) p(VUV|LF0, BAP, C)
- Parameters:
in_dim (int) – Input dimension.
out_dim (int) – Output dimension.
stream_sizes (list) – List of stream sizes.
lf0_model (BaseModel) – Model for predicting log-F0.
mgc_model (BaseModel) – Model for predicting MGC.
bap_model (BaseModel) – Model for predicting BAP.
vuv_model (BaseModel) – Model for predicting V/UV.
in_rest_idx (int) – Index of the rest symbol in the input features.
in_lf0_idx (int) – index of lf0 in input features
in_lf0_min (float) – minimum value of lf0 in the training data of input features
in_lf0_max (float) – maximum value of lf0 in the training data of input features
out_lf0_idx (int) – index of lf0 in output features. Typically 180.
out_lf0_mean (float) – mean of lf0 in the training data of output features
out_lf0_scale (float) – scale of lf0 in the training data of output features
vuv_model_bap_conditioning (bool) – If True, use BAP features for V/UV prediction.
vuv_model_bap0_conditioning (bool) – If True, use only 0-th coef. of BAP for V/UV prediction.
vuv_model_lf0_conditioning (bool) – If True, use log-F0 features for V/UV prediction.
vuv_model_mgc_conditioning (bool) – If True, use MGC features for V/UV prediction.
NPSSMDNMultistreamParametricModel
- class nnsvs.acoustic_models.NPSSMDNMultistreamParametricModel(in_dim: int, out_dim: int, stream_sizes: list, reduction_factor: int, lf0_model: Module, mgc_model: Module, bap_model: Module, vuv_model: Module, in_rest_idx=0, in_lf0_idx=51, in_lf0_min=5.3936276, in_lf0_max=6.491111, out_lf0_idx=60, out_lf0_mean=5.953093881972361, out_lf0_scale=0.23435173188961034, vuv_model_bap_conditioning=True, vuv_model_bap0_conditioning=False, vuv_model_lf0_conditioning=True, vuv_model_mgc_conditioning=False)[source]
NPSS-like cascaded multi-stream parametric model with mixture density networks.
Note
This class was originally designed to be used with MDNs. However, the internal design was changed to make it work with non-MDN and diffusion models. For example, you can use non-MDN models for MGC prediction.
NPSS: Blaauw and Bonada [BB17]
acoustic features: [MGC, LF0, VUV, BAP]
Conditional dependency: p(MGC, LF0, VUV, BAP |C) = p(LF0|C) p(MGC|LF0, C) p(BAP|LF0, C) p(VUV|LF0, BAP, C)
- Parameters:
in_dim (int) – Input dimension.
out_dim (int) – Output dimension.
stream_sizes (list) – List of stream sizes.
lf0_model (BaseModel) – Model for predicting log-F0.
mgc_model (BaseModel) – Model for predicting MGC.
bap_model (BaseModel) – Model for predicting BAP.
vuv_model (BaseModel) – Model for predicting V/UV.
in_rest_idx (int) – Index of the rest symbol in the input features.
in_lf0_idx (int) – index of lf0 in input features
in_lf0_min (float) – minimum value of lf0 in the training data of input features
in_lf0_max (float) – maximum value of lf0 in the training data of input features
out_lf0_idx (int) – index of lf0 in output features. Typically 180.
out_lf0_mean (float) – mean of lf0 in the training data of output features
out_lf0_scale (float) – scale of lf0 in the training data of output features
vuv_model_bap_conditioning (bool) – If True, use BAP features for V/UV prediction.
vuv_model_bap0_conditioning (bool) – If True, use only 0-th coef. of BAP for V/UV prediction.
vuv_model_lf0_conditioning (bool) – If True, use log-F0 features for V/UV prediction.
vuv_model_mgc_conditioning (bool) – If True, use MGC features for V/UV prediction.
MultistreamSeparateF0MelModel
- class nnsvs.acoustic_models.MultistreamSeparateF0MelModel(in_dim: int, out_dim: int, stream_sizes: list, reduction_factor: int, encoder: Module, mel_model: Module, lf0_model: Module, vuv_model: Module, in_rest_idx=1, in_lf0_idx=300, in_lf0_min=5.3936276, in_lf0_max=6.491111, out_lf0_idx=180, out_lf0_mean=5.953093881972361, out_lf0_scale=0.23435173188961034)[source]
Multi-stream model with a separate F0 prediction model (mel-version)
Conditional dependency: p(MEL, LF0, VUV|C) = p(LF0|C) p(MEL|LF0, C) p(VUV|LF0, C)
- Parameters:
in_dim (int) – Input dimension.
out_dim (int) – Output dimension.
stream_sizes (list) – List of stream sizes.
reduction_factor (int) – Reduction factor.
encoder (nn.Module) – A shared encoder.
mel_model (nn.Module) – MEL prediction model.
lf0_model (nn.Module) – log-F0 prediction model.
vuv_model (nn.Module) – V/UV prediction model.
in_rest_idx (int) – Index of the rest symbol in the input features.
in_lf0_idx (int) – index of lf0 in input features
in_lf0_min (float) – minimum value of lf0 in the training data of input features
in_lf0_max (float) – maximum value of lf0 in the training data of input features
out_lf0_idx (int) – index of lf0 in output features. Typically 180.
out_lf0_mean (float) – mean of lf0 in the training data of output features
out_lf0_scale (float) – scale of lf0 in the training data of output features
MDNMultistreamSeparateF0MelModel
- class nnsvs.acoustic_models.MDNMultistreamSeparateF0MelModel(in_dim: int, out_dim: int, stream_sizes: list, reduction_factor: int, mel_model: Module, lf0_model: Module, vuv_model: Module, in_rest_idx=0, in_lf0_idx=51, in_lf0_min=5.3936276, in_lf0_max=6.491111, out_lf0_idx=60, out_lf0_mean=5.953093881972361, out_lf0_scale=0.23435173188961034, vuv_model_lf0_conditioning=True, vuv_model_mel_conditioning=True)[source]
Multi-stream model with a separate F0 model (mel-version) and mDN
V/UV prediction is performed given a mel-spectrogram.
Conditional dependency: p(MEL, LF0, VUV|C) = p(LF0|C) p(MEL|LF0, C) p(VUV|LF0, MEL, C)
Note
This class was originally designed to be used with MDNs. However, the internal design was changed to make it work with non-MDN and diffusion models. For example, you can use non-MDN models for mel prediction.
- Parameters:
in_dim (int) – Input dimension.
out_dim (int) – Output dimension.
stream_sizes (list) – List of stream sizes.
reduction_factor (int) – Reduction factor.
encoder (nn.Module) – A shared encoder.
mel_model (nn.Module) – MEL prediction model.
lf0_model (nn.Module) – log-F0 prediction model.
vuv_model (nn.Module) – V/UV prediction model.
in_rest_idx (int) – Index of the rest symbol in the input features.
in_lf0_idx (int) – index of lf0 in input features
in_lf0_min (float) – minimum value of lf0 in the training data of input features
in_lf0_max (float) – maximum value of lf0 in the training data of input features
out_lf0_idx (int) – index of lf0 in output features. Typically 180.
out_lf0_mean (float) – mean of lf0 in the training data of output features
out_lf0_scale (float) – scale of lf0 in the training data of output features
vuv_model_lf0_conditioning (bool) – If True, use log-F0 features for V/UV prediction.
vuv_model_mel_conditioning (bool) – If True, use mel features for V/UV prediction.
Other single-stream models
ResF0Conv1dResnet
- class nnsvs.acoustic_models.ResF0Conv1dResnet(in_dim, hidden_dim, out_dim, num_layers=4, in_lf0_idx=300, in_lf0_min=5.3936276, in_lf0_max=6.491111, out_lf0_idx=180, out_lf0_mean=5.953093881972361, out_lf0_scale=0.23435173188961034, init_type='none', use_mdn=False, num_gaussians=8, dim_wise=False)[source]
Conv1d + Resnet + Residual F0 prediction
Residual F0 prediction is inspired by Hono et al. [HHO+21].
- Parameters:
in_dim (int) – input dimension
hidden_dim (int) – hidden dimension
out_dim (int) – output dimension
num_layers (int) – number of layers
in_lf0_idx (int) – index of lf0 in input features
in_lf0_min (float) – minimum value of lf0 in the training data of input features
in_lf0_max (float) – maximum value of lf0 in the training data of input features
out_lf0_idx (int) – index of lf0 in output features. Typically 180.
out_lf0_mean (float) – mean of lf0 in the training data of output features
out_lf0_scale (float) – scale of lf0 in the training data of output features
init_type (str) – initialization type
use_mdn (bool) – whether to use MDN or not
num_gaussians (int) – number of gaussians in MDN
dim_wise (bool) – whether to use dimension-wise MDN or not
ResF0VariancePredictor
- class nnsvs.acoustic_models.ResF0VariancePredictor(in_dim, out_dim, num_layers=5, hidden_dim=256, kernel_size=5, dropout=0.5, init_type='none', use_mdn=False, num_gaussians=1, dim_wise=False, in_lf0_idx=300, in_lf0_min=5.3936276, in_lf0_max=6.491111, out_lf0_idx=180, out_lf0_mean=5.953093881972361, out_lf0_scale=0.23435173188961034)[source]
Variance predictor in Ren et al. [RHQ+21] with residual F0 prediction
- Parameters:
in_dim (int) – the input dimension
out_dim (int) – the output dimension
num_layers (int) – the number of layers
hidden_dim (int) – the hidden dimension
kernel_size (int) – the kernel size
dropout (float) – the dropout rate
in_lf0_idx (int) – the index of the input LF0
in_lf0_min (float) – the minimum value of the input LF0
in_lf0_max (float) – the maximum value of the input LF0
out_lf0_idx (int) – the index of the output LF0
out_lf0_mean (float) – the mean value of the output LF0
out_lf0_scale (float) – the scale value of the output LF0
init_type (str) – the initialization type
use_mdn (bool) – whether to use MDN or not
num_gaussians (int) – the number of gaussians
dim_wise (bool) – whether to use dim-wise or not