E1 TTS: Simple and Fast Non-Autoregressive TTS

Zhijun Liu 1 Shuai Wang 2,1 Pengcheng Zhu 3 Mengxiao Bi 3 Haizhou Li 1,2

School of Data Science 1 Shenzhen Research Institute of Big Data 2

The Chinese University of Hong Kong, Shenzhen

Fuxi AI Lab, NetEase Inc. 3


Overview of E1 TTS model.

Abstract

This paper introduces Easy One-Step Text-to-Speech (E1 TTS), an efficient non-autoregressive zero-shot text-to-speech system based on denoising diffusion pretraining and distribution matching distillation. The training of E1 TTS is straightforward; it does not require explicit monotonic alignment between the text and audio pairs. The inference of E1 TTS is efficient, requiring only one neural network evaluation for each utterance. Despite its sampling efficiency, E1 TTS achieves naturalness and speaker similarity comparable to various strong baseline models.

This github repository contains the audio samples, and the evaluation code and file lists.

Prompted Generation

TTS models generate target speech, given 3 seconds prompt speech as reference.

# 237 1089 1188 1320 1995 2300 2830 3570 4077 4446 4970 5105 5683 6829 7021 7176 7729 8230 8463 8555
Text

"Boys," whispered their mother, warningly, "she can't answer you; just look at her face."

After early nightfall the yellow lamps would light up, here and there, the squalid quarter of the brothels.

But in this vignette, copied from Turner, you have the two principles brought out perfectly.

A circle of a few hundred feet in circumference was drawn, and each of the party took a segment for his portion.

The Sun, the Swamp? Then finding all well, she closed her eyes and slept.

Hence the Edison electrolytic meter is no longer used, despite its excellent qualities.

The Reformer had lectured on this Epistle of saint Paul's in fifteen nineteen and again in fifteen twenty three.

The last items of this category of consumption are not given up except under stress of the direst necessity.

This statement assumes, as granted, a distinction between bigamy and the "Mormon" institution of plural marriage.

"Yes, I was happy, wasn't I?" She pressed his hand gently in gratitude. "Weren't you happy then, at all?"

He knew his uncle would be glad to hear that he had at last turned his thoughts to a practical matter.

"Is it not impossible," he murmured aloud, "that any city should disappear so completely?

'It is very happy, for her at least, they are not,' said Rachel, and a long silence ensued.

Give me a check for a hundred and fifty, and I'll turn over to you the forged check and quash further proceedings."

The first person he met was a farm labourer walking alongside a load of peat and smacking at his horse.

For all its appalling speed, the sound of his flight was nothing more than a strong pulsating hiss.

The firm defensive attitude of the people of Lawrence had produced its effect.

Some points may be taken as fixed, and such as any theory of memory must arrive at.

Even so, I had just returned from an arduous journey, exhausted and badly needing a rest.

This was done, the once royal family departing from the palace with shamed and downcast looks.

Prompt
E1 TTS (DMD)
CosyVoice
ARDiT (B=1, DMD)
StyleTTS 2
Ground Truth
Token Reconstruct (DMD)

Speech Inpainting

TTS models try to recover masked speech, given unmasked speech and the complete text as reference. Here we mask the middle one third (about 2 seconds long) for benchmarking.

# 237 1089 1188 1320 1995 2300 2830 3570 4077 4446 4970 5105 5683 6829 7021 7176 7729 8230 8463 8555
Text

"Boys," whispered their mother, warningly, "she can't answer you; just look at her face."

After early nightfall the yellow lamps would light up, here and there, the squalid quarter of the brothels.

But in this vignette, copied from Turner, you have the two principles brought out perfectly.

A circle of a few hundred feet in circumference was drawn, and each of the party took a segment for his portion.

The Sun, the Swamp? Then finding all well, she closed her eyes and slept.

Hence the Edison electrolytic meter is no longer used, despite its excellent qualities.

The Reformer had lectured on this Epistle of saint Paul's in fifteen nineteen and again in fifteen twenty three.

The last items of this category of consumption are not given up except under stress of the direst necessity.

This statement assumes, as granted, a distinction between bigamy and the "Mormon" institution of plural marriage.

"Yes, I was happy, wasn't I?" She pressed his hand gently in gratitude. "Weren't you happy then, at all?"

He knew his uncle would be glad to hear that he had at last turned his thoughts to a practical matter.

"Is it not impossible," he murmured aloud, "that any city should disappear so completely?

'It is very happy, for her at least, they are not,' said Rachel, and a long silence ensued.

Give me a check for a hundred and fifty, and I'll turn over to you the forged check and quash further proceedings."

The first person he met was a farm labourer walking alongside a load of peat and smacking at his horse.

For all its appalling speed, the sound of his flight was nothing more than a strong pulsating hiss.

The firm defensive attitude of the people of Lawrence had produced its effect.

Some points may be taken as fixed, and such as any theory of memory must arrive at.

Even so, I had just returned from an arduous journey, exhausted and badly needing a rest.

This was done, the once royal family departing from the palace with shamed and downcast looks.

E1 TTS (DMD)
ARDiT (B=1, DMD)
Mel Reconstruct

Prompted Generation on Hard Samples

Prompted generation again, but with hard text inputs proposed in ELLA-V.

# 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76
Text

g 2 p 2 g 2 p 2 g 2 p 2 p 2 p

F 1 F 2 F 4 F 8 H 16 H 32 H 64

Amidst the towering skyscrapers, the bustling cityscape echoed with the incessant hum hum hum hum hum of commerce and ambition.

As the ancient ancient ancient ancient ancient ancient manuscript was deciphered, the archaeologist unearthed a trove of forgotten forgotten forgotten forgotten forgotten forgotten wisdom embedded in in in in in in the crumbling parchment.

As the cosmic cosmic cosmic cosmic cosmic cosmic dance of the stars unfolds in in in in in in silence, revealing the mystical mysteries of the celestial celestial celestial celestial celestial celestial realm.

As the quantum physicist delved into the quantum realm, the enigmatic entanglement of particles perplexed even the most astute astute astute astute astute astute minds.

Beneath the celestial canopy, the twinkling stars whispered tales of timeless timelessness and cosmic conspiracies.

Beneath the moonlit night, the solitary wolf’s haunting howl howl howl howl howl echoed through the ancient forest, embodying the primal spirit of the wilderness.

Beneath the shimmering sheen of the silken fabric fascinated the fashionistas, creating a cacophony of compliments.

Coding in in in in in in solitude, the programmer uncovered the intricate intricacies of the algorithm, unveiling a hidden hidden hidden layer of complexity.

Crafting a symphony of flavors, the skilled chef orchestrated a culinary masterpiece that left an indelible mark mark mark mark mark on the palates of the discerning diners.

Delving into the dense foliage, the intrepid explorer uncovered the cryptic trail trail trail trail trail leading to an undiscovered waterfall deep in the heart of the jungle.

In the bustling bustling bustling bustling bustling bustling metropolis, the rhythmic rhythm of life beats to the pulsating pulsating pulsating pulsating pulsating heartbeat of the city streets.

In the enigmatic enigmatic enigmatic enigmatic enigmatic enigmatic maze, the elusive solution remained hidden behind layers of intricate intricacies.

In the realm of robotics, replicating the nuanced nuances of human emotions remains a formidable formidable formidable formidable formidable challenge.

In the timeless tapestry of the galaxy, the mesmerizing dance dance dance dance dance of celestial bodies unveiled the cosmic secrets of the universe.

In the vast vast vast vast vast vast landscape of knowledge, the voracious voracious voracious voracious voracious voracious reader explored the depths of literature, extracting pearls of wisdom hidden within within within within within within the pages.

Navigating through through through through through through the dense forest, the intrepid explorer discovered an ancient ancient ancient ancient ancient ancient relic buried beneath layers of thick underbrush.

Navigating through the labyrinthine labyrinthine labyrinthine labyrinthine labyrinthine labyrinthine code, the programmer discovered an elusive bug.

Precision in decision-making is paramount, as an error could cost countless countless countless countless countless countless countless lives.

Prompt
E1 TTS (DMD)
CosyVoice
StyleTTS 2

Extra Samples: Voice Cloning Celebrities

E1 TTS trained only on LibriTTS is capable of imitating famous figures' voice. Prompts and baseline results are obtained from demo pages of CLaM-TTS and DiTTo-TTS. For E1 TTS results in this section, we picked the best result in 3 generations.

# Michael Caine Jessie Eisenberg Optimus Prime Rachel McAdams Robert Downey Jr. Benedict Cumberbatch Mark Zuckerberg
Text

And sometimes, in both realms, it's not just about shining the brightest, but enduring the longest.

So here we are, trying to catch up, and hoping this day turns around soon.

We must unite and harness our strengths, for the fate of our world hangs in the balance.

But to those who knew her well, it was a symbol of her unwavering determination and spirit.

We have the responsibility to ensure power and technology are used for the greater good.

However, if you choose to stay, know that the truth I unveil may forever alter the course of your journey.

Our goal is to bridge communication gaps and preserve the richness of these unique languages.

Prompt
E1 TTS
DiTTo-TTS
ARDiT (DMD, B=1)
CLaM-TTS

Extra Samples: Compare with Proprietary Systems on Prompted Generation

In this section, we compare our system with proprietary systems (as of 16 Sep 2024) on prompted generation. The following samples are obtained from their online demo pages. All waveforms are downsampled to 16kHz. The audios in this section may sound bad due to the 16kHz sampling rate and noisy speech prompts.

# 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Text

It is this that is of interest to theory of knowledge.

For, like as not, they must have thought him a prince when they saw his fine cap.

What you had best do, my child, is to keep it and pray to it that since it was a witness to your undoing, it will deign to vindicate your cause by its righteous judgment.

The strong position held by the Edison system under the strenuous competition that was already springing up was enormously improved by the introduction of the three wire system and it gave an immediate impetus to incandescent lighting.

They moved thereafter cautiously about the hut groping before and about them to find something to show that Warrenton had fulfilled his mission.

And lay me down in thy cold bed and leave my shining lot.

Number ten, fresh nelly is waiting on you, good night husband.

Yea, his honourable worship is within, but he hath a godly minister or two with him, and likewise a leech.

Instead of shoes, the old man wore boots with turnover tops, and his blue coat had wide cuffs of gold braid.

The army found the people in poverty and left them in comparative wealth.

Thus did this humane and right minded father comfort his unhappy daughter, and her mother embracing her again, did all she could to soothe her feelings.

He was in deep converse with the clerk and entered the hall holding him by the arm.

Indeed, there were only one or two strangers who could be admitted among the sisters without producing the same result.

For if he's anywhere on the farm, we can send for him in a minute.

Their piety would be like their names, like their faces, like their clothes, and it was idle for him to tell himself that their humble and contrite hearts it might be paid a far-richer tribute of devotion than his had ever been. A gift tenfold more acceptable than his elaborate adoration.

The air and the earth are curiously mated and intermingled as if the one were the breath of the other.

I had always known him to be restless in his manner, but on this particular occasion he was in such a state of uncontrollable agitation that it was clear something very unusual had occurred.

His death in this conjuncture was a public misfortune.

It is this that is of interest to theory of knowledge.

For a few miles, she followed the line hitherto presumably occupied by the coast of Algeria, but no land appeared to the south.

Prompt
E1 TTS (DMD)
FlashSpeech
ARDiT (DMD, B=1)
NaturalSpeech 3
NaturalSpeech 2
MegaTTS 2
UniAudio
CLaM-TTS
VoiceBox
VALL-E
Ground Truth