Lina-Speech demo page

Published: demo

Authors

Théodor Lemerle - IRCAM
Harrison Vanderbyl - RWKV Foundation/Eleuther AI/Recursal AI
Vaibhav Srivastav - HuggingFace
Nicolas Obin - IRCAM
Axel Roebel - IRCAM

Initial State Tuning

Initial State Tuning eases voice cloning from multiple speech excerpts , allowing us to capture finer details of the target speaker's identity. We compare this approach to prompt continuation. All speakers in this section are unseen. We crop the target text of VoiceCraft when necessary due to its limited context window.

Speaker Text Original Lina-Speech (Initial State Tuning) Lina-Speech (Prompt continuation) VoiceCraft (Prompt continuation)
3blue1brown The train sped through the countryside, fields of green and gold rushing past the window in a blur. Inside the carriage, passengers settled into their seats, lost in thought or conversation. A child pressed her nose against the glass, marveling at the changing landscape, while her parents whispered quietly, planning their next adventure together.
Sabine Hossenfelder Primary causes are unknown to us, but are subject to simple and constant laws, which may be discovered by observation, the study of them being the object of natural philosophy.
Geoffrey Hinton Newton was the greatest genius that ever existed, and the most fortunate, for we cannot find more than once a system of the world to establish.
Tibees The ordinary operations of algebra suffice to resolve problems in the theory of curves.
LJSpeech Compared to prompt continuation, it allows voice cloning from multiple speech excerpts and full usage of the context window for synthesis.
Andy Stapleton Sometimes when we are generous in small, barely detectable ways it can change someone else's life forever.
Krista King Once the game is over, the King and the pawn go back in the same box.
Frank Zappa Discovery is a child's privilege. I mean the small child, the child who is not afraid to be wrong, to look silly, to not be serious, and to act differently from everyone else. He is also not afraid that the things he is interested in are in bad taste or turn out to be different from his expectations, from what they should be, or rather he is not afraid of what they actually are.

Initial State Tuning Against Parler - Expresso Dataset (Evaluation samples)

Samples from the evaluation in the paper on the Expresso Dataset. Samples are unseen during training for Lina-Speech and the initial states are tuned on each pair (speaker, style). Parler-Mini has been fine-tuned on Expresso.

Speaker ID Style Text Original Lina-Speech (Initial State Tuning) Parler Mini(Textual Prompt)
ex01 whisper The old man sat on his porch, whittling a piece of wood as the sun set.
ex04 default As the storm intensified, the sound of thunder echoed through the valley, and the wind howled ferociously, rattling the windows and causing the tall trees to sway dangerously.
ex01 confused They argued all night long, voices rising and falling like the ebb and flow of waves.
ex01 happy In an old, creaky wooden house at the edge of town, a mysterious ticking noise could be heard every night, filling the silence and keeping the residents awake in a constant state of curiosity and unease.
ex03 enunciated The vibrant colors of the autumn leaves, from fiery reds to deep oranges, fluttered gently in the cool breeze, creating a mesmerizing display as they fell softly to the ground.
ex02 sad At the edge of the desert, a lone cactus stood tall, its spiny arms outstretched toward the sun, while in the distance, sand dunes rolled endlessly, their peaks glowing gold in the late afternoon light.

Initial State Tuning Against Parler - LibriTTS (Evaluation samples)

Samples from the evaluation in the paper on LibriTTS. Samples are seen during training for both models.

Speaker ID Text Original Lina-Speech (Initial State Tuning) Parler Mini(Textual Prompt)
Eileen The chef, with years of experience behind him, carefully prepared the gourmet meal, slicing vegetables with precision and adding a dash of spices, while the aroma of roasting meats filled the kitchen.
James Beneath the surface of the crystal clear lake, fish darted in and out of view, their silver scales glinting in the sunlight, while water lilies floated lazily on the surface, undisturbed by the gentle ripples.
Aaron On the open plains, herds of wild horses galloped freely, their manes flowing in the wind, and the pounding of their hooves creating a thunderous rhythm that reverberated through the earth.
Emily As she walked along the beach at sunset, the rhythmic sound of the waves crashing against the shore was accompanied by the distant cries of seagulls, creating a serene and peaceful atmosphere.
Bruce In the laboratory, scientists worked tirelessly to develop a new form of renewable energy, surrounded by computers, chemical vials, and complex machinery that hummed softly in the background.
Rebecca In the early hours of a misty morning, the golden rays of the sun slowly crept over the horizon, casting long shadows on the dew covered grass and awakening the sleepy countryside.

Zero-shot voice cloning by prompt continuation (Evaluation samples).

Samples from the evaluation in the paper on LibriTTS test split for prompt continuation. Samples are unseen during training.

Speaker ID Text Prompt Lina-Speech VoiceCraft StyleTTS 2 VALLE-X (Plachtaa)
3080 I am beginning it to day; and have chosen to go as far as Gravesend by water, though it be very gloomy weather.
2300 We had an index like a steam gauge, called an ampere meter, to indicate the amount of current going out.
5764 Would john Knox have been any worse had he deserted Christ and become a follower of Confucius?
908 Dearest, teach me so To pour out gratitude, as thou dost, good!
1688 I don't think, mr Hale, you have done quite right in introducing such a person to us without telling us what he had been.
6432 There were three of them, the center figure being that of Harry King, and he was very much intoxicated.
121 He seemed to say it was not so simple as that; to be really at a loss how to qualify it.
2300 The hotel is situated at an elevation of thirty five hundred feet above the sea, and was at that time forty miles from the railroad.
3997 Of course that make a terrible mess an' a powerful lot of smoke an' hot ashes fly up the chimney.
8230 Thus no knowledge as to the past is to be derived from the feeling of familiarity alone.
1688 I knew it all along from mr Bell; so I was aware of what was coming; but I half expected to see you get up and leave the room.'
7729 Ten days were consumed in these negotiations; but the spirit of vengeance refused to yield.
4077 At the inception of plural marriage among the Latter day Saints, there was no law, national or state, against its practise.
121 He seemed to say it was not so simple as that; to be really at a loss how to qualify it.
6938 After a brief conversation I left-and that's all! Let any one present make such an accusation!"
8455 Then there were three or four leading men of the community, with their wives, who were for the most part the fathers and mothers of the young ladies.
1995 At last the Cotton Combine was to all appearances an assured fact and he was slated for the Senate.
3538 The old King went out and fought bravely, but at last his sword broke, and he was wounded and his men fled.
2961 The subject was a very noble one; he described the most famous action in which the Athenian people were ever engaged.
4970 And the lectures, and the dissecting rooms, has thee thought of the dissecting rooms?"
5442 "I ought to own that I don't very well understand the drift of the provincial elections," said Levin.
1284 At the foot of the mountain that separated the Country of the Munchkins from the Country of the Gillikins, the path divided.
4992 A few moments later Nancy entered her mother's room, her arms filled with treasures from the woods and fields.