Weeks after the launch of the AI language ElevenLabs has launched its offering for Sound Effects’ text-to-sound AI, the corporate is releasing Open source tool to display its potential. In “about 15 seconds,” this application allows creators to generate sound effect samples for his or her videos, analyze the imported clip, and supply several options.
While developers have access to the App code on GitHubElevenLabs has released a web site where the general public can check out its Sound Effects API.
When you upload a video, the so-called Video-to-Sound-Effects app on the client side extracts 4 frames, one second apart. It then sends those frames and a prompt to OpenAI's GPT-4o to create a custom text-to-sound effects prompt. That prompt is then used to generate a sound effect via ElevenLabs' Sound Effects API. Finally, the video and audio are combined on the client side right into a single downloadable file that might be as much as 22 seconds long.
“We view it as a proof of concept of what people can do with our SFX API,” Ammaar Reshi, design lead at ElevenLabs, tells VentureBeat. “AI video producers are sometimes in search of the right sound effect and we felt we could intelligently speed up the workflow by understanding the frames of their videos after which suggesting the perfect output.” He says the corporate is worked up about the different sorts of dynamic experiences people could create with this API, highlighting immersive video games as one example, where sounds might be generated based on a player's interaction.
The API mentioned above allows developers to create completely custom AI sound effects using a brief description. ElevenLabs charges 100 characters per generation with automatic duration or 25 characters per second with fixed duration.
In a fast test, the app's ability to convert video to sound effects seemed easy. After importing a soundless movie of a vehicle navigating a terrain, ElevenLabs' AI generated 4 options that each one seemed like a automotive driving along a gravel road. While it's fun to use sound effects to clips, perhaps the actual potential lies in integrating this feature right into a larger system to reap the actual advantages.
And as the event of video using artificial intelligence becomes increasingly exciting, ElevenLabs will probably want to get ahead of everyone else and develop recent audio solutions that the corporate knows will probably be in high demand amongst developers, filmmakers and creatives.
