Home UncategorizedAdobe Max 2016 had an unexpected surprise, Adobe VoCo, think of it as the photoshop of spoken audio.
Adobe Max 2016 had an unexpected surprise, Adobe VoCo, think of it as the photoshop of spoken audio.

Adobe Max 2016 had an unexpected surprise, Adobe VoCo, think of it as the photoshop of spoken audio.

Adobe Max 2016 had an unexpected surprise, Adobe VoCo, think of it as the photoshop of spoken audio.

Many in the vfx community have speculated on what we were seeing but the truth is much more interesting than you might have expected.

At the Adobe Max conference video clip below if you do not watch it to the end it is not immediately obvious how significant this technology is. Being able to edit a recorded audio clip via text is cool but not revolutionary… being able to make an audio clip of someone saying a sentence they never even remotely said is completely game changing. Remarkably, we all thought that Adobe had added deep learning AI technology to the company’s audio team. But fxguide spoke to Zeyu Jin directly and while he and the team at Adobe thought the future lies in the deep neural networks he explained that “the VoCo you saw at the Max conference is not based on deep learning”.

 

 

Near the end of this presentation Zeyu Jin reveals in a brief off script chat that Adobe uses 20 mins of training data. From this 20 minute sample, the system can not just edit existing speech but create whole new sentences, with the correct weighting and cadence needed to be believable. It was assumed by many including here at fxguide, that this was 20 minutes of deep learning training data, but that is not how Adobe manages to drive VoCo, – which in one sense makes it even more remarkable.

Project VoCo allows you to edit and create speech in text, opening the door to vastly more realistic conversational assistive agents, better video game characters and whole new levels of interactive computer communication. Already Google and Apple’s embodied agents such as SIRI produce a passable synthetic conversation dialogue, but if Adobe can commercialise this tech preview to a full product, the sky is the limit.

As VoCo allows you to change words in a voiceover simply by typing new words it was a huge hit of the Adobe MAX 2016 Sneak Peeks. While this technology are not yet part of Creative Cloud, many such technology preview “sneaks’ from previous years have later been incorporated into Adobe products.

Which begs the question how are they doing this, if they are not using deep learning?

“It is not related to decision trees or traditional machine learning but mostly to mathematical optimization and phonetic analysis” Jin explained.

A couple of years ago Adobe showed a text tool that would allow you to search a clip by text, initially many assumed that this was more than it was, in reality the software took your written script and matched it to the audio for searching. At the time it was both exciting to see Adobe doing this work and a tad disappointing they were not showing auto-transcribing software as early reports had indicated. Jump to 2016 and Adobe is demoing that it can do to audio what it has done to editing still images. But this new VoCo technology is not related at all to the earlier Adobe text search-edit demo, it is completely brand new tech. ” It is not built on any existing Adobe technology. It is originated from Princeton University where I am doing my Ph.D.” he explains. “Unfortunately, it also does not imply improvement on auto transcription. We are actually relying on existing transcription approach to perform phoneme segmentation”.

“The core of this method is voice conversion. Turning one voice into another”. The system uses micro clips but with no pitch correction, “in fact, pitch correction is what makes other approaches inferior to our approach” he comments.

Most state-of-the art voice conversion methods re-synthesize voice from spectral representations and this introduces muffled artefacts. Jin’s PhD research uses a system he calls CUTE: A Concatenative Method For Voice Conversion Using Exemplar-based Unit Selection. Or in loose terms: it stitches together pieces of the target voice using examples. It just does it very cleverly. It optimizes for three goals: matching the string you want, using long consecutive segments, and finally smooth transitions between those segments.

CUTE stands for:

  • Concatenative synthesis
  • Unit selection
  • Triphone pre-selection
  • Exemplar-based features

To have sufficient ‘units’ of audio or sound to flexibility use, the approach defines these ‘units’ to the frame level. To make sure there is a smooth transition, it computes features using the 20 minutes of example audio and concatenates the spectral representations over multiple consecutive frames.

To obtain the phoneme segmentation, the system first translate the transcripts into phoneme sequences and then apply forced alignment to align phonemes to the target speaker’s voice.

The original PhD University approach defined two types of examples to use, a target exemplar and concatenation exemplar, to allow controlling the patterns of rhythm and sound (or the patterns of stress and intonation in the spoken words) with source examples and enforce concatenation smoothness in the target samples. Using phoneme information to pre-select the phonemes, we ensure the longest possible phonetically correct segments to be used in concatenation synthesis.

Experiments demonstrate that our CUTE method has better quality than previous voice conversion methods and high individuality comparable to real samples.

You can see Jin’s original paper here.

Read more : https://www.fxguide.com/

 

73 thoughts on “Adobe Max 2016 had an unexpected surprise, Adobe VoCo, think of it as the photoshop of spoken audio.

  1. Oh my goodness! Awesome article dude! Thank you, However I am encountering issues with your RSS.
    I don’t understand the reason why I am unable
    too join it. Is there anybody having the same RSS problems?
    Anyone that knows thee answer can you kindly respond?
    Thanx!!

  2. I see interesting content here. Your website can go viral easily, you need some initial traffic only.
    There is a sneaky method to get massive traffic from social media.
    Search in google for; Twinor’s strategy

  3. Hey great blog! Doess running a blog like this take a
    large amount of work? I’ve very little expertise in coding however I was hopng to start my own blpog soon. Anyway, if you have any idreas or tips
    for nnew blog owners please share. I understand this is off subject but I simply wanted to
    ask. Thank you!

  4. Hello tto all, the contents existing aat this web page are genuinely awesome for people experience,
    well, keep upp the nice work fellows.

  5. Hi my family member! I want to sayy that this post is
    awesome, great wriitten and coje with almost all significant infos.

    I would like to look extra posts like this.

  6. Just wish to say your article iss as surprising.
    The cllarity in your post is simply cool aand i can assume you are aan expert on this subject.

    Fine with your permission allow me to grab your RSS feed to keep updated
    with forthcoming post. Thanks a million and please continue thee gratifying work.

  7. What’s Taking place i aam new to this, I sttumbled upon this I’ve discovered It positively helpful and
    it has helped me out loads. I hope too give a contribution & assist different custoomers lik
    its aidrd me. Good job.

  8. Hello there! I just want to give you a big thumbs up for
    your excellent information you have got here on this post.
    I am coming back to your blog for more soon.

  9. Do you have a spam probloem onn this site; I also am a blogger,
    and I was wanting to know your situation; we have created some nice procedures and we are
    lookinng to trade strategies with other folks, please shoot me an e-mail if interested.

  10. Thanks for your marvelous posting! I seriously enjoyed reading it, you are a
    great author. I will remember to bookmark your blog and will come back
    at slme point. I want to encourage you to ultimately continue your great work, have a nioce day!

  11. Excellent pieces. Keep writing such kind of info on your blog.
    Im really impressed by it.[X-N-E-W-L-I-N-S-P-I-N-X]Hello there, You’ve performed a great job.
    I will definitely digg it and individually suggest to my friends.
    I am sure they’ll be benefited from this web site.

  12. Hello there! I know this is somewhat off topic but I was wondering which blog platform arre you
    using for this website? I’m getting tired of WordPress because I’ve had probleems
    with hackers and I’m looking at options for another platform.

    I would bee great if yoou could pont me iin the direction of a good platform.

  13. Write more, thats all I have to say. Literally, it seems as though you relied on the video to make your point.
    You clearly know what youre talking about, why throw away your intelligence on just posting videos
    to your blog when you could be giving us something informative to
    read?

  14. Great beat ! I would like to apprentice whilst
    you amend your web site, how could i subscribe for a weblog site?
    The account aided me a acceptable deal. I were a
    little bit familiar of this your broadcast offered vivid
    clear idea

  15. Hey there are using WordPress for your site platform? I’m new to the blog world
    but I’m trying to get started and create my own. Do you require any html coding knowledge to make your own blog?
    Any help would be greatly appreciated!

  16. Thanks for any other informative site. Where else could I am getting that
    type of information written in such a perfect way?

    I’ve a mission that I’m simply now running on, andd I hage been att the look out forr
    such info.

  17. Good day! I could have sworn I’ve been to this website before but after browsing through some of the post I realized
    it’s new to me. Anyways, I’m definitely glad I found it and I’ll be book-marking and
    checking back often!

  18. I’m not that much of a online reader to be honest but your
    blogs really nice, keep it up! I’ll go ahead and bookmark your website to
    come back later on. All the best

  19. I was recommended this blog by my cousin. I’m not sure whether this post
    is written by him as no one else know such detailed about my trouble.
    You’re amazing! Thanks!

  20. Cool blog! Is your theme custom made or
    did you download it from somewhere? A theme like yours with a few simple adjustements
    would really make my blog jump out. Please let me know where you got your theme.

    Bless you

  21. I’m now not sure where you are getting your information, but good topic.
    I needs to spend a while finding out much more or understanding more.
    Thank you for magnificent info I used to be on the
    lookout for this information for my mission.

  22. Hey there! Do you know if they make any plugins to assist with SEO?
    I’m trying to get my blog to rank for some targeted keywords but I’m not seeing very good results.
    If you know of any please share. Appreciate it!

  23. Wow that was unusual. I just wrote an extremely long comment but
    after I clicked submit my comment didn’t appear. Grrrr… well I’m not writing all that over again. Regardless, just wanted to say great blog!

  24. Hi, I do believe your site might be having browser compatibility issues.
    When I look at your site in Safari, it looks fine however when opening in I.E.,
    it’s got some overlapping issues. I simply wanted
    to provide you with a quick heads up! Aside from that, great
    website!

  25. Have you ever thought about including a little
    bit more than just your articles? I mean, what you say is
    important and everything. However think about if you added some great photos or video clips to give your posts more, “pop”!
    Your content is excellent but with pics and videos, this website could certainly be one of
    the greatest in its field. Wonderful blog!

  26. This is the right website for anybody who would like to find out about this topic.
    You now so much its almost tough to argue with you (not that I really would
    want to…HaHa). You certainly put a brand new spin on a
    subject which has been written about for decades. Wonderful stuff, just excellent!

  27. You could definitely see your skills within the work you
    write. The world hopes for even more passionate writers
    like you who are not afraid to say how they believe.
    All the time follow your heart.

  28. Hey! Would you mind if I share your blog with my twitter group?
    There’s a lot of people that I think would really enjoy your content.
    Please let me know. Many thanks

  29. Hi there! I know this is kinda off topic but I’d figured I’d ask.
    Would you be interested in trading links or maybe guest writing
    a blog article or vice-versa? My website discusses a lot of the same topics
    as yours and I think we could greatly benefit from each other.
    If you are interested feel free to shoot me an e-mail.

    I look forward to hearing from you! Terrific blog by the way!

  30. I simply want to say I’m new to weblog and honestly loved you’re web page. Probably I’m likely to bookmark your blog . You absolutely have really good posts. Thanks a bunch for sharing your web page.

  31. I blog quite often and I genuinely thank you for your information. This great article has really peaked my interest.
    I’m going to bookmark your website and keep checking for new information about once a week.
    I opted in for your RSS feed as well.

  32. You actually make it seem so easy with your presentation but I find this topic to be actually
    something which I think I would never understand.
    It seems too complicated and extremely broad for me. I am looking forward for your next post,
    I’ll try to get the hang of it!

Leave a Reply