Embodiment - Part 1

A D-ID-like on your local machine

I was delighted to discover an Internet creating an embodiment of an AI - a “talking head” video where the text, speaker’s voice and a video of the speaker actually talking were the result of AI synthesis. Do check it out - click in to navigate to the full tweet for the video:

In English, this was the author’s workflow:

  1. Generate AI image with WebUI
  2. Generate text with ChatGPT (An explanation about sythetic voice AI)
  3. Create a voice AI with COEIROINK and synthesize voice from text
  4. Use D-ID for lipsync and speaking animation

I have a capable graphics card, so I wondered if I could achieve something similar, running locally, for free. I succeeded:

The duration of our passions is no more dependant upon us than the duration of our life.

My workflow:

  1. Generate AI image with local AUTOMATIC1111 WebUI
  2. I don’t have an LLVM running locally (yet), I just made do with “Maxims” - a collection of 500 or so aphorisms left to us by Francois Duc De La Rochefoucauld. They have a winning combination of being brief and being enjoyable to read and reflect on.
  3. Create a voice AI with TorToiSe. In order to lean into spoken word synthesis, I did not build my training data from any living person’s recordings, but took advantage of SynthV Karin
  4. Used SadTalker for lipsync and speaking animation

My hope here is to share the details on setting up this stack locally and my learnings in my future posts, so please stay tuned for Part 2, which will be on local setup and iterative development.