Get – Dialogue Search for Final Cut Pro

AV3 Software’s Get application was one of the highlights at last year’s NAB. Based on the same Nexidia search engine as Avid’s ScriptSync, Get brings dialogue-based search capabilities to Apple Final Cut Pro and eventually other editing applications. (In fact, Avid has recently introduced PhraseFind, which is Nexidia’s implementation of the same technology within the Media Composer 5.5 application.)

Get functions as a standalone application, so its usefulness extends beyond the needs of an editor. For instance, story editors, loggers and news reporters can benefit from Get as a way to research and organize interviews and other dialogue-based media files.

How Get differs

To properly understand Get, you have to first understand how the Avid and Adobe technologies work. Both Get and Avid ScriptSync are based on Nexidia’s patented phonetic search engine. In a simplified explanation, the Nexidia engine analyzes the waveform of phonetic syllables and compares these to its library of known phonetics for specific languages. Because it is based on phonetics, an exact spelling match isn’t critical as long as the misspelled word sounds the same. If you spell “cool” as “kool”, the result will be the same.

Avid ScriptSync requires the editor to first import a written script or transcript. Ingested dailies are then analyzed for sound and matched against sections of the script. The software matches key words and interpolates the in-between sections, when it can’t make an exact word match. Through this process, it is possible to click on a word in the written script (within the Avid bin) and locate the corresponding portion of the media file. ScriptSync does not work on media files outside of the Avid Media Composer project.

Adobe’s speech-to-text translation uses exactly the opposite process. It is based on speech recognition technology and attempts to properly interpret spoken words and correctly translate those into written text. Since it doesn’t start with an existing script and there’s no interpolation of words from a known reference, it falls to the analysis engine to correctly translate spoken words into text from the audio track. This frequently leads to a high degree of error, because of the “guesses” the software is making. These must later be manually reconciled or adjusted. In addition, Adobe stores the translated text as part of the media file’s metadata, as it travels through Premiere Pro and/or Soundbooth.

It’s like Spotlight for words

Get operates as a standalone application and is therefore independent of any written script or transcript. Think of it as a Spotlight-style search tool that is optimized for speech. Get can be installed for various languages and currently supports US English, UK English and Latin American Spanish. For now, Final Cut Pro is the only supported NLE, although Get can be used without FCP, as well.

Once the application has been installed, the first step is to index the media files that are to be analyzed in any Get dialogue search. This can be one or more folders or drives containing QuickTime-wrapped media files. For example, if you are using it to work on a single documentary project, then you might only choose to index that project’s media folder within the FCP Capture Scratch location.

On the other hand, if you’d like indexing across multiple projects, you might opt to index a complete media drive or array. Search locations can be established as Watched Folders, so newly added media is automatically indexed. Content already added to an FCP project can also be indexed. In my testing, several hours of media files only required a few minutes to index, so that step is quite fast. During this process, Get analyzes the dialogue within the audio tracks of these media files and stores a look-up table used for subsequent searches.

Performing searches

After the initial indexing is completed, the rest of it is pretty simple. Typically you perform searches within Get’s user interface. Results will be grouped using various criteria, but normally Get will group search results according to a confidence indication. Search results that are listed as Very Confident will generally always match the search word or phrase, whereas Less Confident might not. One way to dial in search results is to use the Score Threshold pulldown menu. For example, if you search for a term and select a Score > 95 value, Get will return far fewer successful results than if you use a value of Score > 55. Lowering the score will achieve more total results, but also less accuracy.

Targeting the search correctly will improve results. For example, “1900” is the same as “nineteen hundred”, not “one thousand nine hundred”. Acronyms, like “RCA”, need to be spelled with capital letters. To receive more refined results, you can combine searches by adding fields for Boolean-style searches. For instance, a search for media files that contain the terms “family” AND “fortified”. Furthermore, additional attributes can be selected, which may be Finder-level metadata, like creation dates – or from an FCP project, like master comments. This becomes very useful in news and reality TV productions.

Get includes a media player, so any matching files displayed in the results pane can be played directly from within Get. The player displays the file attributes, plus timecode and timeline markers that indicate the location of each dialogue match. If there are multiple matches within a single media file, then there will be a marker for each spot within the file where the word or phrase was spoken. This lets an editor or producer easily preview and save search results without engaging any other software. Since a search will typically return numerous media files, it becomes a quick way to compare several similar shots.

Export to FCP

Get’s Export routine is used to bring clips into a Final Cut Pro project. Individual clips or a group of clips can be exported from the Get interface into any existing FCP project. Get has the ability to access the project, so either clips can be sent directly into existing bins or new FCP project bins containing these search results can be created and exported right from within Get. When you return to the FCP project, the appropriate folder and clips will have been added. Each clip will also contain markers to identify the location of the search term. In addition, Get lets you export only markers, in the case where you are analyzing clips already in FCP.

I performed testing with an existing series of corporate videos produced about a family-owned Australian winery. The original audio was recorded on a Canon EOS 5D Mark II. Some audio was rather low and, of course, everyone spoke with an Australian accent. Get delivered good results using the US English installation, in spite of these challenges. Even specific Australian words, like less-common town names, were easily found. Without question, I’d have to say Get definitely works as advertised.

One unconventional way to use Get’s capabilities is to sync multi-camera shots, in the absence of slates or matching timecode. I hadn’t even considered this until the folks at AV3 Software mentioned that some of their clients were using Get in this manner. A dialogue search would place markers at matching words, based on each camera’s audio track. Once inside Final Cut, these markers would then provide the basis for synchronizing the media files from the different cameras against each other, using matching dialogue as the common reference point.

Whether or not AV3 Software’s Get is the right tool for you depends on your project. It doesn’t replace the need for creating transcripts and it won’t edit a production for you. On the other hand, if you need to quickly locate the places a speaker uttered certain words or phrases within hours of interview footage or news coverage, then there simply is no faster tool available to the Final Cut Pro editor.

Last but not least, Get is available on a rental basis, too. So if you need these features for just a single documentary project, you’re covered!

Written for Videography magazine (NewBay Media LLC).

©2011 Oliver Peters