The popularity of P2 and other tapeless camera formats has had a big impact on the post community. Some editors love it while others view it as a huge pain. Nevertheless – like it or not – file-based production and post production are here to stay. There’s not only P2, but also XDCAM, XDCAM-HD, XDCAM-EX, RED and a whole slew of consumer and prosumer camcorders using SD and CF cards to record various flavors of SD and HD video. And let’s not forget that FireStore and the original Avid/Ikegami EditCam started it all and are still with us today. Sony optical disc XDCAM and XDCAM-HD tend to be the exception, since this media offers a hybrid workflow that bridges the tape and tapeless worlds. To avoid confusion, I’m going to frame my comments around card and drive-based media, like P2. Some of the tips will apply to XDCAM, but others won’t.
There are typically 3 elements to file-based recordings. The first is essence – the actual audio and video content. Audio/video media that is recorded at a particular size, scanning method and frame rate (e.g. 1920x1080p/23.98fps) and uses a specific codec (e.g. DVCPRO HD). This essence is encased in a file wrapper, like MXF, MOV, MP4 and others. The file method used might also include a small metadata file, which is a data file containing information about the essence. When people talk about P2, that terminology should really only be reserved for the actual card and Panasonic product family. P2 devices can record audio and video essence in various formats and with different codecs, yet it’s all still on the same P2 media card.
Even when things look the same, they aren’t. For example, both Sony (XDCAM-HD) and Panasonic (P2) use the MXF wrapper, but the essence inside is not the same. Panasonic P2 MXF files could be natively opened and edited in Avid software, but XDCAM-HD MXF cannot. It doesn’t even stay the same within the same company. Sony’s XDCAM-HD uses the MPEG2 codec for video files, which is wrapped as an MXF file. When the EX-series camera was released, Sony chose to wrap its MPEG2 recordings as MP4 files. You would think the files used an MPEG4 codec by that designation, but not with the EX cameras. In the case of Panasonic, you can now record HD video as either DVCPRO HD or as AVC-Intra and they both appear with MXF file extensions.
When you analyze the file structure of any of these media cards, there is a specific folder and file hierarchy. Depending on the format, this structure has to stay intact. Moving video files outside of their folder often results in the inability of an NLE to read or open these files, so be careful how you handle them. With that in mind, here are some workflow tips for dealing with file-based media in a tapeless world.
Tip 1 – Clone your camera cards or drives
With the exception of XDCAM and XDCAM-HD, all card and hard drive-based media recordings MUST be backed up for protection, because no one plans to leave the card on the shelf. The recommended practice is to “clone” the card, i.e. copy the card in an exact fashion to preserve the original format and codec and maintain its folder and file hierarchy. This step is often done on location using a laptop, so that cards can quickly be reformatted and used for further recordings during the same day. Card capacity has increased from 4GB to 64GB, but it’s important to realize that a large capacity card is not always the best choice. Yes, you can record all day, but that means you’re likely to spend the rest of the entire evening copying and verifying the cards. Even if you have a “data wrangler” on the crew, they will be sitting on their hands if the card is in the camera all day long.
Keep your back-ups native! Some folks have imported their media into FCP or Avid systems and then formatted the cards, thinking that their NLE-compatible media was protected. This may be the case if you also back up your working media drives or your drives are RAID-protected, but the logic is faulty. Once you have imported P2 DVCPRO HD or AVC-Intra files into most NLEs, those files have been altered. Depending on the format and NLE, they have either been rewrapped or transcoded. Destroying the original camera media is tantamount to shooting on film, transferring the film to video and then destroying the negative. If you have maintained a back-up of the camera media in its native form, then you can always go back to these files, should you decide to switch to a different NLE or your working media becomes corrupt.
OK, so we agree that you should back-up your files to match the cards. But how? There are lots of recipes for doing this, but I think the best all-around solution comes from Imagine Products. Their ShotPut software comes in Mac and Windows editions for P2, EX and RED. It’s designed to safely name folders and copy and verify files to as many as three destinations. Having multiple copies is important, because no media product is infallible. People theorize about burning their media to Blu-ray data discs as an archive, but the reality is that transfer rates, burning speeds and BD-R media costs make this unattractive. Other solutions, like LTO3 data tapes and RAID-5 arrays only appeal to a select few. The solution most producers settle on is to buy cheap commodity FireWire, USB or eSATA drives (Maxtor, LaCie, Western Digital, Hitachi, Seagate, etc.) and make at least two copies that will sit on the shelf. The hope is that at least one of these will still spin up and work a year or so down the road when you need to go back to this footage. Remember that this is in addition to the working media used during post production.
Tip 2 – Budget time and media costs
Capture time has been replaced by import time. When I work with videotape, I tend to select a handful of good options for each set-up or scene and digitize only those takes. As a result, I might capture about half of the tape, but this is offset by the review and logging time. Logging plus capture time takes about as long as the full running time of the tape.
With tapeless media, I bring it all in. Yes, I know, the various import modules, like FCP’s Log and Transfer let me cull the footage down, but I just don’t like working with them. I’d rather bring it all in and sort it out in the NLE, which brings us to the point about time and money. Starting a P2 or EX session for example, generally means mounting a cheap USB or FireWire drive and importing all the clips. Unfortunately you are working with one of the slower transfer rates available on computers. The average (good) copy time takes about an hour for every 100GB of data. A typical DVCPRO HD shoot recorded on P2 media might be a few hours of footage delivered on a 200GB USB drive. The import is faster than real time (compared to the running time of the footage), so about 7 hours of 720p DVCPRO HD (at 29.97pN) media might take about 2-4 hours to copy, based on your machine and drives. This is in addition to the original back-up time from the cards, of course. It’s slower with AVC-Intra, because some NLEs (such as FCP) have to transcode this codec during the import. On my MacBook Pro, the transcode to ProRes in FCP’s Log and transfer module was a little slower than real time.
RED footage makes time an even bigger issue. Most editors have been unhappy working with RED’s QuickTime reference files on substantial projects, like feature films. That’s because the QT reference files have to stay linked to the R3D camera raw files and are essentially “windows” that look into the 4K data and extract lower-resolution media on-the-fly. If you want to edit smoothly, then it’s important to transcode the raw files into something easier on your NLE, like DV25, DVCPRO HD, DNxHD or ProRes. In other words, edit using a standard offline/online approach to RED. Exporting transcoded R3D files with a general purpose computer is pretty tedious. Budget between a 3-to-1 and as much as a 20-to-1 ratio to go from RED One’s raw files to your NLE and be ready to start cutting.
Like any other tapeless media, RED camera files also need to be backed-up. REDcode is a variable bit rate codec based on wavelet compression. On average, the files (4096×2048, 2:1 aspect, 23.98fps) consume about 1.5GB for every minute of footage – or about 90GB per hour. An indie feature might shoot around 30 hours of footage, which puts that close to 3TB of required storage, just for the camera raw files. Times 2 if you rely on redundancy for extra safety. To compare, 1080p/23.98 DVCPRO HD would only use about half of that. Same for ProRes and about two-thirds for ProResHQ.
Tip 3 – Organizing files in your NLE
The hardest thing to get used to with file-based media is the cryptic naming conventions used by the cameras. When you import these files, you typically get long alphanumeric file names and not “Scene 1 / Take 1” or “Wide shot of person sitting on the bench”. Some NLEs will let you safely change the file or clip names. Others won’t. Avid has always let you do this, but it has traditionally been a no-no with Final Cut. Recent versions of FCP have made that safer with some formats, but I really urge you to resist the temptation. Remember that at some point you might need to relink media files or restore from the backed-up camera files. You are only going to be able to do this when the file name matches. Changing the name from “0014EF” to “Scene 7 / Take 3” might be fine and safe in an ideal world, but if all else fails and you have to resort to some type of manual search, keeping this name relationship the same will save your butt.
I recommend using one of the other bin description or comments columns as a place to assign a useful name. Both Avid Media Composer and Apple Final Cut Pro include numerous descriptor columns, so feel free to use these for custom names. You can also easily search and sort these, giving you the best of both worlds.
The other organizing factor is reel ID. Since there are no tape reels in the tapeless world, NLEs vary in their approach. Software like that from Imagine Products will let you rename cards. This is a wise approach. All too often, I have been handed a drive containing the contents from several cloned P2 cards. A volume for each card will mount on the desktop (on a Mac), labeled “No Name 1”, “No Name 2” and so on. What do you think is going to be on the next day’s drive? Same thing! So I urge you to properly name the cards in a consistent manner, using either film style (camera rolls) or video style (tape numbers) labeling. This may or may not be important for your NLE, but it is imperative if you have to locate shots on these drives in the future.
Tip 4 – Cataloguing your footage
You have been shooting with your RED One or HVX-200 for a few months and have started to accumulate a bunch of small FireWire drives holding the footage from each project. That’s easy to do, because the drives are so cheap that you buy a new one for each shoot. Just charge it off as part of the production budget, like tape stock. That’s all well and good, but now these are starting to pile up just like the camera tapes you used to have in the library. What’s the next step?
The simple and obvious step is to physically label the drives – just like your tapes. No wait – better than you used to label the tapes! Before you get buried in a pile of portable hard drives, start a cataloguing system. There is plenty of software to choose from and can be as simple or elaborate as you need. The main criteria is that the process be quick and easy when you want to know what’s on each drive or where to look for something shot during a given production. Choices include Apple Final Cut Server, Imagine Products, Bento, Filemaker Pro, CatDV or just an Excel spreadsheet. Whatever it is, start doing it yesterday!
Tip 5 – Mastering
I know, I know – it’s a tapeless world. The truth is, I still feel very comfortable having my finished production on a piece of tape. Most of my clients still own some VTRs. If you have to revise a project a year down the road, it’s often easier to ingest a videotape master and make revisions than to reload the entire original project from data back-ups.
My favorite mastering procedure is to generate four outputs of my edited sequence. These include a final videotape master of the edited program that is mixed, color-corrected and includes all titles and graphics. In addition, I will output a videotape submaster that is “superless” (no titles) with the audio in “stems” (separated dialogue, effects and music). Such a submaster makes any of the common revisions very easy.
That’s two of the four. Next, I’ll also export self-contained media files (such as QuickTime movies) in these same configurations – final master and superless submaster. This level of simple and easy protection neatly fits into the budget of most producers. For example, an hour-long, 1080i, 8-bit uncompressed QuickTime file with stereo audio requires about 400GB of drive space. Dumping a master file onto a FireWire drive is still more expensive than an hour-long HDCAM tape, but you can work with the media, even if you don’t actually own or have access to the tape deck.
Careful planning, organization and a policy for data management and protection will help you survive and thrive in the transition from tape to files.
© 2008 Oliver Peters