Storage Reliability

Recently I’ve written about storage strategies designed to future-proof access to your files. Other than questions of whether future software can still play your files, the biggest issue is whether of not the media is playable at all in a number of years. Unfortunately, there are simply no guarantees. All media can and does fail. Let’s look at various answers.

Everyone touts “the cloud” as the ultimate solution. Although cloud-based storage space is relatively cheap, the cost and data charges for massive uploads and downloads along with local internet speeds pose the stumbling blocks. There’s very little in the near term to change that. Remember, too, that cloud storage is a subscription service than never ends if you want to keep that media in the cloud.

The LTO (Linear Tape Open) data tape format is considered the “gold standard” for physical back-up and retrieval, but it’s really a format designed for long-term industrial and financial data applications. In other words, back it up once and forget it unless you need to restore from a backup tape in the future.

While many studios require original camera footage for major feature films to be archived onto LTO, the format doesn’t fit well into the needs of most small-to-medium production companies and post houses. There are three reasons for this: 1) As file capacities grow, LTO barely keeps up in equivalent capacity and transfer speeds. 2) The LTO standards keep evolving with limited forward or backward version compatibility. 3) If you need to continually go back to your archive to revise and update older projects, the linear design of LTO isn’t very attractive. In addition, frequent shuttling back and forth on LTO tapes to retrieve materials from random sections of the tape will cause an LTO tape to prematurely fail before its rated life.

One alternative to LTO is Sony’s Optical Disc Archive. It’s essentially a videotape deck-sized unit that records on writeable optical media (like a Blu-ray disc). They offer a robotic juke-box type of system for automated retrieval with large library systems. It’s a robust solution, but is mainly relevant to large facilities, such as at broadcast networks.

Storing on a large, RAID-protected array is a good, short-term idea, but it won’t be very cost-effective as your storage needs mount. I don’t recommend small 2-drive or 4-drive RAID enclosures for extended storage. These are more likely to have the RAID structure (whether hardware or software) fail and leave you will nothing accessible on that array. In my experience, single, enterprise-grade drives are more reliable. I buy these as raw drives (so I’m not paying extra for a power supply and interface with every drive) and mount them in a drive dock when I need to use them.

Hard drives do carry a manufacturer’s warranty for a rated lifespan, but I will reiterate that there are no guarantees. A 3-year-warranted drive may last as long as a 5-year drive and either one could fail in one year or last 10 years or longer. I currently have some drives that are as old as that. With drive failure is always a looming possibility, the reasonable strategy is to maintain multiple copies of any media of value. Three duplicate copies is recommend.

Let’s address how to select the drive to buy. Most of these types of drives come in several speeds and warranty levels. 5400 or 7200 RPM are the normal speed offerings. Both are fine for archiving, but 7200 is preferred if you occasionally need to edit directly from them. Warranties are usually three or five years. As with any physical media, it covers the replacement of the product, but not the value of the data stored, which you may have permanently lost.

A warranty is like life insurance. A 3-year drive isn’t necessarily better than a 5-year drive. The company has developed actuarial tables that tell them statistically enough of the  5-year drives last to the 5-year mark, so they won’t lose too much money by replacing the few drives that do fail. Sometimes the difference between three and five years may simply be that drives tested with more minor errors end up in the 3-year pile, while the ones with fewer errors go into the 5-year pile. I haven’t looked into the manufacturing specifics too deeply, but that’s generally how product warranties work.

With those two criteria in mind, I usually purchase 7200 RPM enterprise-grade drives with 5-year warranties. These are drives intended to be used in servers and shared storage systems running 24/7/365. There has been a lot of consolidation in the hard drive business, so regardless of the brand name, there are really only a handful of companies manufacturing the media.

One source to track which drives to buy is Backblaze. They are a cloud provider that publishes their testing results, based on a current pool of over 100,000 drives that they have in operation. Right now the front-runners are ToshibaHGST (Hitachi enterprise) and Seagate. The HGST brand has been absorbed by Western Digital. All these are good options. I also hold back on the largest drives rather than be on the bleeding edge. For example, you can now purchase 14 TB drives, but I’ll tend to stick with 8 TB for a while.

Mechanical hard drives are meant to spin and not to sit on a shelf indefinitely. Periodically load each drive into a dock and spin it up. Make sure the contents are still retrievable and files can be opened. This process should happen no less than once a year. More frequent is even better. And yes, if you have 100 drives in your archive, don’t get lazy. This needs to be done. If a drive sounds odd, has difficulty spinning up or mounting, or has lot of vibration, then clone and replace it ASAP, because it’s likely to fail soon.

Many spinning drives and solid state drives employ S.M.A.R.T. technology. This is a prediction of drive failure. Diagnostics fail the S.M.A.R.T. test when they determine that enough sectors on drive are no longer writeable. Other drive issues, like excessive heat and slow spin-up can cause errors. The drive may outwardly act and seem fine, but it’s time to clone and replace the drive. Shared storage servers monitor for S.M.A.R.T. errors in their RAID drives, but you can also get some diagnostic applications to test individual drives.

The final level of security is to develop a plan to routinely transfer your entire library to the current format of the day. If you use hard drives, then plan on migrating your library to a replacement within five to ten years. Many feature film operations, like ILM, have done that for years, because they sit on a library of material with a ton of value. Your media files, might not be that, but this should be a strategy you follow to future-proof your production investment.

©2019 Oliver Peters

DaVinci Resolve Editor Keyboard

Blackmagic Design doubled-down on advanced editing features in 2019 by introducing a new editing mode to DaVinci Resolve 16 called the cut page. They also added a dedicated editor’s keyboard – something that warms the heart of any editor who started their career in a linear edit suite. After some post-NAB feedback and adjustment, the keyboard is finally ready for prime time, running with DaVinci Resolve 16.1 (currently in public beta) or later.

Blackmagic Design’s Grant Petty comes from a broadcast engineering background and knows how fast tape editing was with the right controller. Speed is lost using a mouse-centric, drag-and-drop approach, so the DaVinci Resolve keyboard is designed to put speed back into modern edit workflows. Blackmagic Design was kind enough to loan me a keyboard for a couple of weeks of testing for this review.

Hardware design

The keyboard is very reminiscent of Sony’s BVE keyboards of the past. That’s not simply cosmetic – there are a number of plastic editing keyboards with a shuttle knob – it’s about precision engineering. The DaVinci Resolve search dial (job/shuttle/scroll wheel) truly feels like it has the same type of ballistics and tactile feedback that a Sony dial gave you. The DaVinci Resolve keyboard is built into a sturdy metal case with keycaps that are designed to take some pounding. They intend for the keyboard to last and will offer replacement parts as needed. In short, don’t think of this as a product you’ll have to toss out in a few years.

The keyboard connects via USB-C. But it also worked on the USB3.0 connection of a two-year old iMac and MacBook Pro by using a USB-A to USB-C cable. The back of the keyboard includes two additional USB-A ports for a thumb drive, mouse, or a DaVinci Resolve license key (“dongle”). The keyboard is wider than a standard extended keyboard due to dedicated edit keys on the left and the search dial on the right. It has a replaceable wrist rest on the front edge and adjustable feet to elevate the keyboard angle.

The Cut Page

The Editor Keyboard is optimized for the cut and edit pages. It does work as a standard keyboard in the color, Fairlight, and Fusion pages. However, I found the dial operation in those modes to be rather finicky. Outside of DaVinci Resolve, it’s a generic QWERTY keyboard, but the special edit keys and dial will not work with other editing software.

It’s hard to talk about the keyboard without delving into the cut page. While the keyboard works effectively and correctly in the edit page, you’ll still find yourself needing the mouse, which defeats the purpose. In short, the design motivation is fast editing where your hands never leave the keyboard. That ideal plays out best in the cut page and the two have been developed in tandem.

While the DaVinci Resolve cut page shares many similarities to Apple’s Final Cut Pro X, Blackmagic Design software engineers added a number of unique functions that improve editing speed. The best of these is the source tape view. The bin can be sorted by timecode, camera, duration, or name order using dedicated keys and then viewed as if from a single source – essentially a virtual string-out. Quickly scroll through the footage using the search dial as effortlessly as using the FCPX skimming function. Large, dedicated buttons for source and timeline, in and out, and sort methods make for easy navigation and quick assembly. Smart edit and special function buttons, such as the unique “close-up” button (automatically does a basic punch-in of high-res footage), round out the picture.

The cut page itself has a number of other unique features that are beyond the scope of this article. Nevertheless, one unique tool that is worth mentioning is the dual timeline view. The timeline pane is divided into a top mini-display of the full timeline, while the lower area always shows the zoomed-in section of the timeline at the current time indicator (cursor). You never have to zoom in and zoom out to navigate your timeline. The search dial makes it a breeze to quickly scroll through the full timeline (top) and then hit the jog key to zero in on the frame you want (bottom).

Trimming is where the dial shines. Dedicated keys quickly select in-point, out-point, roll, slip, or slide trimming. Simply hit the key and DaVinci Resolve automatically jumps to the nearest cut point. Then use the search dial for the rest. As you adjust the head or tail of a cut the rest of the timeline ripples accordingly. It’s one of the best trim models of any NLE.

Some additional thoughts

I do have a few quibbles. Trim functions in the cut and edit pages are inconsistent with each other. The cut page uses a similar model to FCPX, where audio and video from the clip are combined into a single timeline clip rather than on separate tracks. Unfortunately, Blackmagic Design has yet to implement a way to expand a/v clips and perform L-cut or J-cut trimming on the cut page. You’ll have to shift to the edit page to perform those.

This is a right-handed device, so left-handed editors will have the same dilemma that left-handed guitar players encounter. In addition, these are imprinted keycaps based on DaVinci Resolve’s default keyboard map. If you use a custom layout or one of the other keyboard maps that DaVinci Resolve offers, then the QWERTY command portion of the keyboard becomes less useful.

The search dial will not override the J-K-L or the space bar play commands. In order to jog once the sequence is playing, you must first hit the K key or the space bar to stop playback before you can properly jog through frames. Otherwise, playback continues the minute you let go of the dial.

Conclusion

This keyboard is addictive. But, is its $995 (USD) price tag justified? That’s steep, but many plastic gaming keyboards can run up to $200 and some even $500. That’s without any extra pointers, dials, or keys. I’ve also found precision metal keyboards with force-sensitive pointers as high as $3,000. Given that, Blackmagic Design may be in the right ballpark. Just like control surfaces for grading or mixing, this keyboard isn’t for everyone. If you are already a fast, keyboard-oriented editor, then the DaVinci Resolve Editor Keyboard may not make you faster. Likewise, a Final Cut Pro X editor who flies by skimming with a mouse is also going to have a hard time justifying the expense, not to mention a shift to a different application.

This keyboard is designed for DaVinci Resolve editors and not colorists. It’s for facilities that intend to deploy DaVinci Resolve as their full-time editing application. I could easily see DaVinci Resolve and this keyboard used in a fast turnaround edit environment, like broadcast news. Under that scenario, it will certainly enhance speed and workflow, especially for editors who want to make the most out of the new cut page.

Originally written for RedShark News.

Be sure to also check out Scott Simmons’ review at ProVideoCoalition.

©2019 Oliver Peters

Shared Storage Solutions

 

I’m certainly no IT whizz, but as an editor and all-around “workflow guy,” I’ve used and done basic management of a number of different shared storage solutions, going all the way back to Avid MediaShare SCSI. Shared storage solutions, aka storage area networks (SAN), have evolved from SCSI connectivity to Fibre Channel (both copper and fiber optic cables) and now to Ethernet. The latter set-ups are technically considered network attached storage (NAS); but to the user, there are only a few operational differences between SAN and NAS volumes.

A shared storage primer

In a nutshell, shared storage is a chassis of RAID-configured drives that can be simultaneously accessed by multiple workstations. Depending on the needs of the facility and the type of control software used, this storage can appear as one large volume to all users, or it can be parsed so that it shows up as several volumes with lower capacities per volume. Read/write permissions can be controlled in various ways. All users can have read/write access to everything or that can be selectively assigned by the system administrator.

The basic building block of a NAS is the main chassis, which contains storage, but also a small, on-board computer – the “brain” of the system. This is running its own operating system, which is usually a variation of Linux, CentOS, or Sun/ZFS. That internal OS is independent of whether the system is connected to Mac, Windows, or Linux workstations. That computer is the server portion of the NAS, which controls the drives, permissions, and the file structure. The server can be accessed from an external computer via the manufacturer’s installed applications – usually through a web browser. This is where the system administrator can adjust settings and handle general system maintenance, like installing firmware updates.

The volumes can be mounted by the workstations using a number of different network protocols, such as AFP, NFS, or SMB. Through these protocols, the files will look as you expect to see them from the Mac Finder or Windows File Explorer. However, it may not be perfectly compatible. For example, some file names using special characters that are valid in macOS, may not be properly read through one of these network protocols. So be very structured when using naming conventions for files that end up on a network volume. Numbers, letters, spaces, dashes, and underscores are fine. Avoid everything else and do not start or end a file name with a space.

The unformatted capacity of your system is based on the number and size of the installed drives. A 20-drive chassis populated with 8TB drives would tally 160TB. If you rebuilt that same chassis with newer 14TB drives you’d end up with a pool of 280TB. But, you cannot mix and match drive types or sizes within the chassis.

Most manufacturers offer the option to daisy-chain one or more expansion chassis onto this main server chassis. These are “dumb” rack units, meaning there’s no on-board computer in them – only drives with a power supply. Normally these don’t have to be the same capacity as the original chassis, if they are going to used as a separate volume. However, if you purchase and configure several matched units at the start, then they can be grouped together and used as a single volume.

The impact of RAID protection

NAS and SAN configurations are RAID-protected in various configurations. RAID-protection means that redundant data is spread across all of the drives in such a manner that one or more drives can go down without losing all of your media. However, that takes overhead, which means you must give up some of the total capacity to enable this data protection.

The standard set-up with a large rack unit allows you to lose up to two drives in a chassis without losing any data. If a drive is going bad or goes bad, the unit will continue to operate, but with reduced performance. In some cases that may not be noticed by the operator. When a drive goes bad, it can be replaced by a matching raw drive and the unit will rebuild the RAID data, which redistributes it across all of the drives again. This can take up to 24 hours to complete. While many manufacturers say you can operate during this rebuilding period, I have found that in actual practice, performance is so bad, that you don’t want to work during the rebuild.

RAID protection is a wonderful safety net, but at the cost of available storage. Different manufacturers have different ways of handling RAID configurations, so there is no rule-of-thumb as to what percentage you will lose with every NAS. For instance, 256TB of QNAP storage (gross) will yield 206TB of net storage. 480TB of LumaForge storage yields 316TB net. On top of this, the recommendation for all shared storage is to stay under 80-90% of the available net capacity for optimal performance. If you ignore that advice and decide to fill up your drives to something like 97%, your system will crawl and possibly not function at all.

Connecting the system

Most shared storage systems used in modern, small-to-medium post facilities will be Ethernet-based at either 1Gbps or 10Gbps (aka 1GigE or 10GigE). The topology of your network will impact the performance. Your server unit can be configured with individual Ethernet cards that would allow a direct run to each workstation. Or it may connect to an Ethernet network switch, which then distributes the signals to the workstations. Or a combination of the two.

The chassis and/or network switch(es) are connected to the workstations with Cat6 or Cat7 Ethernet cable. Cat6 is generally good up to 100′, while Cat7 is recommended for runs longer than 100′ or if the cable in routed through walls or in the ceiling close to other electrical wiring that can create interference. For a 10GigE storage network, the workstations will require 10GigE ports (like on an iMac Pro) or you will need to add a 10GigE-to-Thunderbolt adapter (Promise, Sonnet, Akitio) to the computer.

Storage racks are very sensitive to power fluctuations, so you’ll want a beefy uninterruptible power supply/battery back-up (UPS) unit. Since these chassis draw power, don’t expect to hook everything to a single UPS if you are putting in an entire equipment rack of gear. Small, desktop NAS units – no sweat. But a faculty with a larger system should plan on several UPS units for its installation. For example, at my day job, we have a large QNAP and a large Jellyfish system (more on that in a minute) – just under 3/4 PB total – plus other peripherals – all in a single equipment rack. Each NAS has its own dedicated UPS. The peripheral gear runs on a third. To make sure the gear also had plenty of juice, we had an electrician run additional dedicated circuits for each of the two UPS units used for the two NAS systems.

Finally, make sure you have adequate air conditioning, because excessive heat will damage electronics. Modern systems no longer require a meat locker environment, but an unventilated closet for a server/storage rack simply won’t do. Any room that falls into the cool to comfortable range for a human will be suitably cool for the gear. Staying on the cooler side of that range will be best for a room with a number of equipment racks.

Practical experience with shared storage in the real world

The creative content production company where I freelance as senior editor and “workflow guy” has had some history with shared storage. In the Final Cut Pro “legacy” days, we were running a sweet Fibre Channel SAN for four workstations. Media was managed through Final Cut Server software on an Apple Xserve computer, but with third-party storage hardware. Up until FCP7 everything ran well. Final Cut Pro X arrived and SAN usage with the early versions was to be avoided. Apple pulled the plug on FCP7, Final Cut Server, and Xserve. Then to make matters worse, the hardware reliability of our storage started to falter. As a result, the production company ended up back on local storage for a while.

Fast forward to about three years ago when we switched to a QNAP shared storage system. We quickly doubled the system capacity with an additional QNAP expansion chassis. Ultimately nine workstations were connected via a 10GigE network switch. General performance was good, but as we started to work steadily with 4K media, performance suffered, especially with nine editors banging away. For example, long-form Premiere Pro projects required a proxy workflow to avoid editor frustration. Certain tasks, like copying a multi-TB batch of files on one of the systems while editing proceeded on the others, slowed performance. Image sequence files really hurt overall system performance. You could not pull media from and render back to the same QNAP volume during Resolve render passes.

In looking for options to improve the system, we decided to shift to LumaForge and spec’ed a larger Jellyfish Rack installation. Other than system optimization (a biggie) the key difference in the two systems is architecture. Unlike our QNAP unit, which uses a network switch, we opted for enough on-board cards on the Jellyfish to enable a direct run to all nine workstations without a separate network switch. There’s also a small NVMe unit used as a dedicated Adobe cache volume.

We didn’t get rid of QNAP, though. It has been very robust and recent firmware updates have actually improved its performance compared to how editing “felt” with it before. We maintain it for some legacy projects (rather than move them to Jellyfish), as well as an additional back-up storage pool.

All workstations get Ethernet cable runs to both NAS systems, so any editor can access any media from any location – Jellyfish or QNAP. We configured Jellyfish with a tenth Ethernet direct port, which goes to a separate 1GigE switch. These Ethernet feeds are distributed to several staffers handling media management and file upload tasks, using MacBook Pro and Air laptops and a Mac Mini in the server room. The connection to Jellyfish gives them the ability to work with media files without tying up editing workstations.

The acquisition of the Jellyfish system has proven itself over time. Direct head-to-head performance between Jellyfish and QNAP with a small project or a few media files is not that dramatically different. But when we compare day-to-day workflow efficiency, the improvements add up. Long-form 4K edits can proceed with native media without the prerequisite of creating proxies. Sidebar tasks, like batch encodes and file copies on one or more stations, don’t impact performance of the other edit sessions. Image sequences are easier to deal with. I can render to and from Jellyfish when I work grading sessions on Resolve.

In general, both brands have worked well for us, but LumaForge has definitely provided an edge. However, I have no qualms about QNAP either for the right customer in the right situation. There are, of course, other shared storage brands that offer outstanding products, including Avid, OpenDrives, Facilis, Synology, and EditShare. If you want to build an all-Avid shop, then Avid storage is probably the best option for you. However, even though Avid storage works with other NLEs, shops that are focused on Premiere Pro, Final Cut Pro X, or Resolve are better served by the other options. In any case, deploying a NAS system is easier than it’s ever been. Heck, you can even buy and configure a smaller Jellyfish through Apple’s online store!

But do your homework, check your OS compatibility, and make sure you tap a workflow consultant who knows video post and not just IT. Plenty of NAS systems developed for the data world don’t perform up to par in the world of video post. And don’t go it alone, no matter how many YouTubers you’ve watched. Qualified systems specialists, like Bob Zelin (Rescue 1, Inc) or the teams at LumaForge or Avid or most of the other companies, can help you get your system up and running at peak performance.

©2019 Oliver Peters