Abstract

This conservation project aimed to develop a digitisation workflow for the magnetic tape collection of the Landesfilmsammlung Baden-Württemberg (LFS) in the Haus des Dokumentarfilms (HDF) to assist them in transitioning from a tape-based video-production workflow into a file-based one.

The empirical part of this study was conducted in HDF in October 2013 using data gathered primarily from the reports of the film archive and interviews with film archivist Anna Leippe. Supporting technical information was obtained through recommendations from product specialists in the postproduction and broadcasting industry.

A comprehensive investigation of the current workflow as well as the present condition of the LFS collection of the film archive was conducted in order to understand the need of a data-centric workflow in a film archive. The focal point of this project was the design of a transitional tape-to-file migration architecture to demonstrate its key steps, which also featured the necessary hardware and software solutions that best fit the technical requirement and at the same time the financial capacity of HDF.

Keywords

digitisation, small film archives, migration, workflow, magnetic tapes, documentary films.

The Challenges of a Film Archive in the Digital Age

The Haus des Dokumentarfilms is responsible for about 14,600 hours of various media and content that include three film collections in the state of Baden-Württemberg. About 8,000 historical film documentaries, 22% of the content, belongs to the collection of the Landesfilmsammlung Baden-Württemberg, and the collection has grown continually since its inception in 2000. From the LFS collection, more than 4,500 titles are already digitised and are recorded in Digital Betacam, IMX, DVCPro or HDCAM tapes, with some more titles still in analogue tape formats waiting to be digitised (see fig. 1). Due to the increased awareness of the existence of the HDF and LFS, the number of film acquisitions through donations over the years has steadily increased. The earliest documentary in the collection is dated as far back as 1904.

The growth of the collection, however, poses two major problems to the archive: public access of the content in high-definition digital format and a centralised repository of the physical and digital media.

Digitisation plays a vital role in the conservation and archiving of historical and cultural video. Outsourcing the digitisation slices a huge portion from an archive’s annual budget. Assembling a digitisation workspace in-house likewise requires investment in equipment, staff training, and maintenance. Since 2012, HDF has been using a tape-based video ingest assembly that can only output to DigiBeta tapes or standard definition H.264 digital videos. But standard definition format is no longer an option. The inability to deliver in HD from a centralised location not only inhibits HDF from complying to the current broadcast and film standards but also prevents them from responding to the public’s demands to instantly access the content in streaming high-resolution file format.

Table 1. The LFS video collection excluding celluloid film, CD, DVD, and audio media as of October 2013

Even though the LFS is supported by the Ministerium für Wirtschaft, Forschung und Kunst Baden-Württemberg (MWFK) and the Medien- und Filmgesellschaft mbH Baden-Württemberg (MFG), digitisation of analogue film material can be only funded through self-financing of the copyright owner or a project-related external funding. HDF’s income through the sale of film rights is too menial to support digitisation.

A central shared storage for data is non-existent. Magnetic tapes are still the main media carrier plus a handful of external hard drives stored in a climate-controlled room maintained at 42% relative humidity and 19°–20° C in the basement of the archive. All are properly labelled with barcode numbers generated from the FESAD database for identification and are encased in original manufacturer plastic housing (see fig. 2).

With the archive’s intention of acquiring more files in time for the 2014 launch of the online video portal project, the amount of data should not be underestimated. Assuming that 900 hours of SD material are to be converted to uncompressed HD video at 10 bit, 1920 × 1080 p, 25 fps, this alone will be equivalent to approximately 90.5 TB¹ of storage space. Moreover, video data must be organised, stored, and secured. In case of disasters, an off-site back-up storage must also be considered.

HDF outlined their media preservation goals for the succeeding years:

•  Produce more video files in HD quality to meet client requirements

•  Present LFS to a larger audience through the video portal

•  Increase income of the house through selling video clips from the collection

•  Capability to edit and process content

•  Have a unified workflow to be able to accommodate the files from digitising laboratories

•  Maintain a centralised data storage for the growing collection

•  Back-up the original analogue video tapes and store them in the Südwestrundfunk (SWR) storage facility in Endersbach

•  Back-up the video files

An Outdated Infrastructure

A tape-based workflow means that the source material comes from a tape medium and then recorded back onto another tape medium. HDF follows the same primitive linear process. In this case, the input material is either an analogue or a digital tape (VHS, U-matic, Betacam, DigiBeta or IMX). The analogue video signal is then digitised and is simply dubbed onto a blank Digital Betacam (see fig. 3). The step-by-step procedure of the current workflow is as follows:

Step 1. Once HDF receives an order, they produce a DVD of the requested clip from the tape library with burned-in timecode that serves as reference and copy-protection and sends it to the client via mail.

Step 2. When the client confirms the order using the clip timecode as reference, HDF processes order and prepares the Digital Betacam source tape for capture.

Step 3. Tape is ingested into VTR (video tape recorder) deck.

Step 4. After manually encoding the given timecode into the deck, the VTR captures video in SD PAL format.

Step 5. The captured video is dubbed out to Digibeta with at least 2-second in- and outpoints.

Step 6. The DigiBeta is sent to the client by post.

This system is no longer effective as the media becomes relatively inaccessible to the end user, who has to capture the clips to convert into more accessible files. In more cases finding a facility that offers the extraction of the data from a DigiBeta is becoming more difficult unless the users have their own capturing device. Likewise, the SD content is no longer accepted by broadcasters since their conversion to HD broadcasting. Even streaming videos are in HD and moving towards to Ultra-HD resolutions.

The Need for a Data-Centric Video Workflow

Digitising audiovisual media can be very complicated because of the many variations in the formats, equipment, and processes. In addition, there is the issue of format and equipment obsolescence, which is becoming increasingly prevalent, but preserving the original content of the media is important in tape and film collections conservation. Digitisation consequently allows access of the content without repeatedly using the analogue media. Scanning once, for example, allows the original film to be stored properly in a suitable climate-controlled environment. Migrating the content of videotape to a digital format will probably extend the shelf life of a carrier media that was never intended for archiving purposes. But once the media is digitised the next big question is storage and the data overflow. Developing technology has answers for that.

Unlike a full-service postproduction house that is required to have quick turnover of materials, HDF receives orders for clips, on average, once a week and with minimum of 1.5 minutes running time. Nevertheless, there is consistent demand for clips from all over the country and even outside Germany. The Internet alone is a great source of such videos but rarely displays acceptable quality. Users need material that is usable for HD broadcasting while 4K is becoming the norm in cinema releases. There is no escaping the UHD digital route. Everyone who wishes to present their works in this platform should only follow the technical requirements dictated by standards.

HDF needs to address this demand through its distribution of video clips. But HDF is not only concerned in presenting their film collection to the public but also is primarily tasked to preserve the contents and its representations as close to the original source as possible. Digitisation is the logical solution to this challenge.

The Benefits of a File-Based Workflow

Many industries, specifically broadcasting and postproduction, have already converted their video workflow into a centralised file-based infrastructure. Several film archives all over the world have also followed suit. This all happened within the span of a decade since HD digital broadcasting took off. Migrating from tape to file and overhauling an analogue system into a data-centric one will benefit a film archive in many ways:

Accessibility. The film archive would be able to access the media without sacrificing the original source, which is ideally kept in proper climate-controlled storage. All media will be in file formats and stored in a centralised storage that can be accessed by anyone in the network.

Consolidation. With a film archive such as HDF that holds four large film collections of the state, not only LFS but also the other collections in the house will benefit from this file-based workflow since all of these will be able to use the same digitisation workflow.

Flexibility. There will be the ability to move the video files around the network—from file transferring to non-linear editing and non-destructive processing of the media.

Self-reliance. A film archive will depend much less on commercial digitisation labs to have their videos digitised, which normally takes a period of time and sum of money. Therefore, having an in-house digitisation system provides savings in cost and time. In this manner, the archive has total control of their media, whereas when digitisation is outsourced, the objects are removed from the facility.

Profit. Income is generated from a faster and increased output. Apart from sales of the clips, the archive may also look into offering digitisation services to smaller local communities with film collections, private collectors, or amateur filmmakers who cannot afford the higher rates imposed by bigger digitisation labs.

Investment. The initial steps for the system upgrade may incur expenses, but this can only be seen as a profitable investment for the consequential growth of the collections, of which rights for distribution and use can be sold.

Broader audience reception. The convenience of being able to upload the video files in a web portal will give the possibility of presenting the collection to a much wider audience.

Personnel skill development. Staff can be trained to operate the equipment and understand the technical operations of the workflow even with the lack of in-house IT or digital intermediate specialist.

Transitional Tape-to-File Migration Workflow

At this point, HDF is not yet ready for a purely file-based architecture, therefore a transitional video-migration workflow that allows them to convert their analogue and digital videos to files is more feasible. Their existing set-up and equipment can still be utilised. A system upgrade rather than a complete overhaul is more practical. After all, they have the basic equipment necessary for ingesting the magnetic tapes (see figs. 4 and 5).

A workflow is necessary to be able to organise the tasks, designate responsibilities, and monitor the progress of the project. Basically the analogue media is ingested as digital files into storage for access and long-term archiving. The transitional workflow will be the crucial stage to complete this project. The workflow ushers the passage from a tape-based workflow into a data-centric one. The magnetic tapes will be the source material in this process and thus will need proper handling as they undergo digitisation and format conversion. The general tasks involved in digitisation are as follows:

Preparation and Inspection. The tape is prepared for capture. This means that it is checked for any sign of damage or deterioration. The equipment to be used is also cleaned, set-up, and tested. A test tape is used to check the operation ability of the equipment. The source tape is tested. Repair and cleaning of both tape and equipment is done, if necessary. Labels and documentation are also compared to the FESAD database to match file naming and metadata recording upon capture.

Capture. The source tape is ingested and digitised into the required format and specifications and then transferred to the network storage for future access. Metadata is collected.

Processing and remastering. The digitised media is edited, in some cases, digitally processed (removed of noise and drop-outs, colour correction, audio syncing, etc.), and converted to different formats and prepared for delivery or archiving. Metadata is collected.

Storage. The digitised media is stored into the network-shared storage (NAS). The media is also managed for archiving. Allowance for backup storage must also be considered.

Delivery. The digitised media is outputted to various formats for access over the Internet, optical discs, mobile devices, digital magnetic tapes, etc.

Figure 6 illustrates the principles of the new architecture as described below:

Step 1. All analogue sources (VHS, SVHS, Betacam, etc.) are converted to digital signals through an analogue to SDI converter and then fed through the existing Blackmagic Design (BMD) Smart Videohub via SDI connection. The digital sources are directly fed through the videohub.

Step 2. The digital output can now be viewed through the BMD Smartview monitors and at the same time can be processed using a hardware-based video processor (i.e., BMD Teranex 2D processor) for high-quality up/down/cross conversion from SD to HD, PAL to NTSC, or interlaced to progressive. The video processor can be directly connected to the workstation for remote control using a Thunderbolt cable.

Step 3. The processed digital signals will return to the Videohub for distribution.

Step 4. Optionally, the processed digital video passes through an SDI to an I/O interface that is connected to Workstation 1 that will do the capturing, metadata gathering, and/or editing (i.e., BMD Ultrastudio 3D SDI input to Thunderbolt output connected to an iMac). This is necessary when capturing using NLE Programs such as AVID MC or FCPX.

Step 5. Workstation 1 also functions as the network administrator. It is connected to the network switch that will connect to the storage and the other workstations all via Gigabit Ethernet.

Step 6. The files in are transferred to the 16TB NAS shared storage, which also functions as a file server.

Step 7. For backup storage an optional LTO appliance can also be connected to the server through fibre channel, mini SAS, or Gigabit Ethernet.

Step 8. The workstations will be able to access the media and do editing or previewing.

Step 9. Final output of video files is through a faster Internet connection through cloud, email, or FTP file sharing for delivery to clients and for web portal uploading. Additionally, dub-out to Digital Betacam or recording to other media is still possible.

Compromise to Deliver

After a series of tests to compare the end results of various digital HD video output formats, the one that best fit the storage capacity and at the same time, usage and visually acceptability in terms of image quality, is the Apple ProRes HD 4:2:2 25p (see fig. 7).

It may not be an archival format, but it is a widely used format in postproduction and broadcasting, where the video clips produced by HDF are mostly used. The reasons are that clients request this format, that most of the files will be uploaded to the web portal, and that the source material, a DigiBeta, is not the best quality anyway, so no need to go uncompressed. If they had to, HDF must go to the original source film and transfer it all over again, which is of course, very costly and time consuming. At the same time, HDF had to be realistic in utilising the capacity of their 16 TB NAS RAID storage and the recording capability of the workstation. HDF’s decision to use this format is for practical and technical reasons—a compromising decision that most archives typically face.

These HD videos are to be uploaded into HDF’s own web-based video portal, DOKSITE (still a beta at the time of writing), which is a major project of the archive that allows wider public access to their film collection. HD video web uploading is done automatically using FFMpeg,² whereas metadata gathering for the portal is automated using a script program.

Preserving Small Film Archives

Success stories of big digitisation, restoration, and migration projects of big audiovisual archives are commonplace despite the big challenges they have encountered. In reality though, there are several smaller archives in the world who are struggling not only financially but also technically. They can only dream of what larger and well-supported institutions can accomplish. They are in dire need of digitisation and migration solutions that balance the financial and technical capacity of the archive.

One may argue that the film collections in HDF are only documentaries done by amateur filmmakers, which may not have commercial value—so why digitise?

Documentary films are the personal narratives of individuals who were witnesses to historical events that had direct impact to their lives in that time and, as the viewers now, we are privileged to witness the events through their shaky lenses. Through them we see evidences of the lives they lived, follow a continuing tradition, or realise a disappearing way of life—their histories are learned so as to avoid the same mistakes in the future.

But are the efforts in digitising these films worth it? The images are raw, grainy, unsteady, and generally degraded. Digital reconstruction or processing will be too intensive, time-consuming, and costly, and yet it should also be argued that such films are supposed to be raw and that every millimetre of that frame tell a story on how the filmmaker dedicatedly worked on his home movie. The real value of a documentary film is the story in front and behind the camera.

A film archive is tasked to preserve these extracts from a historical timeline so that future generations will never forget their heritage. HDF is doing this job just as well but they need all the assistance they can get to improve their operations to provide better services to the public. They are faced with the challenges of a rapidly changing technology worsen by the debilitating and fluctuating economy and must learn to adapt for the sake of upholding their vital role as custodians of national visual treasures. They need to compromise between the longevity of the digitised media and storage capacity and easier accessibility. An uncompressed media may be the best archival format, but in some cases an alternative format fits a certain workflow. Their main objective is to be able to access and work with the digital media.

The solution they need is simple, available, and affordable. An effective digitisation workflow while utilising the appropriate tools makes it easier for a film archive to be adaptable to the evolution of moving images and at the same time capable of adhering to industry standards.

MICHELLE CARLOS is a freelance digital colourist and restoration artist and has worked on commercial and full-length feature film jobs in Manila and Germany as well as digital restoration work of archival films in Singapore. She graduated cum laude with a bachelor of fine arts degree from Tarlac State University. Since completing her master studies in new media preservation management at the Staatliche Akademie der Bildende Künst-Stuttgart (ABK). She has been involved in conservation and digitisation projects for audiovisual collections at the ABK Hochschularchiv and the Haus des Dokumentarfilms in Stuttgart.

Notes

1.  Calculation is based from estimated number of hours of the LFS collection (see table 1).

2.  FFMpeg is an open-source command line tool used to record, convert, and stream audio and video.