Abstract

The research on the optimization of digital video archives restoration is conducted through marginal analysis. Based on the interaction among three production factors (the contents, the techniques, and manpower in the restoration projects), an optimal scheme is formed for reference.

Keywords

digital restoration, restoration principles, technological process, marginal analysis.

Introduction

The passage of time is increasingly aging the historic video archives. The restoration and utilization of these archives has, therefore, become a major issue for current archives management. The digital restoration proposed in this paper uses the frame information (i.e., HSV) to fill and repair the missing and damaged parts of digitized video archives, in particular those recorded on magnetic tapes. It aims to restore images and restore them close to the original condition.

The frequently used technology of video restoration mostly relies on the film postproduction and restoration equipment. However, there is a lack of available equipment to restore the types of damage that particularly affect broadcast magnetic tapes, as shown in figure 1. At present, the restoration work is often performed manually with the consequent low efficiency and high cost of investment in equipment and labor. These problems directly restrict any improvement of the efficiency of restoration.

With the rapid development of Internet applications in multimedia, the demand for the reexploitation and reutilization of precious video archives grows. In order to balance cost and value, the authors, from the perspective of marginal analysis, try to optimize the production process of restoration to maximize the benefits.

Production Issues on the Digital Restoration

The digital restoration of video archives is a long-term process during which the following issues occur frequently.

Low Use Rate after Restoration

Taking actual data (see table 1) of the restoration and reuse in Shanghai Audio-Visual Archives, the annual use rate of the restored video in the archives increases at first, and then decreases with the progressively increasing capacity of restoration.

At the beginning, the use rate increases with the restoration amount, but as the restoration amount increases to a knee point, the use rate stops increasing and a decreasing trend shows up (see fig. 2).

Table 1. Annual restoration and use

High Restoration Costs

So far, the easily available standard products which are widely used in restoring film damage (i.e., the technical equipment can be selected and purchased directly from the market) cannot fully meet the restoration requirements of damaged magnetic tapes. That leads to high costs both in manpower and equipment for the tape restoration.

•  High cost of manpower. Because of the lack of available restoration equipment for magnetic tapes, more manpower has to be allocated to restore magnetic tapes than is required for the restoration of damaged films. That leads to high labor costs.

•  High cost of equipment. In addition, the standard products selected and purchased directly from the market for use in the restoration of magnetic tapes often require purchasing both hardware and software equipment. That leads to a high cost of equipment but poor compatibility with the surrounding system.

Slow Improvement of Restoration Efficiency

The slow improvement of restoration efficiency is reflected as follows:

•  Heavy workload required by manual restoration. The features of the current system using the standard products show that more manual restoration is needed to keep the machines operating. The manual operation a six-month training period. Therefore, simply increasing the available equipment does not raise restoration efficiency.

•  Lack of skilled and experienced personnel. Video archives digital restoration demands high levels of competence (excellent painting skills, image quality appreciation and computer equipment control, etc.) from the operational staff. Hence, professionals are in urgent need.

The Three Production Factors for Digital Restoration

We discovered that three production factors affect the efficiency of video archives digital restoration. They are (1) contents, (2) techniques, and (3) skilled manpower.

Experience has shown that the efficiency of restoration follows the law of diminishing marginal returns. That is, the gains made through improving one productive factor while maintaining the other factors will, sooner or later, progressively decrease.¹

•  Supposing the content to be restored is fixed, an increase in production will not directly increase the utilization of the restored content.

•  If the restoration technology and equipment is unchanged while the strategy of restoring content remains constant, the initial rate of restoration will increase if manpower levels are increased. However, beyond a certain level of manpower increases, the growth rate of production is going to decrease.

•  In addition, more restoration equipment encourages new demands for the existing restoration technology and equipment environment until a bottleneck is reached.

From this it is clear that blindly increasing the labor force or the available equipment does not always enhance the marginal returns of archival restoration.

When the marginal benefits of restoration surpass the marginal costs, the restoration project will be in a state of high value; when costs surpass benefits, the project will be in a state of low value.

Hence, in order to maximize the marginal returns, the input amount and proportion of the most economical factors of production should be set, in the light of the quantity of the restoration. In order to make the greatest gains from the work to improve the efficiency of the restoration processes, the quantity of material to be restored using the most economic methods must be controlled. From the perspective of production factors, the following is going to analyze the interaction among them.

Analysis of the Contents to Be Restored

The restored content is analyzed using the following four factors:

1. Value Classification

The archives can be divided into three grades according to their value:

•  Important: the archives (1980s) reflect most recent social development

•  Very Important: the archives (1950s–1980s) present social changes

•  Highly Important: the archives (-1950s) show the history of the society

2. Amount to Be Restored

Shanghai Audio-Visual Archives, at the moment, holds a total of about 510,000 hours of digitized video archives with about 123,700 hours of them requiring some level of restoration. Listed below is the ratio of the content requiring restoration from each of the three value grades:

•  Important: 70.5%

•  Very Important: 29.2%

•  Highly Important: 0.3%

3. Use Rate after Restoration

The recordings to be restored are chosen at random, but generally speaking, the larger collection has a greater probability of being used. Therefore, the estimated use rate of restored video archives is ranked by value-grade as

Important > Very Important > Highly Important

4. Degree of Quality Damages

Materials from different periods suffer different types and degrees of damages:

•  Important: damage include spots, flickers, scratches, etc.

•  Very Important: most damage is irregular scratches which specifically belong to magnetic tapes

•  Highly Important: most damage consists of vertical scratches, mildew stains, and so forth, which usually arise on the tapes transferred from films.

To raise the use rate of the restored archives, the production plan should be arranged intelligently by the selection of content to be restored by a combination of the estimated use rate and the value distribution. Meanwhile, to improve the restoration efficiency, restoration techniques appropriate to the damage should be adopted (see fig. 3).

Analysis of the Restoring Techniques

Alternatives to the restoration techniques for digital video archives can be provided by

•  purchasing the standard products (mentioned before as topological standard products, mainly suitable for film carrier); and

•  developing in-house techniques and systems (on the basis of the types of magnetic tape damage of the holdings).

As for the video archives on magnetic tapes, we recommend the self-developed software system to conduct the restoration.

For instance, the irregular scratches, as a type of damage of magnetic tape (see fig. 1) are caused by magnetic particles shedding or head clogging of the playback machine. The scratches often appear in thick horizontal stripes with black on the left and white on the right. This varies with time sequences but the position of scratches remained basically unchanged. This kind of damage can be restored by using a damage detection algorithm² based on perceptual code. Extreme points can be labeled through a gray morphology processing method, so that the scratches can be automatically detected using similarity and connectivity rules of perception.

This restoring algorithm brings about a clear improvement in the efficiency of restoring this kind of damage. In addition, improving the level of intelligent restoration software is also helpful for its efficiency.

The feature analysis of the self-developed system and the standard commercial products are shown in table 2, which indicates that the self-developed system and the standard products have their own strengths in restoration. The self-developed system has lower topological cost, higher restoration efficiency and better compatibility.

Table 2. The feature analysis of the standard products and the self-developed system

Note: Average restoration efficiency means ratio of the playing length of the contents to be restored to the real time spent in restoration.

Analysis of the Manpower

The third important production factor of digital video archives restoration project is the manpower.

1. Total Investment in Manpower

The total investment in manpower depends on the cost of the restoring equipment and the environment of the restoring hardware.

2. Specialized Skills of Individuals

Restoring digital video archives requires more operational staff with a higher comprehensive level of training and experience. Without the relevant experience guiding the restoration and operation, it takes a longer time to train the skilled operators. What’s more, the specific restoring effects and efficiency rely heavily on operators’ personal experience.

Summarily, how to strike a balance among these three production factors is the key issue the restoration project of massive video archives faces.

•  The selection of content for the restoration plan should consider the estimated use rate and value distribution.

•  The selecting of restoration techniques and the investment in equipment should be appropriate to the damage types of the archives to be restored. Then standard products can be purchased or restoration software can be developed in-house.

•  The distribution of the labor force complies with the total amount of restoration to be undertaken and the investment in the training in the techniques to be used. Personnel training, especially the transmission of restoring skills from experienced staff to newer colleagues should be paid great attention to improve the production efficiency.

The following discusses the solutions used to create more marginal restoring returns in the Shanghai Audio-Visual Archives.

Optimizing the Restoration Workflow

The digital restoration of video archives should adhere to the mode of project management, that is, be operated and maintained. The restoration workflow should be optimised by managing the following procedures:

•  Making an annual restoration plan

•  Perfecting the working process

•  Improving the restoring quality evaluation system

•  Establishing a restoration database

•  Updating the equipment and techniques

In the light of the three production factors, the best scheme with the best investment ratio has to be devised to maximize the marginal benefits.

Optimizing the Content Selection

There exists a certain connection between the selection of the contents to be restored and its use rate after restoration. The use rate tends not to be ideal if we only consider the value determined by age when selecting the content to be restored. Neither is the restoration efficiency at the highest. According to the feature analysis of the archives’ value grades in 3.1, it is impossible to maximize the marginal benefits while blindly selecting the contents to be restored. We should:

•  Distribute the restoration ratio of archives in the different value grades by the use ratios over the years.

•  Estimate the future use rate in light of the market demand and then make pertinent arrangements for production.

Table 3 shows the proportions of content to be restored that we put forward for an annual plan.

Table 3. Proportioning of contents to be restored each year

Optimizing the Technical Line

1. Different Technique for Different Damage Type

Table 4 explains the different restoration conditions for different value-graded archives:

•  The important archives (1980s–): have higher image quality and a real-time restoration with simple noise reduction algorithm could be adopted.

•  The very important archives (1950s–1980s): have various types of damage and can be restored mainly by real-time restoration supplemented with various algorithms.

•  The highly important archives (–1950s): have a higher estimated use rate and can be restored frame-by-frame to restore the images to their original quality.

Table 4. The contents to be restored and technical requirements

2. Adopt the Self-developed Techniques

Magnetic tapes, because of the shedding of magnetic particles, are likely to produce large quantities of irregular horizontal scratches. The effect of the existing home-made products, however, is limited when faced with this kind of damage. It is considered, therefore, that the independent research and development of restoring software system to improve horizontal scratches on magnetic tapes, is the best technical line.

Shanghai Audio-Visual Archives, considering the characteristics of the types of damage, has developed a digital video restoration system for dealing with the following 10 typical damage types:

•  Color distortion correction

•  Contrast and brightness correction

•  Noise reduction

•  Flicker elimination

•  Random spots removal

•  Big blotches removal

•  Definition improvement

•  Small scratches removal

•  Irregular scratches removal(the restoration effect see Figure 4)

•  Large lost images check and repair, etc.

The algorithm for handling the pictures’ fundamental attributes (i.e. the correction of color distortion, contrast and brightness and improving resolution) can adjust the parameters to improve the picture quality by employing individual experience. The rest of the restoring algorithm has two main steps: automatic detection and automatic repairing. Automatic detection is used to locate and estimate noise by computer vision model in both the spatial and temporal domains. The results of the automatic detection are used for interpolation based on domain correlation by motion estimation and motion compensation at the stage of automatic repairing. Our system has functions of task allocation and process operation, which are suitable for large-scale industrial application.

Optimizing the Human Resources

The human resources optimizing can be achieved by these two suggested methods:

•  Enhance the professional restoring skills of personnel by standardizing the restoration principles and procedures and by strengthening knowledge training.

•  Reduce the burden of professional demands on the operators by upgrading the restoration system with more intelligent functions based on the experience and knowledge gained from operating the restoration system.

Here we have two questions raised to explain further about the above methods.

Q1: What are the restoration principles?

•  Select and preserve the best edition

Before setting about the restoration, it is advantageous to search for the copy of the material with the highest definition and the least damage from the collection. If necessary, the material can be redigitised and the more distinct fragments can be edited in to replace damaged sequences, by which basic information (i.e., colors, brightness, etc.) and adjustment parameters can be provided to the follow-up restoration. As to those archives which cannot be restored by means of the existing techniques, a relatively untouched edition should be kept for future technological innovations.

•  Maintain the Original Untouched

The main principle of restoring historical video archives, “maintain the original untouched”, tells us that altering the original material to “improve it” using subjective decisions is not allowed. We should try to recover the original features, keep the brightness and tonality of the initial videos, repair damage caused by extrinsic factors (i.e., physical damages, humidity, mustiness, etc.), rather than painstakingly try to beautify the damage.

•  Balance Quality and Noise Reduction

In the process of digital restoration, reducing noise always brings about the reduction of image quality. Under such circumstance, noise reduction must be based on the premise that normal recognition and identification of the contents won’t be disturbed, instead of focusing too much on its effects.

Q2: What is the Guiding Process of Restoration?

This following guidance is proposed. It is offered as a result of repeated verification and correction by practical restoration experience, and will keep being optimized and supplemented in the future operations.

(a)  Browse the entire video archives, and section them in accordance with still/motion, black and white/color, single damage/composite damage;

(b)  Record the basic parameters (i.e., brightness, contrast, colors), and unify the restoration targets;

(c)  Observe the defects using the relevant algorithms, and adjust the parameters to improve the restoration effect;

(d)  Add the corresponding algorithm for different sections, and record the section to be restored again;

(e)  Record the parameters of the algorithm, and when the optimal restoration effect is achieved, try to apply them to other sections;

(f)  Because the sequence of the algorithms affects the restoration result, try to determine the best operating sequence of the algorithms if there is more than one algorithm for one section;

(g)  Restore the relevant sections two or three times;

(h)  Examine and revise the image in coherence caused by adding different algorithms in sections;

(i)  Carry out manual restoration on parts of the frames for fine repair.

Overall, the optimization scheme for digital video archives restoration is summarized as follows:

•  Optimize the content selection: Figure out a reasonable proportion of contents from each of the different value-grades for an annual restoration plan. We advise an inverse ratio between the value-grade and the planned restoration volume.

•  Optimize the technical lines: Adopt the software and system developed in-house for magnetic tape restoration to improve the efficiency; choose different algorithms for different types of damage.

•  Optimize the human resources: Reduce the dependence on professional skills to cut the high cost of labor force, by carrying out the management of experience and knowledge, upgrading the equipment and increasing the intelligence of the systems.

Prediction of the Optimized Scheme

Comparison: The Optimized and the Non-Optimized

According to marginal benefits theory, the above concrete optimization scheme was proposed after deep analysis of the following three production factors:

•  The contents to be restored

•  The technical line

•  The human resources

In order to maximize the marginal benefits and promote the development of industrialized restoration, we make a comparison between the non-optimized schemes and the optimized one based on a set-up restoration project model.

Now taking the example of Shanghai Audio-Visual Archives, a restoration project model is set up with:

•  3 restoration workstations (PC)

•  3 restoration operators

•  264 working days annually

The comparison is illustrated as follows between the non-optimized scheme and the optimized one.

Table 5. The annual plan of restoration project non-optimized

Table 6. The annual plan of restoration project optimized

(** 7 hours of per working day in the tables)

As the comparison shows, the optimized annual restoration scheme (see table 6) has archived higher marginal benefits, which means higher outputs and use rate after restoration with lower marginal costs in both manpower and equipment. The specific optimizing measures are as follows:

•  The contents to be restored: the equal distribution has turned into different proportions adjusted according to the use demands

•  Techniques and equipment: Independent developed techniques and equipment help to make great improvement in the restoring efficiency of the very important archives

Actual Data of the Optimized Scheme

We consider the use rate after restoration as the highly important factor for evaluating the restoration benefits; hereby especially analyze and compare the actual use rate after restoration between the non-optimized situations and the optimized one.

The actual data of yearly restoration projects recorded by Shanghai Audio-Visual Archives in the past 6 years shows that the use rate after restoration has been improved in 2015 after optimization (see Table 7).

Figure 5 illustrates the trend of the annual use rate after restoration from 2010 to 2015. The dotted line indicates its trend if the optimization had not been carried out in 2015.

Table 7: Annual Use Rate after Restoration

Conclusion

Digital restoration of video archives involves many factors. Through marginal analysis, archive institutes can put forward methods to increase outputs and improve the use value by researching the interactive relationship among the three main production factors: the contents to be restored, the techniques, and the manpower.

•  Allocate a reasonable proportion of those archives to be restored according to the data analysis and market demands prediction, to improve the use rate after restoration.

•  Restore the archives with the mode of project management, form a reasonable management mechanism, and keep to perfect techniques.

•  Follow the three restoration principles, and accumulate and correct repeatedly the knowledge of restoration, to raise the restoration efficiency.

Considering the cost and benefit, the exploration of the restoration techniques and maximizing the marginal benefits of large-scale production will be our eternal goal.

YANG HAISHENG, Senior Engineer, has long worked in the field of broadcast information technology management, and is director of Copyright Assets Centre of Shanghai Media Group (SMG) and Shanghai Audio-Visual Archives, former vice director of the Information Centre of Shanghai Media and Entertainment Group (SMEG), former director of General Engineer Department of Shanghai Media and Entertainment Group (SMEG).

JIANG HUIJUN, works as an engineer engaging in broadcast media assets management in the Technology Department, Copyright Assets Centre of Shanghai Media Group (SMG).

Notes

1.  Tao, D. (2003). Marginal utility analysis of transportation production of transportation enterprise[J]. Journal of Wuhan University of Technology. (2)

2.  Yang, Z. (2012). Automatic detection method for line scratches in old film and video [J]. Ideo Engineering. (7).