Tweet Pitfalls in Digital Film Restoration

Unless someone objects, I would like to open a topic on the pitfalls of digital film restoration and "digital archival preservation" of images, moving or otherwise. (I think we would also have to enumerate the pitfalls of photochemical procedures as well, so don't feel I a singling-out digital as a whipping boy.)

This posting grows out of a discussion on many sites about the recent NY Times article on the expense and ephemeral nature of digital "preservation".

I was hoping that those who wanted to discuss this in a rational manner could be redirected to this site, rather than run the risk of violating other site's policies or simply running afoul of zealots on both sides who refuse to admit their preferred means of making and storing images might be less than perfect.

I have always maintained that "digital", for all that has been foisted-upon the term, is not the all-encompassing panacea for preservation and restoration problems in this World and would like to have a rational, considered discussion about the strengths and weaknesses the process in general and not a knock-down, drag out flame-fest that seems to always erupt online.

Up for it or not?

[Mike Gebert]

I'd be interested, but not especially qualified to comment... all I know is, I have family photos from the 1870s which I can scan and enlarge and improve today, and projects I worked on five years ago in digital media which you can't open today. That to me is a powerful argument for analog media.

[silentfilm]

Frank,
How often is a digital restoration transferred back to film?

I work for Hewlett-Packard, and well know the problems of technology advances making old version of media obsolete. For many of our large computer installations, tape drives are even becoming obsolete because there are "hot" mirror for data for many telecom applications that have to be available 24/7. And I used to have a shelf-full of laserdiscs, that I can't play anymore.

On the other hand, I know that a digital restoration can certainly remove imperfections from a film and really make it look really sharp.

Mike, no expertise should be required to jump in and explore, because I think the discussion should be as much about the assumptions and expectations that drive our perception of the term "digital" as it is applied to anything.

I also don't mean this thread to be a springboard for me to pontificate either (although I might have to be pointed to this statement on occasion, ahem!), because I don't have all the answers and hope the exchanges give unexpected insights that can help formulate meaningful answers to problems.

Bruce, it is the norm, at least in the case of the major studios and archives, that if the film has been digitally restored at film resolution, that it goes back to film elements for preservation masters, however there are quite a number of "restorations" done by transferring elements to less than film resolution (minimum of 2048 x 1556 pixels per frame 35mm) and electronically cleaning-up the image, with no lay back to film.

In the scope of all restorations performed in a year, I am going to have to assume that the total number of digital restorations is a very small fraction of the total -- at least in my experience, it has been...

Also, as you state, there must be made a distinction between the restoration process and the preservation mechanism; two very different things.

A few major concepts I wanted to explore were:

1. Digital image preservation and "archiving" is not lossless in any practical sense due to the very nature of digital files and the ongoing development of computer language.

This one sentence alone will take quite a while to examine and analyze because I made so many assumptions based on my understanding of the issues involved that it is practically a non-sentence to anyone who doesn't come from my background and experience and it is EXACTLY the type of sentence that starts heated exchanges on websites and listservs.

2. Long term storage of a filmic element as a purely digital file, is more expensive and problematic than a properly made film print.

We need to examine the real costs of storage of both types of carriers.

3. Economic pressures tend to drive preservation rationales in direct relation to the affluence of the entity that owns the title to be restored/preserved.

Without being simplistic and making the observation that archives simply need more funds, what philosophical breakpoints should determine when an archive should lean toward higher risk behavior in making their collections available rather than emphasizing preservation?

What happens when the cost of digitizing film elements drops below the cost of film to film copying, but remains problematic due to the fact that there is no universally accepted digital media considered archival?

Now, I am going under the assumption that we are only discussing motion pictures that originated on film or still images produced on film; no video -- it is a different beast.

Frank,

You bring up some good points, and I'm sure there's going to be intelligent discussion of them here on this forum.

My personal feelings about digital is that it is a tool to be used, but not the be-all-end-all that some people and institutions seem to think it is. Warner Bros., for example, has a digital-only policy. They will no longer be outputting to film. Others in the archival community and myself think this is nonsense.

Why? Because digital is not an archival format-- yet. There hasn't been enough time to test it, and from what we've seen so far, it's anything BUT archival.

On the other hand, 35mm film has the advantage of a) over 100 years of refinement and b) a physicality that digital does not have. A film print is tangible. It can be physically held and the information is photographic-- anyone with a light source and the right amount of tooling can create a projector or scanning device in order to recreate the motion.

On the other hand, digital adds SOFTWARE to the HARDWARE that film is simply. You've got to worry about programs, encoding, etc. Information is stored on magnetic media, which can easily be wiped at any moment, not to mention tape and disc rot. Anyone who has had a disc crash on them can tell you that you don't realize it until AFTER it happens. With issues like decomp and VS, you can see them coming and get to it before it's too late.

Digital is great, but it's a tremendously compromising bandwagon people are jumping on. We have a fantastic tool in our hands, but we're not using it correctly. I'd be willing to wager that what is done in the digital realm at the price of THREE photochemical jobs doesn't have to be done that way 80% of the time. Why not be economical? Why not use digital as a tool when it HAS to be employed, rather than throwing money at it all the time?

Jack, thanks for joining in the discussion.

I was unaware that Warners had gone to a digital only policy. That certainly wasn't the case when Dick May headed up their preservation efforts, but he retired a few years back. That is disturbing...

One of the problems, as I see it, is that it turns archivist and film handlers into database managers and film history, lore and physical knowledge evaporates very quickly.

And, as you point out, data preservation for archives and media holders is still very much in its infancy, so that is dangerous.

A real danger I see evolving is the tendency of some people to dismiss the dangers of digital migration as a consequence of hesitation to adopt new technologies -- simplistic taunts of "luddite" smack of near religious faith in new technology and have no basis in experience or the actual track record of a technology.

This danger extends to the very companies that make raw materials for the motion picture industry, mainly Kodak. While the entertainment division of Kodak has repeatedly assured the public that they intend to continue making filmstocks for the foreseeable future, the CEO of Kodak repeatedly has stated that Kodak's future is in digital imaging, so we not only have to fight the perception of film as being an "obsolete" medium from technology fans, we have to continue to convince a major supplier of their importance in the equation!

I agree that while digital is a very good tool to overcome some serious problems with source material for a restoration or reconstruction, it (in all it's various ways of being stored) is in no way a good archival format to archive film for two reasons;

1. Past and present physical data carriers for digital data are NOT archival; sorry, but the future doesn't count here!

2. The data word, once written to a carrier, unless very carefully migrated and extracted from the carrier, will be changed or corrupted, thus rendering the the "lossless" aspect of digital a fallacy. Bit by bit copying is grand but ultimately useless if the data can no longer be extracted for use OR is significantly changed when interpreted by software that changes the data by simply interpreting the file. There are any number of pitfalls here; multiple undocumented bit-depth changes, undocumented LUT (look up table) manipulations of images in software, forced use of incorrect color spaces, log files incorrectly translated to linear, etc., the list goes on...

The cost of digital restoration and "preservation", if done properly by laying back to film for the preservation master, is very high at present. You basically have to pay both the costs of electronic manipulation and the expense of laying the images back to film elements for preservation.

While it appears on the surface that the difference between what it costs to archive a reel of film VS a couple of Terabytes of data heavily favors the pure data, it must be examined beyond the cost of the carrier that holds the images.

Film Cost (assuming only from point of preservation of completed job):

1. Lab charges, film stock, processing, reel and can costs.
2. Transportation to archive
3. Annual overhead of physical archive -- building, staff, climate control

Data Cost (same parameters)

1. Data transfer charges.
2. Annual overhead of physical data archive -- building, staff, climate control, power consumption -
3. Archiving software and licenses of applications required to extract the data.
4. Salary or contract expenses of data professionals required to migrate data to new digital media as the old either goes obsolete or begins to decompose or both.

It doesn't take long to realize that, although it is more expensive up front to make a film element and store it in a controlled environment, the cost of archiving that element SHOULD remain fairly flat throughout its life from that point forward, while the actual cost of maintaining the digital files will probably rise more rapidly due to the need for frequent migration, data verification and possibly legacy software development when elements are required for exhibition or further restoration work.

However, there is also another fly in the ointment; access.

When an archive does a partial digital reconstruction/restoration of a film and outputs back to film, the archival needs are obviously meet, but access needs are not.

A plus to the fully digital restoration path is that you wind up with a data set, hopefully at full film resolution, from which you can extract all your derivatives (DVDs, streaming cell phone files, web streaming files, film recorder output files, etc.) and the ability to generate these derivative files is largely software based and can economically be produced either in a short period of time OR on the fly by dedicated content servers like the Ripcode Transcoding Appliance.

Of course you can feed the output of the restoration back into a film chain to generate access copies, but that is little less expensive in overhead costs than doing a film resolution scan itself, AND is often of limited use to an archive (this derivative must also be cataloged, archived and stored) so a conundrum arises.

It seems to me a modified digital strategy is in order, but access remains a real problem.

Frank,

I think you're quite right about your last point there. A lot of current restorations are not being done for posterity or archival purposes, but are tailored to be part of a home video release or broadcast on television. The incentive is not long term, and therefore, you have a temporary format that the studios have hyped up as "archival" even though it really isn't.

Ironically, it's this ease of accessibility that is going to be the downfall of the studios if they decide to start offering their latest features only in digital. My guess is that piracy will be made much easier because of the digital switch. How many people out there have access to 35mm projectors and film chains or telecines? On the other hand, how many people have PCs at home and how long will it be before the programs are in place that can master a fantastic recording off of some hard drive of a new film? No more sneaking a camera into the theaters, recording off of screens, etc.

Something tells me that the people that are making these decisions haven't thought this stuff out very carefully.

Economics is a big factor in archiving. If we look at the big failures in archiving of films to date they have been largely driven by the perception of the owners that the costs of archiving were not justified. The reality is that no form of archiving is free and thus the question of who is going to pay for it has to be answered before there is going to be archiving.

The expirations of copyrights are a two edged sword with respect to archiving. On the one hand, the expiration of a copyright reduces the value of an archived copy. On the other hand, it increases the number of people who might see value in restoring and preserving a work that is in the public domain.

I agree that digital transfers have their own benefits and problems with regard to film restoration and archiving. I think that one interesting factor with regard to the digital domain is that once a film is transferred to the digital domain it then adds many more people who have the opportunity to do something with the digitized film.

Jim,

Yes, I'd have to agree that budgets drive decisions as what to archive and its all relative to the type, size and affluence of the archive itself.

A government archive dealing strictly with films in the public domain has different concerns than a commercial studio with holdings stretching back to the silent era or a small archive that deals with regional filmmaking and runs on a strictly volunteer basis.

One may only have the budget to maintain proper environmental controls on their storage area and monitor health of their holdings, while others may have the luxury of prioritizing titles to be preserved/restored in each calendar year and others my base their preservation/restoration work based upon projected profits from ancillary rights licensing; it all varies.

It would seem to me that the real concerns that archivists have are that their dollars are spent most appropriately for their needs and meets the goals of their organization, rather than shoehorn them into an ill-fitting set of generic guidelines. That is not to say that there cannot be overall standards to adhere to, there SHOULD be, but that the finer details should be formulated by the parent archive.

The issue of copyright is an interesting one; as you imply, it all depends on the basic function of the archive; is it a repository of commercial assets, a vault of artistic artifacts, a public record of a government's work or whatever?

While copyright status can play a large role in the access policies of the archive, donor restrictions also can keep even public domain titles out of circulation, so copyright is not the sole determiner in what can be restored and made accessible to the public at large.

Donors, who are often film collectors, have strong, sometimes strange ideas as to what should or should not be made available to the general public and some archives reluctantly agree to these demands in order to save the material and worry about the implications later... makes for weird restrictions sometimes, that is sure...

The promise of converting moving images to digital file formats is improved access -- anyone (under the right circumstances) can download, save and manipulate a high resolution copy of a particular film.

Of course, this is also precisely the danger of converting moving images to digital file formats, it breaks an established mechanism that gives value to preserving and maintaining "official" versions of works of art/commerce and, in some eyes, devalues the need to maintain the originals or even a repository of originals -- argument being that a reasonable representation is available digitally.

Think of what an archive is; what precisely makes an archive? Most are known for specific collections, people who work there who are subject matter experts and the right to maintain access polices to said materials - collectively, they are the "experts" on their materials. Take this away, make the material ubiquitous and what do you have?

So, you are not only fighting rights holders for access, you will also be fighting archives to a degree and we will have to come up with a mechanism that allows access and yet allows the archive to continue to exist and maintain value.

When the proper context of the artifact goes away, it becomes... what?

In a "free market" (let's not go there) environment, the commercial archives of major studios are perhaps best equipped to function in that environment, as they have copyright and deep pockets to enforce relatively tight control over their holdings, but a strictly volunteer, regional archive may simply disappear for lack of funding if their holdings are dispersed to redundant digital repositories where they can be freely downloaded.

We will probably have to live with a more cumbersome version of "easy access" if we want what few non-commercial archives to survive the transition to digital based access of their materials.

l'm having a hard time imagining how or why it would cost $208,000 annually to store and access a digital copy of a film, as compared to $480 or so for film, as was quoted in the NY Times article.

That seems seriously out of whack to me, and as frequently happens when I read articles like this, it makes me think that the figures quoted are either not apples to apples or are seriously skewed in order to support the point that is being made.

One thing that is fairly safe to assume...the cost for digital will continue to go down in the future, while the cost of photochemical will rise. But, I don't think the cost differential now is anywhere near as large as stated in the article. If it was, cost conscious studios would not be so gung ho on digital.

--JA

Frank,

Thanks for the nice comment on the WB preservation at the time I managed it. I was not aware that they abandoned preservation to film, and think this needs comment from somebody with current first hand knowledge.

I just discovered this site today (thanks to Dennis Doros), and it looks like an intelligent group, not just a bunch of TV buffs. I hope to be involved in these discussions wherever appropriate.

Dick May

Frank,

Thanks for the nice comment on the WB preservation at the time I managed it. I was not aware that they abandoned preservation to film, and think this needs comment from somebody with current first hand knowledge.

I just discovered this site today (thanks to Dennis Doros), and it looks like an intelligent group, not just a bunch of TV buffs. I hope to be involved in these discussions wherever appropriate.

Dick May

Dick,

You are most welcome!

Warner Brothers Archival Operation was a tight ship when you were there; hope they appreciate what you did for them! I know everyone on this site does!

It would be interesting to know if WB has gone totally digital...

In speaking for myself (and probably a lot of others as well), Welcome!

I look forward to your knowledgeable posts.

Frank

Last year the Science and Technology Council of the Academy of Motion Picture Arts and Sciences published a booklet entitled THE DIGITAL DILEMMA. This very carefully outlined the comparisons betweel archiving on film vs. digital.

In general, the conclusion is that film is an inactive archiving item: complete the preservation, and put it away in proper environment.
Digital requires regular monitoring, with migration to new media every 5 to 10 years. There is also the obsolescence of the files, possibly making them unreadable over time as new, more efficient recording processes evolve.

The basic conclusion is that storing a 4K digital master is 1,100% more costly than storing a 35mm polyester film master.

I'm sure this isn't the last we will see on this subject.

l'm having a hard time imagining how or why it would cost $208,000 annually to store and access a digital copy of a film, as compared to $480 or so for film, as was quoted in the NY Times article.

That seems seriously out of whack to me, and as frequently happens when I read articles like this, it makes me think that the figures quoted are either not apples to apples or are seriously skewed in order to support the point that is being made.

One thing that is fairly safe to assume...the cost for digital will continue to go down in the future, while the cost of photochemical will rise. But, I don't think the cost differential now is anywhere near as large as stated in the article. If it was, cost conscious studios would not be so gung ho on digital.

--JA

I can't say that those numbers are right or wrong, but the misconception that "the cost for digital will continue to go down..." should be addressed. With each new technology, the older tech drops in price, but the cost for new hardware remains rather consistent.

For example, building a home PC with all modern parts has cost about $1000 for as long as I've been building them. As new CPU's, hard drives and memory become available, the price remains constant to keep up with the advances. Hard Drives these days are CHEAP. Why? Because SSD drives are the future, and they are NOT cheap.

In order to archive data, you'll always need to migrate your data from the old to the new, and each and every time you'll need to buy all new hardware and software. The new expensive kind.

And hope that it works! I tried migrating some data we had stored on DLT tape to LTO. Though we still had an DLT tape drive kicking around, we could not get the tapes to read. That data is GONE.

That said, I'm a huge proponent of digital restoration. Clean that baby up and lay it back down on modern film stock at 4K!

An older thread that I think is worth revisiting since things keep changing in the world of digital and 35mm film.

I think the digital restoration of a film and printing to 35mm film stock is really akin to creating a digital restoration of original audio recordings and creating a 180 gram vinyl lp release of the restored audio. You get a nice format in analogue form to satisfy a particular market and it will last decades as an archival format, but when all is said and done, you'll need to go back to those digital masters to make a further release or to do additional restoration work.

Since this original post in 2011, things have changed quite a bit. The continued manufacture of 35mm film is in doubt with studios moving towards digital distribution - companies like Kodak and Fuji aren't going to make 35mm film stock if the demand isn't there. Even archives will see it difficult or impossible to print material on 35mm within a few years, leaving digital archiving as the only way to go.

I'd disagree that there are disadvantages to digital archiving at this point. In the past, formats for digital stills, audio and video were varied and different types of software were required in the workflow for producers. Now, however, several formats are used as standards, similar to the way that Acadmey 35mm frames and widescreen formats were standardized by the industry many years ago.

What required for archiving is keeping multiple copies of files and copying to new media every few years. The difference between "analogue" 35mm and digital is that this copying is exactly like the original. Sure, media might get corrupted, but if multiple copies are held in different locations, the copying is "bit for bit" with no degradation from one copy to another.

I had to deal with this in archiving my own work as a filmmaker. I shot three feature length works on analogue Hi8 video in the 1990s. I found that the analogue originals had started to deteriorate - I transferred them to digital and still keep the digital versions as my archival copy. I keep them in different locations and copy them to new hard drives every few years - it's the raw footage for each documentary, varying from 20-30 hours of footage. The format I'm storing them in is a standard that's been used for several years and, if something new comes along, can be easily converted.

Digital archiving has many advantages over analogue that don't really become apparent until you see the long-term of how preserving works can be accomplished in the digital domain.

It is a fact that, if made properly, Wetgate, Finely Graded, Estar based Intermediates (I am including SFGMS, Photochemical and Digital Y-C-M's, Duplicate Negatives and Interpositives) are the only way to preserve both Nitrate and Acetate based film for centuries.
I have proven (at least to myself), that a Photochemically preserved element (again, If made properly, not on the cheap), can provide an outstanding scan. And, at the very least, provide a final product equal to a final product derived from the Original Negative. By the way, the digital houses love to scan from priceless Original Elements, as it costs the studios far more to re-master from an Original Element than a finely timed wetgated Intermediate. (Some studios are scanning their entire library at 4K for future use. This, in my opinion, is a pity. Although, I would gladly participate in a scanning project of that magnitude).
There are more than a few of us that believe that a Photochemically preserved element provides a better scanning tool than an Original.
Both "Yellow Submarine" (IP) and Magical Mystery Tour" (Blow-up Colour Duplicate Negative, Eastman 2272) were scanned from these Intermediates and the Colour Dupe. "Help!" was transferred to Hi-Def from an IP and "A Hard Day's Night" was transferred to Standard Def from a new Fine Grain Master Positive (Plans are being formulated to create a 50th Anniversary release and Blu-Ray on "AHDN").
In the case of "Magical Mystery Tour", much more detail was revealed from the Blow-Up 35mm Colour Dupe. than the scan from the 16mm A&B Ektachrome Original (Both "YS" and "MMT" were custom scanned in 4K by Christopher Dusendschon). Yes it is expensive to Photochemically preserve elements. But, you can digitally downscale, but never digitally upscale successfully. That means that if you have scanned a title in 2K or Hi-Def, it is now close to being obsolete. Pro-HD and 4K is becoming the norm and 8K is just around the corner. Thus, the obsolete scans will have to be re-scanned again from the source element(s). Much as VHS had to be re-transferred for DVD and DVD has to be re-transferred or digitally scanned for Blu-Ray and so on and so forth. And Nitrate is headed down the road to decomposition extinction, as Acetate is susceptible to Vinegar Syndrome and Colour fading.
UCLA Archives, at the Packard facillity, has set up a freezer vault for both Nitrate and Safety Film. But not all Original's are, or have been, kept in such pristine conditions. Acetate Colour film will continue to fade and vinegar syndrome runs rampant in all Acetate based films. The jury is still out on the fading of modern colour film stocks.

A Sony LTO tape is guaranteed for 5 years. That is just for the tape, not the content. And, I have seen spinning discs fail duringdigital reconstruction. I have also seen some marvelous work done digitally, like "The Invisible Man" with Claude Rains. It no longer looks exactly like B&W Nitrate Print film, but it is highly palatable to the human eye. And a beat up, basically lost show has been resurrected. "TIM" was scanned in 2K. But, the digital medium the restored version has been tarred on, is fragile and prone to failure. As far as I know, a duplicate B&W Negative has be filmed out on "TIM". In my opinion, a 4K scan would have been better choice, as the title will probably have to be remastered again in the future. Do we really want to have people untrained on the nuances of Motion Picture Film handle priceless Photochemical elements over and over again? I think not.
In the case of "Magical Mystery Tour". A Photochemical protection element has been created and a 4K digital Y-C-M neg of the re-mastered version is being shot, as we speak. "MMT" could not be "restored" photochemically as the original Ektachrome has excessive built in artifacts and golfball sized grain. These artifacts had to be removed digitally. The same is true for Yellow Submarine". Although a very nice Photochemical Answer Print was created from the USA Version Original Negative, which does not contain certain sequences that makes up the "definitive" version, now on Blu Ray.

Don't blame me. Blame it on the Bossa Nova.

Frank,

Thanks for the nice comment on the WB preservation at the time I managed it. I was not aware that they abandoned preservation to film, and think this needs comment from somebody with current first hand knowledge.

I just discovered this site today (thanks to Dennis Doros), and it looks like an intelligent group, not just a bunch of TV buffs. I hope to be involved in these discussions wherever appropriate.

Dick May

Hi Dick.
This is not true. First hand knowledge is that Warner Brothers is creating digital Y-C-M's and Dupes on all of their projects.
For as long as Film Exists. Ned is still championing photochemical preservation.

Paul

Hmmm,

Seems like I know everyone here. LOL! Thanks to Mike Feinberg, I found this site.

Paul Rutan jr.

Hi Paul,
Glad to see you on this forum. I expect a lot of knowledgable comments.

As to your comment about WB's producing preservation elements, you are referencing my first post to Nitrateville, several years ago. Much has probably happened since then, and I hope your information is correct. After 20 years working with WB's historical library, I still have an obvious affection for it.

Hi Paul,
Glad to see you on this forum. I expect a lot of knowledgable comments.

As to your comment about WB's producing preservation elements, you are referencing my first post to Nitrateville, several years ago. Much has probably happened since then, and I hope your information is correct. After 20 years working with WB's historical library, I still have an obvious affection for it.

Yes, Dick,
You are missed. Especially when you missed defects whilst tending to your projectors.

My information is directly from S.A.

Paul

Hi there, Luciano here.

I have to add that I really agree with what Jack Theakston said in his past post.

Jack Theakston:
"Digital is great, but it's a tremendously compromising bandwagon people are jumping on. We have a fantastic tool in our hands, but we're not using it correctly. I'd be willing to wager that what is done in the digital realm at the price of THREE photochemical jobs doesn't have to be done that way 80% of the time. Why not be economical? Why not use digital as a tool when it HAS to be employed, rather than throwing money at it all the time?"

Jack Theakston was very right about that.
However, like others have said, film lasts so much longer, and that's why I feel that digital (WHEN NEEDED ) should be an intermediate and not the final step. Have you ever wondered what will happen to Oscar shows or all the news coverage from 911, etc. All these historical events have been captured and archived on digital, and whether people will be able to see these things in 500 years really makes me think.

I'd also like to truly emphasize how incredible film is. I was very fortunate to be able to screen (by cranking a 1905 movie projector with carbon arc) an original nitrate print of Buster Keaton's "One Week". It was like watching a whole new film, even though I had seen it a million times! Even the version on Blu-ray couldn't capture it's luster. I think that really goes to show that when treated with respect, including nitrate, film can last a long time.

Adieu!