Future Challenges Facing Historians

As a documentarian, I have mostly been drawn to working on historical films. My first experience in this genre, a dipping of my toe in the proverbial water, was a documentary short about one of the remaining Dentzel carousels owned and operated by the City of Burlington NC. That prompted me to work on something vaster and so my first feature-length documentary was about the Regulator Movement in pre-Revolutionary North Carolina. I am currently working on another documentary piece concerning Burlington and which is set in the 50’s. All of these works have one obvious thing in common, which is that they all depend upon the historic record whether it is written, painted, photographed, or filmed.

It occurred to me that, while this day and age is very convenient in terms of digital content, that very convenience will become the bane of decent historic research decades from now. Historians and documentarians will face challenges 20, 50, or 100 years down the road that can only be prevented by addressing them now.

As we begin this discussion, consider your own digital materials. More than likely, you have a treasure trove of photographs you snapped on various digital cameras and on your cell phones. You probably have several wonderful threads of communication in some Email system, be it Gmail, Yahoo, Outlook or some other provider. Maybe, you have some precious videos shot on a digital media video camera or on your cell phone. If you are a writer, you might have an online diary or you maintain an active blog either of which you regularly update.

The question which you may well be anticipating is this: Where are these precious collections of digital data stored and do you implement a valid backup plan to prevent their loss if a hard drive fails or if you lose your cell phone? You see, once those files disappear into the big bit-bucket in the sky, they are gone forever. Rephrasing that last statement a bit we can say that a piece of your personal history, your family history, your community’s history has disappeared forever.

Of course, traditional sources of historical research can also be destroyed by fire, flood, rot, war, and so on. True, but how likely is a fire or a flood to wipe out your home versus the statistical probability of a computer hard drive crashing? I would dare to say that is a number of orders of magnitude less likely. Our little mental exercise serves to touch on one small aspect of the many problems that face historians in the future, that of storage. Add to this the challenge of searching for it and the challenge of being able to use today’s digital material at some time in the distant future.

So, let’s get into the challenge of storage itself. If you are a very astute computer user, you may well have dealt with the above scenario by saying to yourself, “This won’t be a problem for me because I back my data up.” Ok, that sounds really good, but what exactly is your backup strategy? Do you back up to an online service (e.g. Mozy)? Do you make physical backups on DVD-ROM or BDROM? Do you back up to other drives? Or, if you are a very computer-savvy person, you may do proper backups to several generations of drives housed in different safe locations. Perhaps, your precious memories are work-related memories housed within some governmental or corporate shared drives which are backed up every night and stored in data centers scattered around the country.

All of these strategies expose some flaw and other strategies exist that overcome those flaws. If you back up to disks that you keep in the same location of the original drive, then you run the risk of some disaster (fire, flood, tornado) destroying them both at the same time. If you back up to an external service, then you have overcome that flaw but now you have another dependency which we will discuss shortly – what is the longevity of the company that provides that service? If you back up to other drives and keep generations of backups, are you making sure that you exercise those drives every month or so to prevent them from seizing up? If you back up to DVD-ROM or BD-ROM disks, do you diligently check that the backup is perfect and there are no flaws? Do you protect the surfaces of those disks from any sort of scratches or dings? Should something happen to you and you are no longer around, are there other people around you who know where these treasure troves of data are stored and how to ensure that they are preserved?

I know that there are some who are literally shouting at the screen right now that they keep their memorable photos and videos on online services such as Facebook, Flickr, YouTube, Picasa, and Vimeo. You are saying, “Should I lose the photos here, they are always available there and I can always download and print one from there!” Sounds reasonable until you realize that these services probably won’t last forever. Yes, Facebook, Google (YouTube and Picasa), and Verizon (Flickr) are solid companies today, but what will happen to them when the next big thing arrives in a few years? Where is MySpace today after Facebook stomped them? Will these services be around another 25 years? How about 50 years or 100 years from now? Sites like the Internet Archive may help preserve snapshots of their pages but even this is a service that may go belly-up one day.

The majority of corporate and government entities do have archival policies in place but even they can lose critical information that would be important to historians at some later date. Emails, photos, videos, and other recordings may be handled by backup strategy policies but many fall prey to traps set by lawyers. Yes, there are many entities that have legal retention policies attached to their archival policies. Items are run through the digital shredder in a year, 3 years, 5 years, 7 years, 10 years, and so on depending upon the legal nature of their content and thus are not preserved based upon their historical value! Daily, we lose terabyte after terabyte of material that may be significant to some later generations.

Another challenge facing the historian of tomorrow, assuming that precious digital assets are properly saved and transmitted from generation to generation, is the challenge of finding them. Consider that, with traditional photographic and written assets, these may be stored in drawers or even in boxes in an attic. Whenever they are located, it is clear what they are and an historian can quickly browse through them and determine that they are relevant to their research or not. Even home movies on 8mm film can be unspooled a bit and held up to the light and one can gain a quick understanding of what kind of material may be on the spool. How does the same process occur in the digital realm?

How often does someone find a hard drive or a stack of backup disks and immediately be able to view their contents. Disks may be formatted differently depending upon the source operating system and the preferences of the original operator (FAT, Fat32, NTFS, XFS, EXT2, EXT3, EXT4, HFS, etc.). Is the media compatible with today’s equipment? Will it be compatible with tomorrow’s? What if the contents of the disk are the result of some proprietary backup system and not plain files? Would the drive even be pluggable to anything available in the future? For example, what would you do today if someone produced a stack of 5 and 1/4 inch diskettes and said that they contained the backup of critical images archived with an early Apple Mac computer? Where would you even begin to physically read the data and be able to get it into some form that you could use? The point of this is that as technology changes, storage formats change also.

Even if the documents, images, or video are online somewhere in the giant web soup, how would you be able to find them and derive a proper provenance for the material? A large number of family-history images are uploaded by people with the generic names assigned by their camera (IMG0720.JPG instead of JoeAndSuzieQHoldingNewborns.JPG) and so there is not a good tag for locating them. If images have been posted to a private page on a site such as Facebook or Flickr, or a diary is posted privately on LiveJournal, all bets are off in doing any sort of generic search for them. They are, essentially, lost to history forever.

Yet another challenge faces tomorrow’s historians assuming they are able to overcome the previous two. It is the challenge of being able to actually open the material. This becomes more of a challenge with video or correspondence file formats than with photographic material in most cases. Standard photo formats such as JPEG or PNG are well-documented and very mainstream. PNG may have a bit of an edge because its specification is open-source, completely within the public domain. For more serious images, there is also an open-source RAW format created by Adobe called DNG (Digital Negative). TIFF image format is in the public domain also but can contain runs of proprietary data so it is not necessarily completely safe. Most RAW formats (used by serious photographers) which are essentially digital negatives, are proprietary to camera manufacturers such as Nikon, Canon, and Sony. Probably as long as these manufacturers continue to produce cameras these formats will continue to be supported.

Video is a completely different animal. In order to support moving images in an efficient manner, video is encoded using specialty algorithms that compress the data and permit it to be streamed easily to the playing device. The encoders, known as CODECs (Coder/Decoder), are very specialized and very proprietary software components which are used to accomplish these tasks. Note that the file extension of a video file does not have anything to do with the CODEC – a .MOV file or a .MP4 file container holds a video that is encoded with this proprietary format. CODECs include H.264 (several flavors), ProRes (Apple), Cineform (GoPro), DNxHD, and so on. Unlike a home movie’s negatives which can be easily seen, a video is a closed box which may not be visible ever. Why I say “ever” has to do with the fact that some CODECs are invariably tied to a specific computer’s operating system in such a way that they won’t work on others. There are older proprietary CODECs which are 32-bit Windows 95/98 only and which are unavailable for any other devices.

A word of warning about CODECs. There are a lot of sites which offer “CODEC Packs” that seem to be treasure troves of support for lots of these old CODECs but downloading their product merely introduce some nasty virus into the target computer. Mainstream codecs generally are readily available in Quicktime, Windows media players, or the excellent VLC media player. Top end editing packages such as Final Cut, Adobe Premiere Pro, Avid, Magix Vegas Pro, and so on also tend to provide as many CODECs as they can legally license.

Correspondence also falls into this same boat. We have already discussed the longevity problems of online Email services, but consider that they are literal black boxes for searches. How long will they keep their data after the death of the user? How can their contents be archived in any meaningful fashion? Local Email systems such as Outlook do provide the ability to archive their data files with a proper backup process, but their contents are contained in proprietary data formats which only can be viewed by their host program. What will happen with these treasure troves of correspondence information 50 or 100 years down the road when Microsoft Outlook is some distant memory like the Commodore VIC-20 is to us today? The sad thing is, how could someone properly preserve their correspondence short of printing every Email to paper or to an open format such as PDF? Even the most diligent of us would tire of this process very quickly!

Electronic diaries and journals, either online or proprietary software versions, fall into the same boat as Email. The contents of these caches of historical information are just as unsearchable and un-archivable as their correspondence peers.

So, in closing, the purpose of this article is to sound a warning about a problem which we documentarians and historians will encounter soon. I dare say that some have already run into some of these issues depending upon the timeframe of history that they are researching. I, sadly, have no real answers but it would behoove historians to pair up with software folks and attempt to find ways to tackle these problems now. Whatever technological improvements that can be discovered to fix these problems need to work into the mainstream soon so that John Q. Public’s contributions to the writing of our history by future researchers will be available and usable.

Advertisements

Green Screen Layout

I wanted to throw up a quick mini-article just to show my ideal green screen setup.  See my earlier article Green Screen Made Easy for more information about green screen setup. I had to pull this diagram together today for a friend so I figured it was worth several thousand words about green screen techniques.

“And the Winner is…”

It is interesting to watch election cycles play out because the supporters of the two major parties tend to follow different rules. One strategy is a winning strategy every time and the other strategy tends to be a toss-up.   One group of supporters keeps their eyes on the ball and rarely deviate from it while the other group is, well, fickle in execution.

In my observations, Democrats, for the most part, look at the agenda and the platform their party has proposed to make that agenda happen. They rarely, at least after the primary season is over, consider the individual candidate’s character. Additionally, their definition of liberalism is wide enough to encompass viewpoints that extend from centrist through left-leaning principles.

Republicans, on the other hand, seem to confuse primaries with general elections. Instead of pulling out their microscopes during the primary and examining each and every candidate in terms of their character, their electability, and their specific platform (and deviations from the general core beliefs of the party), many times they are lured by other reasons. There are so many definitions of conservative within it any centrist-leaning candidates are parsed and branded as liberal! Finally, when the stakes are high in the general election is when they start applying “Character Matters” and other mantras that they should have used in the primary cycle to select the best possible candidates.

Once the general election cycle begins and their Convention is over, Democrats lock their focus on the win and very rarely will they lose sight of that goal. Republicans, however, will waffle all over the place and threaten to not vote to appease themselves, their families, their constituents, instead of keeping focused on the big picture. It appears to me that they don’t grasp all that is at stake. Cycle after cycle, crucial political battles are lost because large registered Republican voting blocks decide to grow consciences that they should have had during the respective primary seasons.

Democrats and Republicans tend to differ in another key philosophy in the general election. It appears that once Democrats purpose to do their civic duty, they go forth and ensure that they vote. They realize the cost of not voting and “get” that when they don’t vote, they are essentially casting a vote against their party. They understand that when they don’t vote for their choices that their agenda has no chance of moving forward. Even if the person is not ideal, the platform must go forward.

Republicans, on the other hand, are notorious for deciding to stay home on Election Day or (worse) for choosing to vote for an Independent or a Democrat because some aspect of the running Republican doesn’t sit right with them. Somehow it does not sink in that either of these courses of action weakens the party’s platform or renders it inoperable. It appears that such voters think that they are choosing the better of two candidates and don’t realize that, at this point, it is less a matter of the candidate than the political aspirations of the entire party that are at stake.

After decades of this electoral behavior we can see the results. The Democratic Party is strong and focused on meeting its goals while the Republican Party is in shambles and is, quite frankly, a joke. The solution for Republicans is twofold. They need to, as a group, start to hone in on the big picture and not let themselves be swayed during the general election cycles and they need to become very astute and informed while picking candidates during the primaries. Notice that “astute” doesn’t mean “nasty and brutal.” It means wise and discerning. If they don’t, the party is doomed.

“Green Screen Made Easy” Does What It Says!

When I first entered the fray as a fledgling movie-maker in the opening months of 2014, I had a grandiose vision of filming my documentary’s talking head experts on green screen and placing them against my backgrounds.  It seemed so mind-numbingly simple in concept that nothing could go wrong…

Right, it became a nightmare and I deserved the pain for not being properly prepared for this critical aspect of the documentary.  How I wish that I had this book in-hand while doing the ever-important planning for these interviews.  Much of its sage advice I have gleaned since my first dabbling in green screen but this was not all found in one convenient place as it is in “Green Screen Made Easy.”  While this article will be partly a review of the book, the rest of it will be a reflective look back at my rookie mistakes that many newcomers to chroma-keying are affected by.

In filming “The Regulators” in 2014, I had five on-screen subject matter experts.  Two were female and three males.  While chroma keying is gender-agnostic in principle, it isn’t in terms of fine details such as hair.  Actually, the three males sported short hair.  One of the females had longer hair and the other had a full, flowing set of hair.  This lady was my first interviewee and I was completely naive in setting up the lighting for her.  Needless to say, my results ranged between disastrous and  mildly acceptable.  I know that I pushed Keylight in After-Effects as hard as I could but with my 20/20 hindsight, I could have made much better lighting choices and better utilized the resources at my disposal.  This is where a book like this one can help a beginner to get up to speed within hours or, at most, a few days.

Here is an example of the problems I created and could have avoided early on.  This is a perfect example of what not to do.  Ever.

Another problem which I did not include in the picture is that the subject to screen distance was too close.  This made the green spill even more intense.  Also, this was shot on a DSLR with 4:2:0 color which really caused issues with the fine details and the spill.  Some of the footage of the interview was totally unusable while the others were marginally useful.

I learned the hard way and was, over time and with lots of diligent investigation online, able to resolve these issues. My later interviews for that same documentary improved but were not perfect by a long shot.

If I had this book in hand I would have discovered that within the first few chapters of the book, all of these items are dealt with and some additional important suggestions are presented!  One of these is using backlighting with a complementary color to the screen color and also ways to make the screen glow in its color.

Now that I am reworking “The Regulators” for a director’s cut 250th anniversary edition, I was able to re-shoot the interview with this person.  Notice the difference in the image quality by shooting on blue screen with a 4:2:2 Atomos Ninja and concentrating on the lighting of both the screen and the subject.  The procedures that I used were similar to the processes outlined in the book. Also in the book are discussions about how to optimize the camera and its ability to capture as clean an image as possible of both the background and the subject.

Once good images have been captured then they need to be processed properly.  Using the matte keyer is not a simple “press the button and Hey! Presto!” it is done in most cases.  It is important to know how to manipulate the results to optimize them.  The book leads the reader through important techniques such as how to reduce jagged edges, reducing artifacts, and ensuring that you can pull as clean a key as possible.  Several software packages are presented along with their workflows.  Another aspect of the book that makes it an invaluable resource for the filmmaker’s bookshelf is that it touches on how to deal with several problems that may crop up such as dealing with holes and reflections.

A decent section of “Green Screen Made Easy” deals with selecting and even making your own green screens and cyclorama stages.  One thing that I appreciate with how the authors deal with this subject and the subject of lighting is that they present a number of options that are within the reach of beginner who may be on a shoestring budget all the way up to well-established and well-funded film makers.

The final section of the book introduces several subjects that need to be considered.  Chroma-keying is not done in a vacuum, it has to believably exist within the realm of the composite.  Light wrapping, dealing with shadows, considering the directionality and color of light on the subject relative to the background, and other such important topics are explored.  We have all seen the typical green screen extraction pasted against a background which doesn’t match in tone, color, or shading and have immediately been able to say, “That is some bad green screen!”  Even the crucial topic of adding grain is discussed.

 

 

Open Mic Ethics?

It has been literally forever since I have posted a blog article.  The poor blog is the victim of too many chores and too little time.

Anyway, I have to take the time to write this short article because I have been stewing over it for the past week or so.  As an aspiring professional in the film and broadcast world, I feel compelled to speak my mind about the rash of leaks of off-the-record open-mic information.  Last week, there was the release of a recording of Donald Trump and Billy Bush that could not even remotely be considered to have consent of either party.  Now, before someone points a finger at me and labels me, let me remind you that the same thing happened to the then-candidate Barack Obama talking to Putin.  This ongoing rash of misbehavior, while heralded enthusiastically by opponents of the victims, strikes across political boundaries.

My focus is upon the total lack of professionalism in the industry itself and is based upon the simple fact that in order to accommodate the convenience of body microphones, proper lighting and focusing of cameras, and ensuring that a set is ready beforehand, people are wired and posed in advance.  Anything said or done while not officially on the air must be considered sacrosanct and if this mess continues happening, political and other well-known figures should refuse to be wired up until they are actually on air.  How would the industry like that?  The convenience factor is not an excuse to become a sneaky news-gathering organization!

This is even more disturbing considering how Americans are opposed to illegal wiretapping and even the brouhaha of a few years ago concerning the NSA’s tracking of cellphone conversation metadata.  The capturing and releasing of off-the-record audio and video is spying.  If a person on-camera but off the record decides to pick their nose, so be it.  That must never be released.  If a person on audio makes a statement but is off the record, it must be as if they had never said it in the first place.  To do anything else is, well, totally unprofessional.  It casts a black-eye on the entire industry.  What’s next, to air a wired person’s visit to a bathroom?

Something needs to be done about this.  If there is not integrity in the individuals who are essentially the guardians of their craft, then perhaps it is time for the courts to step in.  Maybe some healthy payouts and even some jail time for everybody involved in the distribution of illegally-obtained recordings might put a cap on the problem.  Wouldn’t it be better for all, though, if we all just apply professional ethics to what we do?

What Easter is all about

Anyone who knows me well knows that Easter is my favorite time on the Christian calendar.  Without Easter mankind would be hopeless!  In the course of three days, our sin-debt was paid off and we had the example of resurrection to give us hope for our eternity.

Anyway, I worked up a script and shot the following with the son of our music minister at Crosslink Community Church.  Hope that you like it as much as we enjoyed making it.

Adobe Creative Cloud Suite is Junk!

The title says it all.  Ever since the CC model appeared, the Adobe video tools have been on a steady march downhill.  The march is that of reliability and the march is that of robustness.  I had to make my feature documentary “The Regulators” using Sony Vegas Pro 13 because Premiere Pro CC had a slew of unbelievable problems.  That was CC 2014 but now that I am on CC 2015.2, there are still unbelievable issues plaguing it AND plaguing their flagship After Effects!

Maybe it is the new licensing model that has made Adobe lose its competitive edge.  Before, they had to strive to produce awesome software that would entice folks to loosen their pocketbooks and purchase such expensive software.  However, everytime that they did, they felt good about what they had done…the software allowed them to get their work done and, even if a problem popped up, their tech support worked diligently to get around the issue.  Now, they don’t have to care that much – every month or year, a subscription is collected from the umpteen suckers like me who hold out hope for things to improve.

Case in point – here I am sitting for a 28 second segment of video to render.  It is taking over 2 minutes to render 1 second of video which is already prerendered to previews!  Note that I am not strapped for memory and my processing power is not necessarily a slouch.  There is nothing spectacular in the segment in question – just some H264 video with an UltraKey.

This goes hand-in-hand with a very weird problem in After Effects CC 2015.2.  The segment is a very simple Shatter and yet produces a phantom garbage frame in the middle of any render.  Tech support was brought in on it last Friday and has nothing to offer yet.  My question is, if AE doesn’t do its basic functionality then why have it?  Why pay for it?

So, once again I find myself at the crossroads.  I have Sony Vegas but really don’t feel it produces the absolute best.  It certainly doesn’t feel as polished and nice as Adobe’s products do when they work properly.  FCP is an interesting option but I will definitely need to make a very large paradigm shift in my life to go with it.

I just wish that Adobe would get its act together…

 

(Nov 23, 2015 – To add insult to injury, I went to purchase a piece of Adobe Stock a few minutes ago just to have the cart crash while checking out.  “Your purchase cannot be completed due to an internal error.  Please call xxx-xxx-xxxx for immediate sales assistance.”  Really?  Purchase carts are common and relatively easy to handle and yet Adobe can’t even get THIS right?  What a mess….)