Archiving a TV News Segment

January 6, 2013

A few weeks ago during the peak of the holiday season our 3 1/2 year-old was interviewed for the evening news. It was a cute segment about the weather and his thoughts on Santa. He did great.

We wanted to save a permanent copy of the segment when it aired, but we don't use a PVR or any other TV recording device. Even if we did have a TiVo or cable-company provided PVR it is unclear whether a recording made with one of these things would serve as a good archive, given the proprietary formats and ubiquitous DRM.

What to do?

Most news networks now make their segments available for free viewing online. But have you ever tried to save one locally? It is enough of a challenge that I think most people would give up and just keep a link to the online version hosted by the news station. But again, this is not suitable for permanent archiving: links change, media companies and TV stations get bought and sold, and so on.

Here is the step-by-step process I used to download and store a permanent, high-definition version of the news segment on which my son was interviewed. I should probably point out that the legalities of following this procedure may depend on your geographic location and whether you have a reasonable fair-use claim on the content. In this case, as my son was the interview subject I think it's probably ok.

I should also point out that this procedure is highly dependent on the TV station website itself, and different websites might use different approaches, content delivery networks and so on. After reading through this procedure if you can't make heads or tails of what is going on you may want to delegate this process to a tech-savvy friend or relative. I would rate this at a difficulty level of 8/10¹.

1. Identify the URL of the specific page on the TV network website where the video is available to view.
2. Fire up Firefox with the Download Helper extension and navigate to the page.
3. Start playing the video.
4. If Download Helper picks up an .mp4 version of the video that is playing (not just Flash), you're done. Grab the .mp4, preferably in 720p or 1080p if available, and thank the lucky stars you don't have to go through the rest of this process.²
5. Brace yourself. This would be a good time to make a coffee or something stronger. Clear your schedule, hug your family.
6. Uninstall Flash from your computer. You want the TV website to think that you are incapable of viewing Flash. ClickToFlash and other Flash blockers are probably insufficient, although if you can find a way to selectively (but completely) remove Flash support from one of your browsers (preferably Safari), go ahead and try that instead.
7. Set your browser user agent to the one used by Safari 5.1 on the iPad. I used the Safari Develop menu to do this³. The purpose here is to make the TV website believe you are on an iOS device, and are capable of viewing high-definition H.264 video. This is our preferred video archival format.
8. Clear all your browser cookies and history. You want the site to forget that you ever visited with a Flash-capable browser.
9. Navigate to the page where the video is shown. Play the video and wait until the pre-roll ads are done and the video is actually playing. Right-click on the playing video and Inspect Element (in Safari).
10. See if you can identify something that looks like the video source URL, which (in my case) was a file called master.m3u8.
11. This is a UTF-8 playlist file, and we need to download it to take a look. Download the .m3u8 playlist file (use wget or curl if you'd like), and open it in a text editor.
12. We're getting warmer. This master playlist file references a number of stream playlists, each at a specific bandwidth level. Since we want an archive in the best possible quality, identify the line corresponding to the highest bandwidth rating. Note that this might not be the bottom-most line in the file. Download this stream playlist URL using wget or curl and open it in a text editor.
13. Here we go. We're now looking at a sequential list of video files, probably with the .ts extension, that together make up the segment we want to download. Use wget or curl to download each one of these ts files locally, making sure to keep the sequence number in the filename.
14. Recombine these .ts files (losslessly of course) into a single .ts file. I used tsMuxeR for this.
15. You now have a single .ts file that contains your video segment. If you'd prefer the video in an mkv container (which might be easier for your playback environment), you can use tsMuxeR to demux the combined ts file into separate video and audio streams, and then use mkvmerge to re-package the streams into an mkv file.

Voila. A perfect archival copy of your news segment in 15 easy steps!

_{1. Relative to what, you might ask. As a rough guide, consider difficulty 1 to be playing the video from the TV website, and difficulty 10 to be replacing the video on the site with Never Gonna Give You Up. RickRolling the nation is 10 for difficulty and 10 for style.}

_{2. Steps 1-4 are usually sufficient for almost all other difficult-to-grab video on the web. TV and news sites in particular seem to make things difficult.}

_{3. You should be able to use the User Agent Switcher to get the same effect in Firefox, but for whatever reason I couldn't get it to work right with the segment I was trying to download. Safari worked a treat though, and its Inspect Element feature comes in handy for later steps.}