[ale] OT converting old videos, cataloging dvds, storing forever
Ron Frazier (ALE)
atllinuxenthinfo at techstarship.com
Tue Oct 16 12:29:58 EDT 2012
HI all,
After a few hours of Googling and reading forums and such, I've got some
new data that I wanted to share. My little video conversion project
seems to be hitting a snag. The 800 hours of videos I have are in the
category of important to me, but not critical. If I lose them, I won't
die tomorrow. So, the snag I'm hitting is the cost in terms of storage
cost and time to convert them. I don't have a huge budget, certainly
not all at once. I wanted to share my results, and see if you guys
think I'm missing something.
In terms of time, I think it will take me 10 - 15 minutes of tinkering
with equipment, software, discs, cases, labels, etc. per one hour show
to convert. More if I do edits and cut commercials and sync the sound,
etc. Perhaps less if I store only to HDD and don't edit. That adds up
to 200 hours of my time, a non trivial amount. Obviously the video tape
must play, but I don't necessarily have to watch it.
Here is what I believe to be the state of the art in storage options.
I'm not considering other types of tape at this point. I'm also not
considering SSD, based on data retention issues I mentioned in another post.
Option 1 is using HDD's to store the videos.
Option 2 is some other type of disc, bluray, dvd, or mdisc.
Option 3 is online storage.
Based on my reading, I have concerns about the long term reliability of
bluray, so I'm probably dismissing it.
Let's discuss discs first. Assume a single layer single sided DVD can
hold ~1 hour of good quality video at a data rate of about 8 Mbps. For
a HDD, assume 4.7 GB / hr of video. Note, in some of the forums where
people burn LOTS of DVD's and spend hours checking their error rates
etc, it was mentioned that 16x write speed DVD's are not reliable. So,
if available, I'm mentioning only 8x DVD's here. There are also many
types of dyes, and very confusing data, which I'm not going to try to
sort out here as I don't understand it all. There are may types of
printable surfaces on the top of the disc, for automated printing of
labels. Lightscribe and inkjet surfaces are examples. These usually
increase the cost. I don't need them. For my purposes, writing on the
disc with a DVD safe sharpie pen is enough. Based on my reading, double
layer discs suffer more compatibility problems so I'm not considering
them. Double sided discs are much more likely to get scratched and
cannot easily be labeled so I'm not considering them. It is actually
very hard to figure out all the DVD media options, but I've selected a
few to look at. Prices are Amazon, if they sell the product in question.
Gotta account for the case for the DVD, about $ 0.25 each.
The DVD-R format seems to be the most universally compatible. I'm not
trying to list all discs here, just get representative prices. Of
conventionally constructed discs, Verbatim seems to have a good
reputation, but there are varying reviews. Longevity estimates for
conventional disc media range from 5 years anecdotal to 50 years from
the factories. I will say that I have some 5 year old Maxell DVD-R's
that are still readable, but I will probably copy them to archival discs.
VERBATIM 94852 DVD-r media 8x 4.7gb 50-pk spindle (shiny silver) - $ 19
/ 50 = $ 0.38 / disc. Cost with case (cwc) = $ 0.63 / disc
Verbatim Shiny Silver DataLife Plus 8x DVD-R Spindle, 50 Discs - also $
19 / 50
It seems that JVC Taiyo Yuden is the top of the heap in terms of quality
for conventional media. Taiyo Yuden bought part of JVC as I understand
it and they're now shipping with JVC part numbers and packaging. Many
people still know the Taiyo Yuden name and so that's usually listed in
the product listings. Apparently, some TY discs themselves are
unlabeled, so if you want TY, make sure you get TY.
100 Taiyo Yuden/JVC 8X DVD-R 4.7GB Silver Thermal Lacquer - $ 29 / 100 =
$ 0.29 / disc. cwc = $ 0.54 / disc Better product, cheaper price. Cool.
Still, I don't consider these discs long term storage. The next step up
is archival grade discs, which are much more expensive. Archival grade
discs usually have a solid gold layer to minimize corrosion and maximize
reflectivity. Some have gold only. Some have gold and silver, which is
cheaper - less gold, more silver. They also usually have an extra hard
coating to prevent scratches. Conventional DVD's are insanely easy to
scratch.
Verbatim 95355 UltraLife 4.7 GB 8x Gold Archival Grade DVD-R, 50-Disc
Spindle - $ 93 / 50 = $ 1.86 / disc. cwc = $ 2.11 / disc
My research tells me that MAM-A / Mitsui is state of the art in archival
grade discs. They have lots of data on their website. Their gold /
silver DVD discs are projected to last 83 years. Their gold DVD discs
are projected to last 100 years. And their gold CD discs are projected
to last 300 years. Also, note the same risk as above in buying
unlabeled discs.
http://www.mam-a.com/
MAM-A Mitsui Gold Archive 8x DVD-R with no logo in bulk - 50 Count = $
117 / 50 = $ 2.34 / disc. cwc = $ 2.59 / disc (100 year discs)
MAM-A 8x SILVER PLUS GOLD DVD-R Branded Gold Surface 50pk Beehive with
Hard Coat - $ 93 / 50 = 1.86 / disc. cwc = $ 2.11 / disc (83 year
discs) (factory price)
MAM-A 84019 Silver plus Gold DVD-R 16x 4.7GB MAM-A Logo 50pk Spindle -
84019 - $ 80 / 50 = $ 1.60 / disc. cwc = $ 1.85 / disc (83 year
discs) (dealer price)
This last one is probably the best compromise between price and longevity.
Finally, if you want longevity for 1000 years, there's m-disc, as
mentioned previously. This is overkill for what I'm doing, but here are
the prices.
M-DISC DVD+R 4X Cake Box 50 Pack - $ 136 / 50 = $ 2.72 / disc. cwc = $
2.97 / disc. That's only $ 0.38 / disc more than the 100 year discs.
Not to bad if you need that kind of lifespan.
So, the best I can do for an archival grade disc is $ 1.85 / disc.
Now, what about the HDD option. Here's a 3 TB drive I found on Amazon.
Western Digital Caviar Green 3 TB SATA III 64 MB Cache Bare/OEM Desktop
Hard Drive - WD30EZRX - $ 140
Let's say I want two for redundancy, since this is NOT archival media.
I would definitely spinrite this thing a couple of times / year to
refresh the magnetic fields. That process would probably take 5 days.
2 HDD - $ 280
How about a nice dual drive case with fan.
Vantec Dual 3.5-Inch SATA to USB 3.0 and eSATA with JBOD/RAID 0/1
External Hard Drive Enclosure (NST-400MX-S3R) - $ 83
So the total cost is - $ 363
And we can store 3000 GB / 4.7 GB per hour = 638 hours of video
approximately equivalent to 638 DVD's.
So, cost per equivalent DVD works out to - $ 363 / 638 = $ 0.57 / "disc"
or hour. This is slightly more than the cost of cheap DVD's when cost
of cases is accounted for but substantially less than the cost of
archival DVD's.
So, here's how I see my choices.
- 83 year archival DVD's
* relatively low up front cost for 50 pack
* substantially larger long term cost
* some additional cost for storage rack, etc. (and where do I put it)
* discs are easy to access and play in dvd player
- two big hard drives
* relatively high up front cost
* substantially lower long term cost
* minimal space required
* movies are harder to access since I have to hook up computer to tv
and start player, etc.
There is the possibility of online storage. Storing 5 GB on Amazon S3
will cost me $ 0.75 / month for 1 hour of video. It will cost me about
$ 0.75 each time I retrieve it. Given the quantity of data, not very
practical.
None of these choices are really great for me. What I may do is to
select the best 50 - 100 hours of source video and put them onto 83 year
archival discs. Even that would cost me $ 185. I'm not sure it's worth
it. If I'm comparing the HDD to the 83 year archival discs, the hard
drive becomes more economical than the discs once I store 196 hours of
video. At this point, it seems there are no good (within my budget)
options to store 800 hours of video. One other option is to convert
everything to cheap DVD's, then refresh it and recopy it every 5 - 10
years. That doesn't sound too attractive either.
PS, from a hardware perspective, the Hauppauge USB-Live2 analog video
digitizer looks cool. Haven't tried it and don't know about it's
drivers or software. I can get mdisc / bluray drives in the $ 50 - 100
range.
Let me know what you think, and whether I've left out anything major.
This took two hours to type and I didn't proofread it, so sorry for any
errors.
Sincerely,
Ron
On 10/15/2012 4:45 PM, JD wrote:
>> I also wanted to share a great way to track and catalog DVD's that
>> you've purchased. It's an android app called MyMovies (free - limited
>> to 50 titles) and MyMovies Pro ($6 - unlimited). It's ultra cool. Just
>>
> Hopefully, someone else has found a fantastic Linux video editor and will be
> able to let us all know about it.
>
>
>
> Previously I used a F/LOSS web server + back-end DB to catalog DVDs and
> recordings. Over time, that became too much effort and I didn't like that the
> program forced me to a specific DB and the export was only to HTML files. Tried
> a Java desktop client too, but it was toooo damn slow. It didn't play nice on
> my network.
>
> Ended up using the time-proven 'ls -Rl' technique with numbered DVDs. With a
> large number of data DVDs holding TV recordings in xvid/avi or h.264/mkv, a
> quick 'egrep -i' searches by title for the specific numbered disk containing the
> recorded show/movie. As a CLI person, the most-used desktops all have a script
> that will ssh over to the main box with all the dvd-0xyz.txt files and perform
> the query. For more information, XBMC has the catalog with actors, directors,
> year, genre, title, etc .... grep is just so useful and text is so easy to
> search.
>
> I love that the grep clearly returns the numbered data DVD I need to load.
>
> As to converting VHS to MPEG or XVID or h.264, almost any $20 USB TV adapter can
> record SD content to MPEG2. Just pick from a MythTV compatible device to be
> happiest long term. If you want to record OTA HD content, I'd suggest more
> effort for the selected hardware. I love the HD-Homerun network tuners. Those
> work well with OTA except 1 channel that doesn't come in at my house. It is
> hardly a major issue, since 69 other channels do come in OTA from Marietta.
>
> For finding commercials and editing them out, I searched for Linux-based
> solutions for years and finally gave up. Comskip does run nice under WINE, I've
> never found a Linux-based editor that will accept the comskip created cut files
> (there's like 6 different formats that comskip will create) as the starting
> point for where to cut commercials. Comskip is about 90% accurate. Comskip can
> output cut marker files that XBMC and mplayer will honor without altering the
> source video file. That can be handy when you just want to skip commercials, but
> not waste time actually modifying the file contents.
>
> Instead of fighting it, I bought - yes, purchased a Windows tool called
> VideoRedo TV Suite. It appears to use comskip code internally to create cut
> markers. There's a point where $50 is worth it to me. That $50 program has
> been heavily automated here to generate the cut marker files overnight. It takes
> about 45 seconds of human effort to confirm the cut locations and request the
> modified file be saved. If you are willing to live with 90% accurate cuts, it
> could all be automated. Networks have changed up the markers commonly searched
> by VRDP and comskip to trick them into keeping commercials and removing some of
> the desired content. Older stations with less money like RTV are almost 100%
> accurate with cut locations.
>
> At this point, my systems are almost automatic with just a few manual
> "confirmation" places. WTV --> MPEG2 --> h.264/mkv with CC1 and CC3 closed
> captions included. If there are multiple audio channels in the stream, that can
> require special processing, but that usually happens with a few PBS programs
> with audio tracks for the deaf or background-only audio.
>
> I've blogged about almost all of these solutions over the years. The most recent
> articles are about building your own DB4 antenna for $20 or so that actually
> beat some Antennas Direct models.
>
>
> And just to be clear, I also wimped out and use Windows7 to record TV. It is
> just so easy to setup with free schedule data. That OS runs inside a virtual
> machine under KVM and has been doing it nicely for over a year. THAT was not an
> easy thing to accomplish to get satisfactory results. Microsoft did (does?) some
> things trying to protect Joe-end-user from doing foolish things, like blocking
> recording directly to NAS storage.
>
> I suspect there are hundreds of years of knowledge lurking related to this problem.
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://mail.ale.org/mailman/listinfo/ale
> See JOBS, ANNOUNCE and SCHOOLS lists at
> http://mail.ale.org/mailman/listinfo
>
>
--
(To whom it may concern. My email address has changed. Replying to former
messages prior to 03/31/12 with my personal address will go to the wrong
address. Please send all personal correspondence to the new address.)
(PS - If you email me and don't get a quick response, you might want to
call on the phone. I get about 300 emails per day from alternate energy
mailing lists and such. I don't always see new email messages very quickly.)
Ron Frazier
770-205-9422 (O) Leave a message.
linuxdude AT techstarship.com
More information about the Ale
mailing list