Data Wrangling

Hey Jax,
We're shooting with two Nikon D-800s, with Ninja 2s, directly into 4 separate 125G SSD cards. We're getting about 400g footage for the two cameras per day.

I'm asking for the setup. I decided last night to hookup two 2TB portable drives to my laptop to help with the download on set. Two SSD Cards will start downloading after lunch and the other two SSD cards will record while the first two are downloading.

Right now, my DP goes home and dumps the data on a 2TB hard drive he purchased. There is no backup.

Let me know if you see any holes in what I just decided.
Thanks,
Aveek
 
400g footage for the two cameras per day.........There is no backup

Back it up and fast with 400GB a day I would hate to see someone lose that.

Right now, my DP goes home and dumps the data on a 2TB hard drive he purchased

I would set up RAID 1 on the 2TB with another 2TB just to have a backup in case somthing goes wrong with the drive.

again would hate to see someone lose all that work

Best of luck
 
My minimum data wrangling setup would be something along the lines of this:

-UPS Powerboard (powerboard with a battery) - for those times at the end of the day when the Gaffer rips your power out and cuts power to your HDDs 5 minutes before they'd finished transferring.
-Shotput Pro (or maybe Red Giant Bulletproof) - Checksum management and automated copy.
-Dedicated laptop my choice would be a MBP and a Thunderbolt breakout box
-Thunderbolt (or fastest interface you have) RAID enclosure - Something with hot-swappable drives like the G-Tech G-Dock. I think LaCie has something similar, and there are other manufacturers who make cages that can take any standard SSD or spinning HDD drive into the enclosure. Set up to RAID 1 (i.e. exact copy). Drives must be of a size that will allow you to dump at the very last one day's worth of data. Personally, I prefer drives that will be able to hold the whole productions worth of data (assuming short-form production - say short film, music video or TVC).
-2x external HDDs with enough space
-Card reader(s) for the raw footage (whether that's an SSD reader, SD reader, or both in this case)
-I would also suggest having REDCineX Pro for RED footage and DaVinci Resolve Lite (both are free) - you never know when the Director or DP is going to ask you to call up a clip from earlier in the day and wants to view it with a LUT, or if you're going to be wanting to view .dngs etc.

Insert card into reader, ingest using Shotput onto the RAID enclosure, then onto the other 2x external drives. RAID enclosure goes home with the Data Wrangler, 1x external hard drive goes home with the DP and 1x external hard drive goes home with the Producer (or however you want to split it up).

Now, I've worked on a number of lower budget productions where the RAID enclosure is skipped, opting instead for three external drives. Also, it's relatively common for the Editor to have a similar drive, and to swap with the Producer's drive at regular intervals - whether that's daily or weekly, depending on the project. I.e. the Producer drops the drive to the editor, who then gives the Producer their drive to continue dumping data to.

You should expect Production to provide external drives. The Data Wrangler usually provides their own laptop (production usually has a laptop but it is generally not suited to the kind of work), RAID enclosure, or at least one external drive, basic card reader (SD, CF, etc. though many Wranglers I've worked with have also had SxS readers).

Now the low, low budget way is to have only a laptop and three drives, and then a copy is made from the card to each drive one by one. The data is 'checked' by scrubbing through some clips and checking to make sure that the no. of files count and size is identical between the two. This is not a method I would recommend at all, but it does happen (more often than it should).
 
Last edited:
Back
Top