A primitive HFR test, and cubes. Of course.

Posted 30 December, 2012 Anthony Tan (staring at yet another render. But still loving it for some reason..)

« previous | next »

tags: tech

It's shiny, it's new (in a buzzword context, not in a 'lets shoot at high frame rates' context). It's also something that's not that hard to play with at home once you get curious... and well, HFR is kinda neat. I've a soft spot for higher data rates.

Having just seen The Hobbit this weekend (note: I'm not lazy, it only got released in Australia on the 26th..), and finally seeing what the deal was with this HFR thing, I almost got into that trap of speculating and discussing the relative pros and cons based on the movie... and then I caught hold of myself - I'm an engineer dammit. And I have a computer. Why the heck don't I just create a fairly artificial, but potentially instructive test?

It's not like I can't render something out at varying sample rates, and even better, I can avoid being constrained by little things like say, cameras, and working out how to reproduce the real world.

For the purposes of this example, I'm calling HFR as 48fps with a baseline of 24fps. It'll do for this test, but note that I'm being very loose with my terminology. Also not that this is highly artificial, and that in the Real World (esp if I had to match a 48fps plate..) there be a lot more to worry about and little artifacts to manage.. but hey, even stuff like this is instructive.

Anyhoo, after all of that, the TL/DR: I can see why HFR works.. but I can't quite tell why I felt like it didn't work so well in the cinema.. (that's another question though). As this is all just experimentation, there's not really any conclusions or massive takeout, just that if you're at all curious, you might as well just play with it yourself. It's fun!

Setting up an experiment

Since my major curiosity is on how HFR impacts on motion, it's time to invoke the rule of the falling cube. 27 of them in fact. Some basic rotations are applied, a bit of a heave up, radial out, and then letting gravity and the magic of RBD to create motion. There's a mix of slow and fast here and linear and rotational. Of course it's not a perfect, hypertechincal HFR test but as I've learnt - quick is good.

To approximate reality (which runs somewhat higher than 48fps..) the dynamics were done in the scene at the higher frame rate, then keys baked out and framerate dropped back to 24fps with Maya being allowed to interpolate. That should have given me the same motion regardless of how many frames I'm using (a curve is a curve is a curve)

From that, I've rendered out two completely independent sequences and the associated motion vector pass. Materials are basic MIAs so there's a bit of shinyness (incase that was going to show off anything else interesting) and there's some cheap AO/FG effects in there just for prettification.

Handball to Nuke, and the only effects I'm applying will be the vectorblur node, and packaging up into H.264 encoded quicktimes. Eaaasy.

The baseline

To set the scene, here's a literal side by side with HFR-A* (i.e. dropping every other HFR frame). I've hue-shifted one of them^ so you can clearly see the dividing line and see how closely the two of them match up. You can see a little bit of the difference in position between the two when the cube catches the light (see frame 47) but overall the nature of the motion is consistent. Doing the same with HFR-B is a bit more distracting since it's further out of sync (it's sampled half a frame later) but the overall characteristic of the motion remains unchanged.

Some raw quicktimes for you to play with

Since this is primarily an anim test, I've prepped out four files, two with motion blur, and two without (there's a bonus 'super-extra-blurred' file too). In both renders I'm using the same normalised motion vector pass with a max pixel displacement of 256, so theoretically I should be able to use the same vectorblur settings. Inspecting the movec pass pretty much bears this out - for the same region on Baseline the normalised X/Y displacement is say, 0.02284/-0.0264 (about 30 pixels), while for HFR-A and HFR-B, it's roughly half that. Makes sense too since for each HFR frame, the relevant pixel has 1/48th of a second to move instead of 1/24th So, without prejudicing your thoughts, here's some raw data:

You may want to use a flipbook to make sure you're getting the appropriate framerate btw.

Observations & Thoughts

I spent 20 mins eyeballing the normal 24FPS files (with and without moblur) and then going to HFR was a bit of a shock, it _does_ look a bit more errm, well, sharp for lack of a better term. There's a noticeable additional level of data that I'm processing and assimilating that I adapt to pretty quickly.

Going *back* to 24FPS after getting used to HFR is a shock too, it immediately feels jittery, as if I can see the substeps in the scene. Motion blur doesn't help any here, obviously. Again though, adaptation doesn't take long before this looks 'fine' again, but I can still now sort of see frame steps, and yes, well. I can understand why once you've acclimatized, it might be hard to go back.

I rather like the feel of HFR - but that's only in this test. My largest scale sample (i.e. The Hobbit) wasn't nearly as positive so I'm a bit wary here. I still can't shake the feeling it's not an artifact of the increased frame rate, but some kind of glitch. I dunno, it felt too much like a sync issue when it felt wrong (very opening scene with the old Bilbo for example).

Artificially doubling the motion blur actually works decently, for some odd reason. Must keep that in mind.

As I'm writing this, I've got both the HFR and moblurred baseline clips looping out of the corner of my eye, and it's actually really noticeable which is which, more than if I'm staring at them directly.

It's only in high movement zones that I'm getting a lot of benefit from HFR, but it's taking a straight x2 on cycles to generate the frames. More if you count the extra processing/storage/etc. Kind of a 'handle with care' situation - do you really want to double your storage/management/render/etc requirements? Didn't think so...

Upsampling is the devil's work. No, really. I had a stab at upconverting the 24fps baseline to 48, but quickly realised that it was either a simple task (motion blur and hold frames for double the time where there's low motion) OR you're going to get absolutely craptastic results for the effort you're putting in. This may not hold as techniques get developed (for example: Luke Letellier's work) but it's still kinda clear that trying to create data where none exists is just going to hurt. You're better off taking the render hit, getting the data and throwing it away when it's not helping you (see the initial HFR-A splitscreened with baseline) IMHO.

Experimentation rocks. Why the hell would I spend hours on a forum debating this when I could just go out and render out 500 frames to actually see for myself? :)

Poking Maya's transform nodes and the worldMatrix
The difference between .getValue and .toScript for Nuke knobs
tags: tech

* In Nuke I'm using the baseline 24FPS frame as a reference point and referring to HFR-A and HFR-B as the two frames that cover the same approximate timeslice. The relationship between them is: HFR-A = (base frame * 2), and HFR-B = (base frame * 2) -1. Or more simply, base frame 1 refers to HFR frames 1, and 2. Base frame 2 refers to HFR frames 3, and 4. Etc.

^ HFR-A is hueshifted