Video Troubleshooting with Keyframes

I made a 65 minute screencast with Screenflow and knew I had a problem. I normally record in a second space (with the spaces feature of macOS) that doesn't have anything private in it. If I ever go back to my regular space then private information gets recorded, like my Messages window, and I have to edit it out.

I paused and checked my messages. Then I accidentally unpaused with my messages showing. I paused it again immediately and hit my marker hotkey to tell Screenflow to mark that timestamp.

After recording I saw Screenflow did not mark the timestamp. After some testing, I discovered Screenflow doesn't create markers while paused. That's dumb.

So I had a half second problem in a 65 minute video. I didn't have much memory of where in the video the problem was. I tried to find the spot in Screenflow by dragging the scrubber around quickly. Then I exported the video and tried watching it at high speed with VLC and quicktime. VLC didn't work well, at 16x or higher playback it often gets stuck on one frame and stops showing anything. Quicktime can play at 60x without messing up, and I tried it, but that was too fast for me to see the problem. I tried 30x too and checked a couple spots where I saw something flash, but they were just other relatively large changes on the screen.

I was unwilling to sit there and watch the whole video at more like 4x – and pay close attention to spot a fraction-of-a-second problem (it'd only show up for maybe an eighth of a second at 4x).

So I googled for how to find keyframes. I figured there'd be a lot of motion when I instantly switch from the screencast to my messages window, and a keyframe would get created.

First I found this:

http://superuser.com/questions/885452/extracting-the-index-of-key-frames-from-a-video-using-ffmpeg

time ffprobe -select_streams v -show_frames -show_entries frame=pict_type -of csv myvideo.mp4 | grep -n I | cut -d ':' -f 1 > frame_indices.txt

It takes 10 minutes to run and a bunch of CPU but it worked. While it ran I found this:

https://gpac.wp.mines-telecom.fr/2015/11/02/check-key-frame-alignment-with-mp4box/

time MP4Box -std -diso myvideo.mp4 2>&1 | grep SyncSampleEntry > keyframes.txt

After a quick "brew install mp4box", that ran in a quarter second.

I saw keyframes like this:

1 26 51 76 101 126 151 176 201 226 251 276 301 326 351

I skimmed around and it kept going like that. A keyframe every 25 frames. Uh oh. That's not very useful. It did give me the idea that I could get my video sizes a lot smaller if I could generate fewer keyframes. When I'm writing I don't need many keyframes.

The end of the list looked liked this:

117293 117318 117343 117368 117393 117418 117443 117468 117493 117518

That's still every 25 frames. Same boring pattern.

But wait. The pattern shifted. It starts at 18 instead of 1. Something must have changed somewhere.

I manually performed binary search to find the change point. How? Go to the middle. If I see a frame ending with 18, I'm still too near the end, go up a bunch. If I see a frame ending with 1, I'm too near the start, go down a bunch. After I narrowed it down a few times that way, then I scrolled quickly in the right direction until I saw the pattern change.

36201 36226 36251 36276 36301 36319 36343 36368 36393 36418 36443 36468 36493

At that point, scrolling down, the last digits weren't just 1 and 6 every time, there's different digits.

36319 frames, at 30 frames per second, is 1211 seconds. That's 20 minutes and 11 seconds.

I checked the video and that was my problem. Right at 20:11 I showed my chat messages for half a second. It worked perfectly. I edited it out, problem solved!