Download Pack
This pack contains 3,345 VJ loops (87 GB)
https://www.patreon.com/posts/125675861
Behind the Scenes
We live in the age of everything being tracked. Corporate chasing that money. Gov watching the masses. But I think computer vision is a strange thing to directly experience since it's typically hidden away. It's particularly interesting to watch the tracking errors and remember that the world is becoming increasingly automated by AI models with little human oversight. I've long wanted to explore this theme but couldn't nail down the tools that would allow me to work at scale and pull off what I had in mind. But recently the Vision plugin for After Effects was released and it was perfect for what I had in mind. And plus I saw that the Blace plugin wasn't going to be supported for future AE versions. And also some interesting ComfyUI extensions too. So it was go time!
I've always steered clear of using any pre-made stock footage in my VJ packs, but in this specific instance it was actually the perfect way forward. I was aiming for two central themes: tracking data of crowds/cars/airplanes and then also the banal corporate obsession with tracking data but ironically mapped back onto itself. So I start by gathering a collection of websites that offer public domain footage and then carefully read through each of their licenses. I ended up sticking with using footage only from Pixabay and Pexels since their license terms were very clear that I could commercially distribute the footage so long as I creatively modified it. I curated 121 corporate clips, 187 crowd clips, and 119 travel clips... Which comes out to 427 clips in total. Plenty!
There's nothing more I loathe than dealing with drop frames when importing footage with differing frame rates into a single comp set to 30fps. Doing this leads to regular stuttering in the footage and it drives me absolutely crazy. But then I realized that some of the clips were slomo or timelapse and so the sense of time was already distorted. So I decided to adjust the frame rate for all the clips to 30fps. This is normally blasphemous but in this particular context I think it's a fine solution. Although I didn't anticipate that a few of the clips had doubled up duplicate frames, which I cannot fathom why someone would do this, and so they don't slow down nicely. But I didn't realize this until much too late in the process, ah well.
I had to jump through all sorts of technical hoops to pull off the next steps. Since I had 2 hours 42 minutes of uncut footage, first I had to do some minor clean up to all of the footage by trimming bad cuts, editing clips which were too long, and doing some basic color correction. Then I started doing tests with the Vision and Blace FX. But I realized that the Vision FX was likely relying on the Ultralytics YOLO model for the object tracking and so I experimented with using ComfyUI, but I had some specific compositing experiments in mind and so I had to rely on the Vision AE plugin instead. I wanted to use the Blace FX to block out the eyes of each person and then use the Vision FX to overlay tracking boxes/labels, although I realized that both of these FX are very heavy to render out from After Effects and yet I wanted to create some glitch variations. I gave it some thought and realized that the render times would be outrageously heavy with all of the comp variations and it wouldn't be done in time. So I had no choice but to pre-render so that I could bake the Vision and Blace FX into the footage. To simplify things I assembled all of the clips into 3 giant comps: corporate, crowds, and travel. I also made some comp variations of the Vision FX that I had in mind. Then I pre-rendered these comps out to a JPEG sequence or PNG sequence (if alpha was needed). I rendered out these 8 scenes and the average render time for each of these giant comps was between 22-30 hours. Blarg I hate waiting for renders to finish! Overall I did object tracking on 476,496 frames... My VFX eyes are too big for my computing stomach.
While waiting for those AE renders to finish, I got curious of what sort of other object tracking visualizations that I could do. Then I remembered back to the ComfyUI-MotionDiff extension that I had experimented with last year but didn't have a use for at the time. I was most interested in the 4D-Humans implementation for the 3D pose estimation but it seems like the normal map and depth map generation had broke since I last used it. I tried all sorts of fixes but just couldn't get it working, which was a bummer. But the OpenPose output was working correctly and it was mesmerizing to watch. So from there I had to do some tinkering to allow for a huge frame sequence to be ingested (158,832 frames). The default ComfyUI "Load Image Directory" node loads all of the images into memory at once, which definitely wasn't going to work here. So after some research I found the "Load Image List From Dir" in the ComfyUI-Inspire-Pack extension which did the trick if I set the "image load cap" to 1 and then link the "start index" to a widget. In this widget I could set a start frame for the value and set the "control after generate" to increment. Then if I set the global mode to "queue (on change)" then it would only load and render 1 image at a time. In this way it would run through the entire frame sequence of arbitrary length. Lastly to get the correct frame number attached to the output frames, I used the "Frame Counter" and "Image Sequence Saver" nodes from the ComfyUI-Dream-Project extension and linked the prior said widget. That was annoying to figure out but at least I can reuse this setup in the future. There is almost too much jittering in the OpenPose footage since the AI model doesn't have a temporal awareness, but felt right for this theme.
And of course I had a last minute idea that resulted in a heavy render. I was still curious of what other processing I could do on these videos and I've long been interested in the depth estimation AI models that have been coming out. First I experimented with the Depth Scanner and Quick Depth 3 AE plugins, but then realized that they were most likely relying on the DepthAnything model since it's are open-source and SOTA. Since I had just processed a ton of frames in ComfyUI, I did some research and tested out the ComfyUI-DepthAnythingV2 extension. It worked right out of the box and so I used the same batch processing technique as discussed prior to render out 158,832 frames. I also figured out how to have 2 different instances of ComfyUI running at the same time so that I could render using both of my GPUs. This aspect is not well documented and I finally figured it out by copying the whole ComfyUI directory and then edited the run_nvidia_gpu.bat files within each folder with either: --cuda-device 0 --port 8188 OR --cuda-device 1 --port 8189. There is some flickering in the depth map footage since the DepthAnything V2 model doesn't have a temporal understanding, but in this case I think it nicely added the feeling of glitchy machine vision. From there I took it into AE and applied the BFX Map Ramp FX so that I could have a custom color gradient visualize the depth map into something sexier. I would have loved to apply the depth map back onto the original videos and then wildly exaggerate the z depth, but it didn't really match the theme and I also ran out of time. I chose to render out both the original depth map and the tweaked color version so that VJs could do some really interesting stuff in real-time with the depth map. Make my dreams come true!
When all of those renders were finally finished then I imported these frame sequences back into AE. For VJs that don't want to jam with the 427 individual scenes, I created a pre-edited video that cuts every 1 second. After putting that together I realized that it'd be useful to also have version which cut every 2 seconds. Using the Rift and Sortie AE scripts sped up this process. Finally something easy.
From here I made variations using JPEG Glitch and Pixelsorter FX. It was much easier to add FX onto the 3 giant comps instead of 427 solo comps. And so that meant that I needed to render out each of the 427 solo scenes from the giant comp. I wondered if there was a way to use markers to automatically cut up the giant comp into individual scene comps. So after some research and experiments, I used the pt_LayerMarkers AE script to add markers at the scene cuts of the original scene. Then I used the MarkerMover AE script to copy the markers over to each of the giant comps. Then I again used the pt_LayerMarkers AE script but this time split the layers at markers within the giant comp. Although it seems there is a floating point precision issue since the in/out out points of the layers were not quantized to the timeline and so all of the edit were off by 1 frame. So I had to select all layers, extend all in/out points just one frame to the left, then pull back all in/out points one frame to the right, and that fixed it. From here I had planned on using the rd_RenderLayers AE script to automatically make each layer into a new render job. But it turns out that this script has been broken since AE 2020. So I got curious and wanted to see if ChatGPT-o3-mini Reasoning model could fix it. It took me a few different attempts at polishing my text prompt but after describing what I thought the issue was then it was able to fix the script and it worked perfectly! (I posted the fixed script into the comments of the rd_RenderLayers webpage.) But because I was queueing up a thousands of jobs and the After Effects render queue stops responding after about 700 jobs are added into the queue, I decided to export the batch render jobs out to the Thinkbox Deadline render manager app. This allowed me split up the AE project to make it more manageable so that I could queue up the render jobs via the rd_RenderLayers AE script and then use the Deadline Submission AE script to get everything into the Deadline app. And plus that meant any failed jobs would auto restart, which is important with thousands of render jobs in the queue. What a slog. "Never trust a computer you can’t throw out a window.”
This has been the most tedious project I've worked on in a long time but I love the end result. I wanted to make even more variations but this VJ pack was already humongous. But I'd suggest throwing a threshold FX on the DataOverlaySolo scenes for some very gritty vibes. Or maybe some VHS noise FX to make it feel like security cam footage. Welcome to the future, it's full of glitches.
Discussion (0)