The Rain Was the Easy Part

The Rain Was the Easy Part
Today's music video is my most ambitious one to date...

12:41 pm

Every render on my computer had frozen.

AI helpers had spent the morning building a music video in parallel, and almost every scene was written. None of them had been rendered end to end, and now nothing would render at all. The graphics card showed 0% busy and free memory was down to about 1 GB. Nothing had crashed. Everything was just waiting, and it would have waited forever.

The deadline was 6:30 that evening, so I could put the video on YouTube the same day.

How it started

Today's music challenge theme was Rainy night in a neon city. My answer is a song called Neon Rain, built around Prince's Purple Rain.

Two strangers miss the last train out of Shinjuku and spend one rainy night together in the neon city. She tilts her clear umbrella so it covers him too. He buys two hot cans of coffee from a vending machine. A billboard spells a name in letters he can't read. By the last chorus the rain itself has turned into light, and the whole city glows violet, magenta and cyan.

The look came from two examples on Pinterest: a purple skyline with a needle-shaped tower, a moon and a galaxy over a bridge, and a couple under a clear umbrella by a vending machine in a rainy neon alley. Violet nights, wet streets, crisp anime line work. I wanted four minutes and seventeen seconds of that.

At 7:28 in the morning the song, the lyrics and those two pictures went into a new folder. I asked Claude, the AI assistant I work with, for a storyboard before anything else. When I asked how long the whole video would take, the first answer was "tomorrow". A quicker, simpler version was on offer for 2 pm. I turned it down: the full version, finished by 6:30.

Catching a style in JSON

Image models drift. Ask fifty times for "anime, neon, rainy night" and you get fifty slightly different films. So instead of a paragraph, every image prompt was made of two JSON files (JSON is a plain, labelled data format): one for the style, shared by every picture, and one for the shot.

The style file gives each decision its own line:

  • Medium: crisp modern cel animation, never a photo, never a 3D render.
  • Line work: thin indigo-black lines, a little thicker on the silhouette.
  • Shading: one base tone and one hard-edged shadow tone, and nothing muddy.
  • Palette: navy and violet, with magenta, cyan and amber neon as the only strong accents. Shadows are violet, never grey.
  • Light: every glow comes from a sign, a lamp or a vending machine.

One line is pure strategy. Every sign must be blank, with no letters at all, because every readable word in the film would be added later in code. Image models scramble lettering; code gets it right every time.

The shot file holds the rest: the view, the pose, the expression, the light on the face. The choice that saved the most time was painting the characters alone on a perfectly flat mid-grey background and painting the streets empty. A figure on flat grey cuts out cleanly, and nobody ever has to be painted out of an empty street.

All 49 images came from Nano Banana 2, run on the WaveSpeed service, for $4.63 in total. One lesson cost a few redos. When you give the model a reference picture, it copies the framing as well as the look, so for unusual camera angles we used text only.

Why paintings at real depth

I'd first asked for Remotion, a tool that builds videos out of web code. Then I remembered the method behind my futuristic 2300 concert video and asked whether it would suit anime better. It did. The method is the free Render Kit from AI2Play's "Music Video in Code" tutorial, with three.js, a library that draws 3D scenes in a web browser.

The reason was depth. In three.js each painting becomes a flat card standing at a real distance from the camera: the far skyline, the street, the couple in front. When the camera drifts, near things slide faster than far ones, the way they do through a real lens. That effect is called parallax. The rain isn't a sheet pasted over the top, either. It's code raindrops falling between the cards, in front of her face and behind his back, tinted by the painting behind them. The neon, the reflections on the wet street, the camera moves and the lyrics are all code. A copy of Chrome with no window draws each frame, and FFmpeg, the standard video tool, stitches the frames into a film.

Timing came before scenery. The song runs at 75 beats per minute, so one bar lasts 3.2 seconds, and all 62 shots start and end on a bar line. The shot list and the camera path are data files generated by scripts, never edited by hand, so changing one shot rebuilds the whole timeline. By 9:42 we had an animatic, a rough cut using the real paintings, timed to the song, with the camera moving.

I asked for a sample of the lyrics first, because I was afraid words would ruin the picture. Claude made three versions: the busy style from my country video, a light one, and none at all. I picked the light one: a quiet italic serif with one bold capital word per line, each word appearing on the syllable where it's sung.

Six builders, one contract

Then the build split six ways. Each AI helper (Claude calls them agents) owned a few scenes and nothing else, and they all followed one short contract. A scene is a function that takes the time and returns the picture for that moment. If a scene isn't ready, a plain stand-in takes its place. Because every frame depends only on the clock, any frame can be drawn alone, in any order. So the film could play from start to finish at any point in the morning, even half built.

By 12:45 all nineteen scenes existed. By then the computer was already in trouble.

The crash

At about noon, several helpers finished within minutes of each other, and each one started a test render. The gate that was meant to make them take turns let them all through at once. Each render opened its own copy of Chrome, and each copy loaded the paintings.

Two other things were eating memory without anyone noticing. ComfyUI, the image tool that had filled in the holes behind the cut-out figures that morning, was sitting idle with 29 GB of memory and 17 GB of graphics memory still held. Worse, the part of Windows that draws everything on your screen (dwm.exe), together with the graphics driver, had quietly taken about 47 GB of shared graphics memory and never let it go.

That's thrashing: the computer spends all its effort looking for room and none of it doing work. Renders hung with the graphics card idle. One clip reported "finished" and turned out to be cut short, with no error at all. That's the worst kind of failure, because nothing warns you. Claude freed ComfyUI's memory, built a lock so only one render could run at a time, and stopped the stuck jobs. At 12:41 all rendering stopped. Shortly after one o'clock the computer was so short of memory that it couldn't start a new program, and it stopped responding. I physically turned it off.

Starting again

The restart gave back 47 GB. This time the rules came first:

  • One render at a time, through the lock, and the lock checks free memory before letting anything start.
  • ComfyUI stays closed.
  • The film renders in chunks, cut at shot changes. Every chunk is decoded and its frames counted, so a clip that "finishes" short gets caught and redone.
  • If the browser loses its graphics connection mid-render, that chunk renders again.

On a healthy machine it was fast. The 720p draft took about five minutes and was ready at 1:30. It went up on the review page, and I typed: No notes, go.

The automatic checks still found work. In six shots the characters' feet slid across the ground as the camera moved. Each painted figure stood nearer the camera than the street under it, so parallax moved the two at different speeds. The fix: find the depth of the ground right under the feet at the moment the camera lands, stand the figure exactly there, and scale it so the framing doesn't change. The billboard's name, ハナ (Hana), finished spelling itself two-thirds of a second after the word "name" was sung, so it was moved earlier. A switched-off sign was showing through the final fade to black, which a change in drawing order fixed.

The 1440p master took about fifteen minutes. At 2:24 pm the final checks passed:

  • all 6,173 frames, with the sound in sync;
  • no flashing faster than twice a second, under the three-per-second safety limit for viewers with photosensitive epilepsy;
  • all 61 cuts exactly where the shot plan puts them;
  • no colour cast.

That was four hours before the deadline. From the Pinterest examples to the last check, the whole video took about eight or nine hours.

The takeaway

The rain was the easy part. Code can draw rain all night, every drop tinted by the sign behind it, and it never gets tired. What it can't do is notice that six helpers are about to reach for the same graphics card in the same second. Adding helpers is the easy decision. The queue they wait in has to exist before the first one starts, not after the screen goes dark.