The official music video behind any song, locked to the song

How it works

A music video is rarely the album track with pictures added: it has an intro and often edits — a bar repeated, a bar cut, a longer outro. A single constant offset is right for part of the song and wrong for the rest, so OpenCanvas measures a map: one offset per stretch of the song, and nothing for stretches the video has no scene for.

NEW IN THE KOTLIN CORE SOURCE

Keep the people in the picture.

YuNet faces, YOLOX-Nano people and a shot-aware camera replace the fixed center crop. Follow a subject, keep a group together, and show the whole scene when a tight crop would lose the action.

Benchmarks

Historical sync/stream baseline, measured before publisher-selection and subject-framing updates on one Windows 11 machine from an empty cache. These are not speed claims for the new path. Current framing results and limits appear above; repeat the matrix and stream tests with OPENCANVAS_MANUAL=1.

Cold start — 16 songs, 6 languages, 16 of 16 aligned

Median 550 ms to a playable stream (451–688 ms) and 954 ms to the map (596–1 688 ms, p90 1 139 ms). "Offset" is where the song starts on the video's timeline; many videos carry the same master (offset 0).

Compared with the other ways

We did not benchmark other libraries. This compares approaches and says what each one costs; the OpenCanvas row and the yt-dlp numbers are measured above.

Quickstart

Not tied to any app: you need the song's title and artist, and the id of the audio you are playing (or a place its audio can be read).

repositories {
    mavenCentral()
    maven { url = uri("https://jitpack.io") }
}
dependencies {
    implementation("com.github.Vijay-Janarthanan:OpenCanvas:1.0.0")
    implementation("com.github.Vijay-Janarthanan.OpenCanvas:opencanvas-compose:1.0.0") // optional

OpenCanvas.configure(
    OpenCanvasConfig(
        cacheDirectory = File(context.filesDir, "opencanvas"), // measured maps live here
        log = { Log.d("OpenCanvas", it) },                     // one line per stage, with timings
    ),
)

// when the song starts (returns at once; the work runs on the library's own scope)
OpenCanvas.prefetch(
    title = "Blinding Lights", artist = "The Weeknd", durationSec = 200,
    mode = OpenCanvasMode.FULL_SYNCED_VIDEO,
    resolution = OpenCanvasResolution.HD_720P,
    trackVideoId = "fHI8X4OXluQ",        // YouTube (Music) id of the audio you are playing
)

// in the screen that shows the video
OpenCanvas.resolveFlow(
    title = "Blinding Lights", artist = "The Weeknd", durationSec = 200,
    mode = OpenCanvasMode.FULL_SYNCED_VIDEO,
    resolution = OpenCanvasResolution.HD_720P,
    trackVideoId = "fHI8X4OXluQ",
).collect { track ->
    when (track.syncStage) {
        SyncStage.PENDING  -> player.prepare(track.videoStreamUrl, track.videoHeaders) // buffer, don't show yet
        SyncStage.MEASURED -> { policy = CanvasSyncPolicy(track.syncMap(), track.videoDurationMs); showVideo() }
        SyncStage.NONE     -> showStillArtwork()   // no video lines up with this song
    }
}

// every 250 ms, and at once when policy.seekDetected(...) is true or the track changes
when (val action = policy.decide(songMs, video.positionMs, songPlaying, video.isReady)) {
    Hidden    -> showStillArtwork()            // no matching picture in this stretch of the song
    Hold      -> Unit
    Ended     -> showStillArtwork()            // the song outlasts the video: never loop
    is SeekTo -> video.seekTo(action.videoMs)
    is Nudge  -> video.setSpeed(action.speed)  // at most ±5 %, imperceptible
}

OpenCanvas.resolveFlow(
    title, artist,
    songAudio = SongAudio.Stream(File("/music/song.m4a")),
    mode = OpenCanvasMode.FULL_SYNCED_VIDEO,
)

OpenCanvas.configure(OpenCanvasConfig(
    streamBackend = MyBackend(),         // implements StreamBackend
    // desktop only, optional: YtDlpBackend()
))

Try it

The sync engine is compiled to JavaScript and runs in this page. Pick a real song: its official video from YouTube plays behind it as a vertical canvas, held on the song's second by the same code the apps use. Drag the slider and the picture follows.

Demo not loaded

▶ Load the demo Loads YouTube's embedded players. Nothing else leaves the page.

Blinding Lights

The Weeknd

▶

0:00

The song, as YouTube's audio upload. This is the sound you hear.

Real songs Measure in this page Your own files

Or paste a YouTube link

Play

What the engine foundNot loaded

song

—

video

—

should be at

—

drift

—

decision

—

hard seeks

—

What runs here, and what does not

• Running in your browser: the library's Kotlin sync code (opencanvas-engine.mjs, 99 KB compressed): MP4 header parsing, envelopes, alignment and the policy that decides seek, hold or still cover. On the second and third tab it measures a new pair live.
• Real songs: the maps come from the library running on a device against YouTube's own files, because a page is not allowed to read YouTube's media bytes. The page then drives the policy with them.
• Different from the apps: YouTube's embedded players cannot change speed in fine steps, so the real-songs tab corrects drift by seeking (past 0.3 s). The apps nudge the speed by up to 5 %.
• Measuring any YouTube link in a page needs a small relay that adds the headers browsers require; the engine itself already runs here.

Install the demo builds

Demo builds of BitChord with OpenCanvas, so you can see it on your own songs without building anything. Test builds, not official BitChord releases.

Download for Windows (zip, 308 MB) Download for Android (APK, 86 MB) All files and checksums

Play a song without label artwork, open the full player and seek: the video follows. Checksums and notes are on the release page. BitChord is GPLv3; the source of the builds is the pull request branch.

Use it from your language

The engine is Kotlin, so Android and the JVM get all of it. Everything that is pure maths (reading the MP4 headers, aligning, deciding where the picture belongs) also runs in a browser, and the part every player needs, the map and the policy, is ported to TypeScript and Dart. All of them are tested against the same vectors.

import { measure, WebCanvasPolicy } from './opencanvas-engine.mjs';

// anything that can read a byte range: a File, or fetch() with a Range header
const reader = (file) => ({
  length: file.size,
  read: (from, count) => file.slice(from, from + count).arrayBuffer(),
});

const map = JSON.parse(await measure(reader(songFile), reader(videoFile)));
// { ok, confidence, segments: [{ songStartMs, songEndMs, offsetMs, rate }], ... }

const policy = new WebCanvasPolicy(JSON.stringify(map.segments), map.videoSeconds * 1000);
setInterval(() => {
  const a = JSON.parse(policy.decide(audio.currentTime * 1000, video.currentTime * 1000,
                                     !audio.paused, video.readyState >= 2));
  if (a.kind === 'seek') video.currentTime = a.videoMs / 1000;
  if (a.kind === 'nudge') video.playbackRate = a.speed;
}, 250);

// TypeScript (@opencanvas/core)
const map = SyncMap.fromJson(storedJson);          // the JSON the Kotlin core stores
const policy = new CanvasSyncPolicy(map, videoMs);
const action = policy.decide(songMs, videoMs, playing, ready);
// { kind: 'seek', videoMs } | { kind: 'nudge', speed } | 'hold' | 'hidden' | 'ended'

// Dart (open_canvas)
final policy = CanvasSyncPolicy(SyncMap.fromJson(json), videoMs);
switch (policy.decide(songMs, videoMs, songPlaying: true, videoReady: ready)) {
  case SeekTo(:final videoMs): player.seekTo(videoMs);
  case Nudge(:final speed): player.setSpeed(speed);
  case Hidden() || Ended(): showCover();
  case Hold(): break;
}

Status, platforms and limits

Limits you should know

• It rides on YouTube's undocumented player endpoints, the ones its own apps use. They can change; backends are pluggable for that reason, but today's defaults may one day need an update. Check YouTube's Terms of Service for your use case.
• A different mix, a live cut or a cover has no map, so it gets no canvas (the still cover stays) instead of a wrong one. A heavily edited video is aligned piecewise and unmatched stretches are hidden.
• In BitChord a synced music video wins over a label's short loop (Apple, Tidal, Spotify, community); the label sources are only the fallback for songs without a video that lines up.
• Measured on an emulator and a desktop, not on phones. 720p decode cost and battery are not measured.
• Maps are measured from the audio as your player plays it: pass the id that matches the master you play (or songAudio).

Collaborate or ask a question

OpenCanvas is a one-person project and I would like to hear how you would use it.