Skip to content

Write plugins and elements ​

Use this page to extend the player. A plugin adds controllers or elements to it, with the SETUP parameters they need. An element is one step of the chain from a track to the screen: a source, a transform or a sink. How the player works explains the chain they plug into.

Write a plugin ​

definePlugin() checks a plugin definition and returns a factory. Calling the factory with the plugin's config gives an entry for the plugins option. Every hook is optional:

HookRunsDoes
init(config)Once per player, in the constructorReturns the plugin's context, ctx, which the other hooks get. Its parameter type is the factory's config.
setup(ctx, clientSetup)In the constructor, after initAdds SETUP parameters to clientSetup. Returning false drops the plugin.
negotiated(ctx, serverSetup)After the relay's SERVER_SETUPReturning false leaves the plugin inactive: none of its features are used.
features(ctx)Once per player, after initLists what the plugin offers.

Without setup or negotiated, the plugin counts as accepted. A feature is one of these:

typeitemHow the player uses it
buffer-controllera BufferControllerOnly the first active one. Attached while a negotiated stream is live.
abr-controlleran AbrControllerOnly the first active one. Attached like the buffer controller, except while a media source switches itself or a rendition is picked by hand.
elementan element definitionA candidate source, transform or sink, as if passed to registerElement().
switch-sourcea source element definitionLoaded by the catalog for each media switching set whose first track it accepts, as ssts() does.

Active plugins are ranked by priority, lowest first (default 100), then by their order in plugins. liveBuffer() and abr() have priority Infinity, so a controller of yours wins over them.

Offer an ABR controller ​

This plugin keeps the video at or below a height, in place of abr(). Its config is what init takes, cap.ts:

ts
import { definePlugin, type AbrController, type MOQtailPlayer } from 'moqtail/player'

/** Keeps each video source on its best rendition at or below `maxHeight`. */
class HeightCap implements AbrController {
  readonly #maxHeight: number
  readonly #pick = (): void => {
    for (const source of this.#player?.mediaSources ?? []) {
      const best = source
        .getRenditions()
        .filter((r) => r.selected && r.height !== undefined && r.height <= this.#maxHeight)
        .sort((a, b) => b.bitrate - a.bitrate)[0]
      if (best && !best.active) source.switch(best.id).catch(() => {})
    }
  }
  #player: MOQtailPlayer | undefined

  constructor(maxHeight: number) {
    this.#maxHeight = maxHeight
  }

  attach(player: MOQtailPlayer): void {
    this.#player = player
    player.on('catalog', this.#pick)
    this.#pick()
  }

  detach(): void {
    this.#player?.off('catalog', this.#pick)
    this.#player = undefined
  }
}

export const heightCap = definePlugin({
  name: 'height-cap',
  init: ({ maxHeight }: { maxHeight: number }) => {
    if (!(maxHeight > 0)) throw new TypeError('maxHeight must be a positive number')
    return new HeightCap(maxHeight)
  },
  features: (controller) => [{ type: 'abr-controller', item: controller }],
})

List it in plugins:

ts
import { MOQtailPlayer, liveBuffer } from 'moqtail/player'
import { heightCap } from './cap'

const video = document.querySelector('video')!
const player = new MOQtailPlayer({ plugins: [liveBuffer(), heightCap({ maxHeight: 720 })] })
  .source('moqt://relay.moqtail.dev#msf:moqtail-ch07--catalog')
  .sink(video)

await player.start()

A controller gets the player and the media sink in attach(), and undoes everything in detach(). A TypeError thrown by init reaches the caller of the MOQtailPlayer constructor. The player throws one itself when two entries in plugins have the same name.

Add a SETUP parameter ​

A protocol extension sends a SETUP parameter, and turns its features on only when the relay answers with it:

ts
import { KeyValuePair } from 'moqtail/model'
import { definePlugin } from 'moqtail/player'
import { textSource } from './text-source'

/** A setup option the relay echoes when it supports the extension. */
const TEXT_EXTENSION = 0x7f00n

export const textTracks = definePlugin({
  name: 'text-tracks',
  setup: (_, clientSetup) => {
    clientSetup.addRaw(KeyValuePair.tryNewVarInt(TEXT_EXTENSION, 1))
    return true
  },
  negotiated: (_, serverSetup) => serverSetup.setupOptions.some((option) => option.typeValue === TEXT_EXTENSION),
  features: () => [{ type: 'element', item: textSource }],
})

textSource is the source from Register a source. The parameters of every plugin are sent, active or not. Two plugins that set one parameter to different values make the constructor throw UNSUPPORTED_ORIGIN. Players on one page share a client, so they need the same parameters too.

Write an element ​

defineElement() checks an element definition and returns it with its types. An invalid definition throws a TypeError. A definition has these fields:

FieldWhat it is
type'source', 'transform' or 'sink'.
nameNot empty, no whitespace. Errors and warnings name the element by it.
capabilitiesCapability bundles: each in bundle describes a pad the element takes, each out bundle a pad it makes.
probe(track)Sources only, and required for them: whether the element handles the track.
priorityAmong the probes that accept a track, the lowest wins. Default 100.
providesThe interfaces the element implements.
supports(track)Whether the element can handle the track's media, asked before any instance exists. Leaving it out means yes.
threads(environment)Where the element's engines run: main, the default, both or worker. See Run the player in a worker.
init(context, options)Creates the instance. Reject with a PlayerError to set the code, for example when a browser API is missing. Any other rejection of an element you declared becomes UNSUPPORTED_SOURCE.

The instance extends one of three classes:

  • SourceElement creates its pads with createPad('src', capabilities), writes media to them, and emits nomorepads once every pad it will make exists.
  • TransformElement says in negotiate(pad) whether it takes an upstream pad, gets the pad's items in receive(pad, item), and writes its output to source pads it creates.
  • SinkElement negotiates and receives like a transform, and emits ready once it can play, after the player passes on nomorepads.

Pads and capabilities ​

A capability is a { direction, code, value }. The code is a CapabilityCode, such as PACKAGING (cmaf, loc, raw) or ROLE (video, audio). In a bundle, capabilities with the same code are alternatives, and different codes must all hold.

The player routes each new source pad to the first transform after its element that takes it, or else to every sink that does. A pad that none of them takes gets the fewest transforms that make it acceptable, such as the built-in webcodecs decoder in front of a canvas. If there are none, the player warns PAD_UNLINKED.

A pad carries items in order:

  • packet: the payload of one object.
  • frame: a decoded VideoFrame or AudioData. The element that receives it owns it, and closes it or passes it on.
  • properties: pad properties, such as mimeType or initData.
  • event: a discontinuity or the end of the stream, eos.

await pad.write(item) applies backpressure: it waits while a connection holds 32 items. Events and signals an element doesn't handle pass through it to its other pads.

Element interfaces ​

What the player asks of an element beyond its pads goes through the interfaces it lists in provides. TypeScript checks the list against what init returns.

KeyInterfaceThe player uses it for
MEDIA_SOURCEMediaSourceElementRenditions: getRenditions(), switch(), selectRenditions(), setTracks(). player.mediaSources lists these sources, except those another media source loaded.
ABRMediaSourceElementA media source that switches among its selected renditions itself. The ABR controller leaves it alone.
MEDIA_SINKMediaSinkElementPlayback: play(), pause(), seek(), setPlaybackRate(), the time and the media events. The first one is player.mediaSink.

hasInterface(element, MEDIA_SINK) narrows an element to the interface, when its definition lists it.

What an element can reach ​

Each instance gets an ElementContext in init, kept as this.context:

  • player: the player's events and options, and its buffer controller.
  • client: the current client. It changes after a reconnect, so listen for the player's client event.
  • signal: aborted when the instance is destroyed.
  • tracks and catalog: the catalog tracks the element was loaded for, and the catalog.
  • fail(error): reports a fatal error from outside receive(), as a PlayerError with the element's name in element.

Write a transform ​

This transform turns video gray. It takes raw video pads, the decoded frames a canvas plays, and makes one output pad for its one input, grayscale.ts:

ts
import {
  CapabilityCode,
  TransformElement,
  defineElement,
  type ElementContext,
  type Pad,
  type PadItem,
} from 'moqtail/player'

class GrayscaleTransform extends TransformElement {
  readonly #canvas = new OffscreenCanvas(1, 1)
  readonly #draw = this.#canvas.getContext('2d')!
  #output: Pad | undefined

  constructor(context: ElementContext) {
    super(context)
    this.on('padadded', (pad) => {
      if (pad.direction !== 'sink') return
      this.#output = this.createPad('src', [
        { direction: 'out', code: CapabilityCode.PACKAGING, value: 'raw' },
        { direction: 'out', code: CapabilityCode.ROLE, value: 'video' },
      ])
    })
    this.on('padremoved', (pad, failed) => {
      if (pad.direction === 'sink' && this.#output) this.removePad(this.#output, failed)
    })
  }

  negotiate(pad: Pad): boolean {
    // One video pad per instance
    return !this.pads.some((p) => p.direction === 'sink') && this.matchesCapabilities(pad)
  }

  async receive(_pad: Pad, item: PadItem): Promise<void> {
    if (item.type !== 'frame' || !(item.frame.payload instanceof VideoFrame)) {
      await this.#output?.write(item)
      return
    }
    const frame = item.frame.payload
    if (this.#canvas.width !== frame.displayWidth || this.#canvas.height !== frame.displayHeight) {
      this.#canvas.width = frame.displayWidth
      this.#canvas.height = frame.displayHeight
    }
    this.#draw.filter = 'grayscale(1)'
    this.#draw.drawImage(frame, 0, 0)
    const gray = new VideoFrame(this.#canvas, { timestamp: frame.timestamp })
    frame.close()
    await this.#output?.write({ type: 'frame', frame: { ...item.frame, payload: gray } })
  }
}

export const grayscale = defineElement({
  type: 'transform',
  name: 'grayscale',
  capabilities: [
    [
      { direction: 'in', code: CapabilityCode.PACKAGING, value: 'raw' },
      { direction: 'in', code: CapabilityCode.ROLE, value: 'video' },
    ],
    [{ direction: 'out', code: CapabilityCode.PACKAGING, value: 'raw' }],
  ],
  init: async (context) => new GrayscaleTransform(context),
})
  • matchesCapabilities(pad) checks the pad against the definition's in bundles.
  • Items other than video frames, such as pad properties, go on unchanged.
  • Events need no code: unhandled, they pass on to the output pad. A removed input removes the output, keeping failed.

Add it to the chain with transform(), between source() and sink():

ts
import { MOQtailPlayer, liveBuffer } from 'moqtail/player'
import { grayscale } from './grayscale'

const canvas = document.querySelector('canvas')!
const player = new MOQtailPlayer({ plugins: [liveBuffer()] })
  .source('moqt://relay.moqtail.dev#msf:moqtail-testsrc--catalog')
  .transform(grayscale)
  .sink(canvas)

await player.start()

The chain loads the webcodecs decoder in front of the transform, so it receives decoded frames. The transform runs on the main thread: Run the player in a worker moves it into the worker.

transform() adds one instance to the chain, which is offered every pad that reaches it. registerElement() instead makes a transform a candidate that the chain loads only where a pad needs it to reach a sink, one instance per pad. A canvas already takes raw video, so the chain would never load the grayscale transform that way.

Register a source ​

The player picks the source element for every track by probe: the one from the URL, and each one the catalog lists. A source you register wins a tie with the built-in ones:

ts
import { CapabilityCode, MOQtailPlayer, SourceElement, defineElement } from 'moqtail/player'

class TextSource extends SourceElement {
  // Creates its pads, writes each object to them as a packet, then emits 'nomorepads'
}

export const textSource = defineElement({
  type: 'source',
  name: 'text',
  capabilities: [[{ direction: 'out', code: CapabilityCode.PACKAGING, value: 'text' }]],
  probe: (track) => track.packaging === 'text',
  init: async (context) => new TextSource(context),
})

export const player = new MOQtailPlayer().registerElement(textSource)

Call registerElement() before source(). For each track, the candidates come in this order:

  1. The definitions passed to registerElement().
  2. The element features of the active plugins, by plugin priority, then order.
  3. The built-in sources: catalog, CMAF, LOCMAF, LOC, media timeline and SAP timeline.

Among those whose probe accepts the track, the lowest priority wins, then the earliest. Transforms the chain loads come from the same three places, with the built-in webcodecs last.

A track URL that no probe accepts alone is requested, and probed again with its trackProperties, then after each object it reads with objects: see Which tracks play. The source's options then carry the identifiers it was accepted with, as source. The built-in media sources play on the request the player sent, which a source of your own can't read: the player cancels it after init().

The pads of the source still need a transform or sink whose in capabilities match them, such as a sink you pass to player.sink(definition, options). Without one, the player warns PAD_UNLINKED. Subscribe and fetch shows how to read a track with context.client.

More information ​