Write plugins and elements
Use this page to extend the player. A plugin adds controllers or elements to it, with the SETUP parameters they need. An element is one step of the chain from a track to the screen: a source, a transform or a sink. How the player works explains the chain they plug into.
Write a plugin
definePlugin() checks a plugin definition and returns a factory. Calling the factory with the plugin's config gives an entry for the plugins option. Every hook is optional:
| Hook | Runs | Does |
|---|---|---|
init(config) | Once per player, in the constructor | Returns the plugin's context, ctx, which the other hooks get. Its parameter type is the factory's config. |
setup(ctx, clientSetup) | In the constructor, after init | Adds SETUP parameters to clientSetup. Returning false drops the plugin. |
negotiated(ctx, serverSetup) | After the relay's SERVER_SETUP | Returning false leaves the plugin inactive: none of its features are used. |
features(ctx) | Once per player, after init | Lists what the plugin offers. |
Without setup or negotiated, the plugin counts as accepted. A feature is one of these:
type | item | How the player uses it |
|---|---|---|
buffer-controller | a BufferController | Only the first active one. Attached while a negotiated stream is live. |
abr-controller | an AbrController | Only the first active one. Attached like the buffer controller, except while a media source switches itself or a rendition is picked by hand. |
element | an element definition | A candidate source, transform or sink, as if passed to registerElement(). |
switch-source | a source element definition | Loaded by the catalog for each media switching set whose first track it accepts, as ssts() does. |
Active plugins are ranked by priority, lowest first (default 100), then by their order in plugins. liveBuffer() and abr() have priority Infinity, so a controller of yours wins over them.
Offer an ABR controller
This plugin keeps the video at or below a height, in place of abr(). Its config is what init takes, cap.ts:
import { definePlugin, type AbrController, type MOQtailPlayer } from 'moqtail/player'
/** Keeps each video source on its best rendition at or below `maxHeight`. */
class HeightCap implements AbrController {
readonly #maxHeight: number
readonly #pick = (): void => {
for (const source of this.#player?.mediaSources ?? []) {
const best = source
.getRenditions()
.filter((r) => r.selected && r.height !== undefined && r.height <= this.#maxHeight)
.sort((a, b) => b.bitrate - a.bitrate)[0]
if (best && !best.active) source.switch(best.id).catch(() => {})
}
}
#player: MOQtailPlayer | undefined
constructor(maxHeight: number) {
this.#maxHeight = maxHeight
}
attach(player: MOQtailPlayer): void {
this.#player = player
player.on('catalog', this.#pick)
this.#pick()
}
detach(): void {
this.#player?.off('catalog', this.#pick)
this.#player = undefined
}
}
export const heightCap = definePlugin({
name: 'height-cap',
init: ({ maxHeight }: { maxHeight: number }) => {
if (!(maxHeight > 0)) throw new TypeError('maxHeight must be a positive number')
return new HeightCap(maxHeight)
},
features: (controller) => [{ type: 'abr-controller', item: controller }],
})List it in plugins:
import { MOQtailPlayer, liveBuffer } from 'moqtail/player'
import { heightCap } from './cap'
const video = document.querySelector('video')!
const player = new MOQtailPlayer({ plugins: [liveBuffer(), heightCap({ maxHeight: 720 })] })
.source('moqt://relay.moqtail.dev#msf:moqtail-ch07--catalog')
.sink(video)
await player.start()A controller gets the player and the media sink in attach(), and undoes everything in detach(). A TypeError thrown by init reaches the caller of the MOQtailPlayer constructor. The player throws one itself when two entries in plugins have the same name.
Add a SETUP parameter
A protocol extension sends a SETUP parameter, and turns its features on only when the relay answers with it:
import { KeyValuePair } from 'moqtail/model'
import { definePlugin } from 'moqtail/player'
import { textSource } from './text-source'
/** A setup option the relay echoes when it supports the extension. */
const TEXT_EXTENSION = 0x7f00n
export const textTracks = definePlugin({
name: 'text-tracks',
setup: (_, clientSetup) => {
clientSetup.addRaw(KeyValuePair.tryNewVarInt(TEXT_EXTENSION, 1))
return true
},
negotiated: (_, serverSetup) => serverSetup.setupOptions.some((option) => option.typeValue === TEXT_EXTENSION),
features: () => [{ type: 'element', item: textSource }],
})textSource is the source from Register a source. The parameters of every plugin are sent, active or not. Two plugins that set one parameter to different values make the constructor throw UNSUPPORTED_ORIGIN. Players on one page share a client, so they need the same parameters too.
Write an element
defineElement() checks an element definition and returns it with its types. An invalid definition throws a TypeError. A definition has these fields:
| Field | What it is |
|---|---|
type | 'source', 'transform' or 'sink'. |
name | Not empty, no whitespace. Errors and warnings name the element by it. |
capabilities | Capability bundles: each in bundle describes a pad the element takes, each out bundle a pad it makes. |
probe(track) | Sources only, and required for them: whether the element handles the track. |
priority | Among the probes that accept a track, the lowest wins. Default 100. |
provides | The interfaces the element implements. |
supports(track) | Whether the element can handle the track's media, asked before any instance exists. Leaving it out means yes. |
threads(environment) | Where the element's engines run: main, the default, both or worker. See Run the player in a worker. |
init(context, options) | Creates the instance. Reject with a PlayerError to set the code, for example when a browser API is missing. Any other rejection of an element you declared becomes UNSUPPORTED_SOURCE. |
The instance extends one of three classes:
SourceElementcreates its pads withcreatePad('src', capabilities), writes media to them, and emitsnomorepadsonce every pad it will make exists.TransformElementsays innegotiate(pad)whether it takes an upstream pad, gets the pad's items inreceive(pad, item), and writes its output to source pads it creates.SinkElementnegotiates and receives like a transform, and emitsreadyonce it can play, after the player passes onnomorepads.
Pads and capabilities
A capability is a { direction, code, value }. The code is a CapabilityCode, such as PACKAGING (cmaf, loc, raw) or ROLE (video, audio). In a bundle, capabilities with the same code are alternatives, and different codes must all hold.
The player routes each new source pad to the first transform after its element that takes it, or else to every sink that does. A pad that none of them takes gets the fewest transforms that make it acceptable, such as the built-in webcodecs decoder in front of a canvas. If there are none, the player warns PAD_UNLINKED.
A pad carries items in order:
packet: the payload of one object.frame: a decodedVideoFrameorAudioData. The element that receives it owns it, and closes it or passes it on.properties: pad properties, such asmimeTypeorinitData.event: adiscontinuityor the end of the stream,eos.
await pad.write(item) applies backpressure: it waits while a connection holds 32 items. Events and signals an element doesn't handle pass through it to its other pads.
Element interfaces
What the player asks of an element beyond its pads goes through the interfaces it lists in provides. TypeScript checks the list against what init returns.
| Key | Interface | The player uses it for |
|---|---|---|
MEDIA_SOURCE | MediaSourceElement | Renditions: getRenditions(), switch(), selectRenditions(), setTracks(). player.mediaSources lists these sources, except those another media source loaded. |
ABR | MediaSourceElement | A media source that switches among its selected renditions itself. The ABR controller leaves it alone. |
MEDIA_SINK | MediaSinkElement | Playback: play(), pause(), seek(), setPlaybackRate(), the time and the media events. The first one is player.mediaSink. |
hasInterface(element, MEDIA_SINK) narrows an element to the interface, when its definition lists it.
What an element can reach
Each instance gets an ElementContext in init, kept as this.context:
player: the player's events and options, and its buffer controller.client: the current client. It changes after a reconnect, so listen for the player'sclientevent.signal: aborted when the instance is destroyed.tracksandcatalog: the catalog tracks the element was loaded for, and the catalog.fail(error): reports a fatal error from outsidereceive(), as aPlayerErrorwith the element's name inelement.
Write a transform
This transform turns video gray. It takes raw video pads, the decoded frames a canvas plays, and makes one output pad for its one input, grayscale.ts:
import {
CapabilityCode,
TransformElement,
defineElement,
type ElementContext,
type Pad,
type PadItem,
} from 'moqtail/player'
class GrayscaleTransform extends TransformElement {
readonly #canvas = new OffscreenCanvas(1, 1)
readonly #draw = this.#canvas.getContext('2d')!
#output: Pad | undefined
constructor(context: ElementContext) {
super(context)
this.on('padadded', (pad) => {
if (pad.direction !== 'sink') return
this.#output = this.createPad('src', [
{ direction: 'out', code: CapabilityCode.PACKAGING, value: 'raw' },
{ direction: 'out', code: CapabilityCode.ROLE, value: 'video' },
])
})
this.on('padremoved', (pad, failed) => {
if (pad.direction === 'sink' && this.#output) this.removePad(this.#output, failed)
})
}
negotiate(pad: Pad): boolean {
// One video pad per instance
return !this.pads.some((p) => p.direction === 'sink') && this.matchesCapabilities(pad)
}
async receive(_pad: Pad, item: PadItem): Promise<void> {
if (item.type !== 'frame' || !(item.frame.payload instanceof VideoFrame)) {
await this.#output?.write(item)
return
}
const frame = item.frame.payload
if (this.#canvas.width !== frame.displayWidth || this.#canvas.height !== frame.displayHeight) {
this.#canvas.width = frame.displayWidth
this.#canvas.height = frame.displayHeight
}
this.#draw.filter = 'grayscale(1)'
this.#draw.drawImage(frame, 0, 0)
const gray = new VideoFrame(this.#canvas, { timestamp: frame.timestamp })
frame.close()
await this.#output?.write({ type: 'frame', frame: { ...item.frame, payload: gray } })
}
}
export const grayscale = defineElement({
type: 'transform',
name: 'grayscale',
capabilities: [
[
{ direction: 'in', code: CapabilityCode.PACKAGING, value: 'raw' },
{ direction: 'in', code: CapabilityCode.ROLE, value: 'video' },
],
[{ direction: 'out', code: CapabilityCode.PACKAGING, value: 'raw' }],
],
init: async (context) => new GrayscaleTransform(context),
})matchesCapabilities(pad)checks the pad against the definition'sinbundles.- Items other than video frames, such as pad properties, go on unchanged.
- Events need no code: unhandled, they pass on to the output pad. A removed input removes the output, keeping
failed.
Add it to the chain with transform(), between source() and sink():
import { MOQtailPlayer, liveBuffer } from 'moqtail/player'
import { grayscale } from './grayscale'
const canvas = document.querySelector('canvas')!
const player = new MOQtailPlayer({ plugins: [liveBuffer()] })
.source('moqt://relay.moqtail.dev#msf:moqtail-testsrc--catalog')
.transform(grayscale)
.sink(canvas)
await player.start()The chain loads the webcodecs decoder in front of the transform, so it receives decoded frames. The transform runs on the main thread: Run the player in a worker moves it into the worker.
transform() adds one instance to the chain, which is offered every pad that reaches it. registerElement() instead makes a transform a candidate that the chain loads only where a pad needs it to reach a sink, one instance per pad. A canvas already takes raw video, so the chain would never load the grayscale transform that way.
Register a source
The player picks the source element for every track by probe: the one from the URL, and each one the catalog lists. A source you register wins a tie with the built-in ones:
import { CapabilityCode, MOQtailPlayer, SourceElement, defineElement } from 'moqtail/player'
class TextSource extends SourceElement {
// Creates its pads, writes each object to them as a packet, then emits 'nomorepads'
}
export const textSource = defineElement({
type: 'source',
name: 'text',
capabilities: [[{ direction: 'out', code: CapabilityCode.PACKAGING, value: 'text' }]],
probe: (track) => track.packaging === 'text',
init: async (context) => new TextSource(context),
})
export const player = new MOQtailPlayer().registerElement(textSource)Call registerElement() before source(). For each track, the candidates come in this order:
- The definitions passed to
registerElement(). - The
elementfeatures of the active plugins, by plugin priority, then order. - The built-in sources: catalog, CMAF, LOCMAF, LOC, media timeline and SAP timeline.
Among those whose probe accepts the track, the lowest priority wins, then the earliest. Transforms the chain loads come from the same three places, with the built-in webcodecs last.
A track URL that no probe accepts alone is requested, and probed again with its trackProperties, then after each object it reads with objects: see Which tracks play. The source's options then carry the identifiers it was accepted with, as source. The built-in media sources play on the request the player sent, which a source of your own can't read: the player cancels it after init().
The pads of the source still need a transform or sink whose in capabilities match them, such as a sink you pass to player.sink(definition, options). Without one, the player warns PAD_UNLINKED. Subscribe and fetch shows how to read a track with context.client.
More information
definePlugin,PluginDefinition,PluginandFeatureBufferControllerandAbrControllerdefineElement,ElementDefinitionandElementContextSourceElement,TransformElement,SinkElementandPadCapabilityCode,PadItemandPadEventMediaSourceElement,MediaSinkElementandhasInterfaceMOQtailPlayer, forregisterElement(),transform()andsink()