Lip Sync Approaches
Video conferencing endpoints generally take two approaches to achieve lip sync:
- Poor Man's lip sync—This method assumes that delays in the end-to-end media paths are known and constant. It relies on packet arrival times for synchronization.
- Common Reference lip sync—This method assumes that delays in the end-to-end media paths are not easily known and might vary. It relies on a common reference timebase for both audio and video streams.
Poor Man's Lip Sync
The simplest incarnation of a lip sync algorithm is known as Poor Man's lip sync. In this method, the receiver uses one criterion to synchronize audio and video: Packets of audio and video that arrive simultaneously at the network interface of the receiver are considered to be synchronized to each other. This approach is fundamentally flawed because delays in the end-to-end path vary both in space (at different points of the path) and time (fluctuations in delay from one moment to the next). In addition, trying to measure, predict, and compensate for these variable delays in the end-to-end path is a futile effort.
When using Poor Man's lip sync, the conferencing system must make several assumptions about the sender, receiver, and network infrastructure:
- Sender—For Poor Man's lip sync, the sender generally assumes that the network delay to the receiver is the same for both audio and video streams. However, scenarios might arise in which the delays differ. For instance, a transcoder might be present in the audio path but not the video path. Or, the network might assign a higher QoS to one path, resulting in lower delay for that stream.
- Receiver—When operating with Poor Man's lip sync, the receiver must derive a relationship between the time stamps of the two streams by observing the relationship between the packet arrival times and timestamps for each individual stream, and then using that information to derive a relationship between the packet timestamps of the two streams. However, the receiver might have difficulty deriving an accurate relationship, because packet arrival times vary because of arrival-time jitter.
Network infrastructure—Poor Man's lip sync makes the following invalid assumptions:
- - The average network delay remains constant over the long term.
- - The instantaneous network delay remains constant over the short term.
When you are using Poor Man's lip sync, if the sender cannot compensate for network delays, specialized video conferencing network infrastructure might be necessary between the two endpoints. This infrastructure readjusts the synchronization of audio and video streams by adding delay to one stream or another.
An unfortunate byproduct of Poor Man's lip sync is that it often results in the sender or the network infrastructure delaying one or more streams to attempt synchronization at the receiver. However, for maximum flexibility, delays should be introduced only at the receivers, which leads to one of the corollaries of robust lip sync:
- Only the receiver should delay media streams to achieve lip sync.
This corollary is necessary for two reasons:
- If the end-to-end audio delay is already significant, the receiver might prefer to avoid adding more delay to the audio stream and forego lip sync. Instead, the receiver might want to go without audio and video synchronization to maintain the lowest audio end-to-end delay for the best interaction between conference participants. The end user makes this decision via the user interface (UI) of the receiver endpoint. Therefore, other entities on the network should not overrule this decision.
- The process of delaying one stream to achieve lip sync should be left to the receiver, because the receiver can take into account its own internal delays in each media path at the same time that it delays one stream or the other to synchronize media. For instance, if audio arrives at the receiver ahead of video, normally the receiver must delay the audio stream to achieve lip sync with the video. However, an input buffer in the receiver might already provide some or all of this delay.
Common Reference Lip Sync
The goal of lip sync is to preserve the relationship between audio and video in the presence of fluctuating end-to-end delays in both the network and the endpoints themselves. Therefore, the most important restriction to keep in mind when discussing lip sync for video conferencing is the following:
- Video conferencing systems cannot accurately measure or predict all delays in the end-to-end path for either the audio or video stream.
This restriction leads to the most important corollary of lip sync:
- A method of lip synchronization must not use a mechanism that attempts to measure and compensate for individual delays in the end-to-end path.
The second corollary addresses the method that systems should use to achieve lip sync:
- A method of lip synchronization must use timestamps that can be correlated to a common timebase.
Before considering a robust method of synchronization using a common reference, it is necessary to cover the data path inside the sender and receiver of a video conference.