For the past few months, I've been working on Project Silk which improves smoothness across the browser. Very much like Project Butter for Android, part of it is finally live on Firefox OS. Silk does three things:
- Align Painting with hardware vsync
- Resample touch input events based on hardware vsync
- Align composites with hardware vsync
What is vsync, why vsync and why does it matter at all?
Vertical synchronization occurs when the hardware display shows a new frame onto the screen. This rate is set by specific hardware, but major displays in the US occur at 60 times a second, or every 16.6 ms. This is where you hear about 60 frames per second, one frame every time the hardware display refreshes. What this means in reality is that no matter how many frames are produced in software, the hardware display will still only show at most 60 unique frames per second.
Currently in Firefox, we mimic 60 frames per second and therefore vsync with a software timer that schedules rendering every 16.6 ms. However, the software scheduler has two problems: (a) it's noisy and (b) can be scheduled at bad times relative to vsync.
In regards to noise, software timers are much noiser than hardware timers. This creates micro-jank for a number of reasons. First, many animations are keyed off timestamps that occur from the software scheduler to update the position of the animation. If you've ever used requestAnimationFrame, you get a timestamp from a software timer. If we want smooth animations, the timestamp provided to requestAnimationFrame should be uniform. Non-uniform timestamps will create non-uniform and janky animations. Here is a graph showing software versus hardware vsync timer uniformity:
Wow! That's a lot better with a hardware timer. With hardware timers, we get a much more uniform, and therefore smoother timestamp to key animations off of. So that's problem (a), noisy timers in software versus hardware.
With part (b), software timers can be scheduled at bad times relative to vsync. Regardless of what software does, the hardware display will refresh on it's own clock. If our rendering pipeline finishes producing a frame before the next vsync, the display is updated with new content. If we fail to finish producing a frame before the next vsync, the previous frame will be displayed, causing jankiness. Some rendering functions can occur close to vsync and overflow until the next interval, so we actually introduce potentially more latency since the frame won't be displayed on the screen anyway until the next vsync. Let's look at this in graphic form:
At time 0, we start producing frames. Let's say all frames for the sake of example take a constant time of 10 ms. Our frame budget is 16.6 ms because we only have to finish producing a frame before the hardware vsync occurs. Since frame 1 is finished 6 ms before the next vsync (time t=16 ms), everything is successful and life is good. The frame is produced in time and the hardware display will be refreshed with the updated content.
Now let's look at Frame 2. Since software timers are noisy, we start producing a frame 9 ms from the next vsync (time t=32). Since our frame takes 10 ms to produce, we'll actually finish producing this frame at 1 ms AFTER the next vsync. That means at vsync number 2 (t=32), there was no new frame to display, so the display still shows the previous frame. In addition, the frame just produced won't be shown until vsync 3 (t=48), because that's when hardware updates itself. This creates jank since now the display will have skipped 1 frame and try to catch up in the upcoming frames. This also produces one extra frame of latency, which is terrible for games.
Vsync improves both of these problems since we get both a much more uniform timer and the maximum amount of frame budget time to produce a new frame. Now that we know what vsync is, we can finally go onto what Project Silk is and why it helps create smooth experiences in Firefox.
The Rendering Pipeline
Gecko's rendering pipeline in super simplified terms does three things:
- Paint / draw the new frame on the main thread.
- Send the updated content to the Compositor via a LayerTransaction.
- Composite the new content.
In the ideal world, we'd be able to do all three steps within 16.6 ms, but that's not the case most of the time. Both steps (1) and (3) are on independent software timers. Thus there was no real synchronizing clock between the three steps, they are all ad-hoc. They also had no relation to vsync, so the timing of the pipeline wasn't related to when the display would actually update the screen with the content. With Silk, we replace both independent software timers with the hardware vsync timer. For our purposes, (2) isn't really affected useful but only here for completeness.
Align Painting with Hardware Vsync
Aligning the timer used to tick the refresh driver with vsync creates smoothness in a couple of ways. First, many animations are still done on the main thread, which means any animation using timestamps to set the position of an animation should be smoother. This includes requestAnimationFrame animations! The other nice thing is that we now have a very strict ordering of when rendering is kicked off. Instead of (1) and (3), which are related occurring at a synced offset, we start rendering at a specific time.
Resample Touch Input Events Based on Vsync
With Silk, we can enable touch resampling which improves smoothness while tracking your finger. Since I've gone over what touch resampling is quite a bit, I'll leave this short. With Silk, we can finally enable it!
Align Composites with Hardware Vsync
Finally, the last part of Silk is aligning composites with hardware vsync. Compositing takes all the painted content and merges it together to create a single image you see on the display. With Silk, all composites start right after a hardware vsync occurs. This has actually produced a rather nice side benefit: reduced composite times. See:
Within the device driver on a Flame device, there's a global lock that's grabbed when close to vsync intervals. This lock can take 5-6 ms to get, greatly increasing the composite times. However, when we start a composite right after a vsync, there is little contention to grab the lock. Thus we can shave off the wait, therefore reducing composite times quite a bit. Not only do we get smoother animations, but also reduced composite times and therefore better battery life. What a nice win!
With all three pieces, we now have a nice strict ordering of the rendering pipeline. We paint and send the updated content to the Compositor within 16.6 ms. At the next vsync, we composite the updated content. At the vsync after that, the frame should have gone through the rendering pipeline and will be displayed on the screen. Keeping this order reduces jank because we reduce the chances that the timers schedule each step at a bad time. For example, in the current implementation without silk, the best case would be that a frame could be painted and composited within a single 16.6 ms frame, which is great. However, if the next frame takes 2 frames instead, we just created extra jank even though no stage in the pipeline was really slow. Aligning the whole pipeline to create a strict ordering reduces the chances that we mis-schedule a frame.
Here's a picture of the rendering pipeline without Silk. We have Composites (3) on the bottom of this profile. We have painting (1) in the middle, where you see Styles, Reflow, Displaylist and Rasterize. We have Vsync, which are those small little orange boxes at the top. Finally we have Layer Transactions (2) at the bottom. First, when we start Compositing and painting are not aligned, so animations will be at different positions depending if they are on the main thread or compositor thread. Second, we see long composites because the compositor is waiting on a global lock in the device driver. Lastly, it's rather difficult to read any ordering or see if there is a problem without deep knowledge of why / when things should be happening.
Here is a picture of the same pipeline with Silk. Composites are a little shorter, and the whole pipeline only starts at vsync intervals. Composite times are reduced because we start composites exactly at vsync intervals. There is a clear ordering of when things should happen and both composites and painting are keyed off the same timestamp, ensuring smoother animations. Finally, there is a clear indicator that as long as everything finishes before the next Vsync, things will be smooth.
Overall, Silk hopes to create a smoother experience across Firefox and the web. Numerous people contributed to the project. Thanks to Jerry Shih, Boris Chou, Jeff Hwang, Mike Lee, Kartikaya Gupta, Benoit Girard, Michael Wu, Ben Turner, Milan Sreckovic for their help in making Silk happen.