Many areas of LibreOffice have been hugely improved in recent times, from the general cleanup of the code, to the huge VCL / UI layout re-work touching all of our dialogs, the significant re-work of Calc's internals - many areas of the code have had big improvements. One area that has however sadly fallen behind is the Visual Class Libraries (VCL) rendering model - that is used to draw nearly everything in the document, and chrome around it.
Open GL - as-is
OpenGL from the Khronos Group is, of course, the premier cross-platform rendering API for hardware acceleration in the modern world, with implementations from iOS and Android to Windows, Linux & Mac. We have started to use OpenGL in LibreOffice in several places, but unfortunately OpenGL has some pretty unfortunate all-or-nothing type decisions in its platform implementation. What does that mean ? Take for example our infamous Collada duck:
To fix this we need to move LibreOffice rendering into the modern world of hardware accelerated / OpenGL graphics. By rendering all the normal, 2D work into an OpenGL context, we can embed rich, 3D visualisations and data seamlessly into the suite with good performance.
Faster, higher quality rendering
The graphics compute units in modern hardware GPUs are great at rendering (as well as calculating complex spreadsheets). They work in parallel to the CPU, thus (effectively) asynchronously off-loading lots of work from the CPU which can get on with all the heavy lifting to layout your documents.
VCL in contrast, has had extremely variable performance. This is extremely noticeable on mobile devices like Android - where VCL has had to use pure basebmp / CPU rendering on some already under-powered CPU hardware. Meanwhile the powerful GPU cores in your phone/tablet have been idling. By using OpenGL we can ensure that each compute unit is doing the right thing and make optimum use of the available power.
Image scaling is another area where we currently suffer; with several open bugs - first one complains about performance, and then when you lower rendering quality to get performance, another bug complains about rendering quality. Doing high quality image interpolation of large images takes time, even when threaded. People love to whack large, high-DPI images into their documents and presentations. By moving all of the image interpolation work to the GPU we should be able to have our cake: pretty scaled images, and also eat it quickly: with fast rendering.
Another area that is currently slow is gradient rendering; with current VCL implementations, most gradients that you see are decomposed into innumerable smaller polygons by VCL and then many hundreds of these are rendered. This can bring a significant performance penalty. Thanks to Chris Sherlock we now have the ability to render arbitrary gradients in the platform / OpenGL backend. This makes a lot of sense since rendering gradients is something that can be done almost for free by a GPU - lovingly shading each pixel at incredible speed.
VCL / Structural issues
As the Linux world moves to Wayland, LibreOffice will need to adapt. In particular the option of client-side rendering in software (as used in our current gtk3 port) is not incredibly attractive - that suffers from a number of existing rendering problems, as well as being very substantially slower. By moving straight to an OpenGL solution, we should be able to have reasonably optimal Wayland rendering too.
One other problem plagues VCL rendering however; and that is 'immediate' rendering. LibreOffice renders in one of two ways - either immediately: so when you press an 'A' it tries to nail the pixels for 'A' immediately to the screen; or - a very deferred idle rendering which happens a hard-coded 35ms later. This situation is really non-ideal for modern rendering where we want to ensure the scene is perfect before showing it on-screen. However, this is not something that we can fix immediately. Faster deferred rendering will miss 4.4 - the problems are too wide-spread. Having said that, some great work has been done by Jennifer Liebel and Tobias Madl from the Limux team (Munich) - who are implementing a new way for 'idle' tasks - such as re-drawing invalidated rectangles to be run immediately with prioritisation instead of hoping that various combinations of timeouts: 35ms, 50ms etc. arrive in the right order. This should form a great foundation for us to remove immediate rendering in favour of idle rendering without a performance impact in subsequent releases. Thanks also to Kendy (Collabora) for finding and nailing a nasty Windows timer issue; using high-resolution Timer Queues to avoid Windows' (unbelievable) Timers which cannot sleep for less than 10ms.
What we're doing
The new OpenGL rendering code provides a VCL back-end implementation that
can be used to render all VCL Windows' content via shared code on all platforms,
poke in vcl/opengl for that. Of course, there is also a certain amount
of per-platform setup and context management work that has to be done for each
platform as well, but the great bulk of the code is fully shared. Markus has also
re-factored to help split the Native Widget framework (used for platform theming)
and the Text APIs (which are also different per-platform) from the stock GL
rendering code.
The new OpenGL backend, thus allows all application rendering to happen to the same OpenGL context. This allows us to avoid a dichotomoy of rendering with GDI or X11 on windows, and then needing to box all 3D / OpenGL content into its own window. This should allow seamless integration of 3D content, as well as giving much better performance particularly on mobile platforms.
Testing VCL implementations
One area that VCL has not been ideal is the lack of a well written test / demo application. This is somewhat ironic since the LibreOffice code-base stated off decades ago as the demo app for the VCL toolkit. Over the last week, we've done a little expansion and started to implement something that aims at exercising all of the relevant backend / VCL API in an easy to extend fashion.
Naturally, the OpenGL code is not for everyone - neither now, or for the forseable future. Many OpenGL implementations have debilitating bugs, despite using only a small and simple sub-set of OpenGL features : we believe that will need to black-list these. Similarly, we will want to white-list newer and more stable drivers, but of course users will be able to choose between legacy and OpenGL rendering too.
What it looks like
So - when you put all of that together; it is advancing rapidly. Markus just checked some code into master which shows some of the VCL primitives that we can render already:
Ongoing work
You can play with the work on master; clearly LibreOffice is not rendering perfectly with this yet - that goes without saying; and help is much appreciated in the lead-up to LibreOffice 4.4; we love to work with other people.
Credits
Thanks to Markus Mohrhard (Collabora), Louis-Francis Ratté-Boulianne (Collabora), Ptyl Dragon (CloudOn), Chris Sherlock, along with more future / pending work in-progress from Michael Jaumann working on OpenGL canvas and Stefan Weiberg on OGL Transitions both mentored by Thorsten Behrens (SUSE).
