Today we announced a collaboration between IceWarp and Collabora to start the creation of LibreOffice On-Line, a scalable, cloud-hostable, full featured version of LibreOffice. My hope is that this has a huge and positive impact for the Free Software community, the business ecosystem, personal privacy, and more. Indeed, this is really one of the last big missing pieces that needs solving (alongside the Android version which is well underway). But wait - this post is supposed to be technical; lets get back to the code.
A prototype - with promise
At the beginning of the LibreOffice project, I created (for our first Paris Conference) a prototype of LibreOffice On-Line using Alex Laarson's (awesome) GTK+ Broadway - you can still see videos of that around the place. Great as the Broadway approach is (it provides essentially a simple Virtual Desktop model into your browser), the prototype taught us several important things which we plan to get right in LibreOffice On-Line:
- Performance - the Broadway model has the advantage of presenting the full application UI, however every time we want to do anything in the document - such as selecting, panning, or even blinking the cursor; we had to send new image fragments from the server: not ideal.
- Memory consumption / Scalability - another side effect of this is that, no matter how un-responsive the user is (how many tabs are you long-term-not-looking-at in your browser right now) it was necessary to have a full LibreOffice process running to be responsive & store the document. That memory consumption naturally significantly limits the ability to handle many concurrent clients.
- Scripting / web-like UI - it would have been possible to extend the gtk javascript to allow tunnelling bespoke commands through to LibreOffice to allow the wrapping of custom UI, but still the work to provide user interface that is expected on the web would be significant.
Having said all this, Broadway was a great basis to prove the feasibility of the concept - and we re-use the underlying concepts; in particular the use of web sockets to provide the low-latency interactions we need. Broadway also worked surprisingly well from eg. a nearby Amazon cloud datacentre. Similarly having full-fidelity rendering - is a very attractive proposition, independent of the fonts, or setup of the client.
An improved approach
Caching document views
One of the key realisations behind LibreOffice On-Line is that much of document editing is not the modification itself; a rather large proportion of time is spent reading, reviewing, and browsing documents. Thus by exposing the workings of document rendering to pixels squares (tiles) via LibreOfficeKit we can cache large chunks of the document content both on the server, and in the client's browser. As the users read though a document, or re-visit it, there is no need to communicate at all with the server, or even (after an initial rendering run) to have a LibreOfficeKit instance around there either.
Thus in this mode, the ability of the browser's Javascript to understand things about the document itself allows us to move much more of the pan/zoom reading goodness into your client. That means after an inital (pre)-fetch that responsiveness can be determined more by your local hardware and it's ability to pre-cache than remote server capacity. Interestingly, this same tiled-rendering approach is used by Fennec (Firefox for Android) and LibreOffice for Android to get smooth mobile-device scrolling and rendering, so LibreOfficeKit is already well adapted for this use-case.
Editing live documents
In recent times, The Document Foundation has funded, via the generosity of TDF's donors a chunk of infrastructure work to make it possible to use LibreOfficeKit to create custom document editors. There are several notable pieces of this work that intersect with this; I provide some links to the equivalent work being done for Android from Miklos Vajna:
Cursors & selection
Clearly blinking a cursor is something we can do trivially in the javascript client, rather than on the server; there are however several other interactions that benefit from browser acceleration. Text selection is a big piece of this - re-rendering text on the server simply in order to draw transparent selection rectangles over it makes very little sense - so instead we provide a list of rectangles to render in the browser. Similarly, drawing selection handles and interacting with images is something that can be handled pleasantly in the browser as well.
Keyboard / touch input
Clearly it is necessary to intercept browser keystrokes, gestures and so on, transport these over the websocket and emit them into the LibreOfficeKit core.
Tile invalidation / re-rendering
Clearly when the document changes, it is necessary to re-render and provide new tile data to the client; naturally there is an existing API for this that was put in place right at the start of the Android editing work.
Command invocation
Another piece that is required, is transporting UNO commands, and state (such as 'make it bold', or 'delete it') from the client javascript through into the LibreOfficeKit core. This is a matter again of proxying the required functionality via Javascript. The plan is to make it easy to create custom, bespoke UIs with a bit of CSS / Javascript magic wrapped around and interacting with the remote LibreOfficeKit core.
Serializing selections
Clearly as & when we decide that a user has wandered off, we can save their intermediate document, serialize the cursor location & selection - free up the resources for some other editing process. As/when they return we can then restore that with some small document load delay, as we transparently back their cached view with a live editable LibreOfficeKit instance.
What does that look like roughly ?
Of course, lots of pieces are still moving and subject to change; however here is a perhaps helpful drawing. Naturally integrating with existing storage, orchestration, and security frameworks will be important over time, contributions welcome for your pet framework:
The case for simple collaboration
A final, rather important part of LibreOffice On-Line; which I've left to last is that of collaborative editing.
The problem of generic, asynchronous, multi-instance / multi-device collaborative document editing is essentially horrendous. Solving even the easy problems (ie. re-ordering non-conflicting edits) is non-trivial for any large set of potentially intersecting operations. However, for this case, there are two very significant simplifying factors.
First there is a single, central instance of LibreOfficeKit rendering and providing document tiles to all clients. This significantly reduces the need to a re-order asynchronous change operation stream, it is also the case that editing conflicts should be seen as they are created.
Secondly, there is a controlled, and reasonably tractable set of extremely high-level operations based on abstract document co-ordinates - initially text selection, editing, deletion, object & shape movement, sizing, etc. which can be incrementally grown over time to extend to the core set of editing functionality.
These two simplifications, combined with managing and opportunistically strobing between users' cursor & selection contexts should allow us to provide the core of the document editing functionality.
Show me the code
The code is available as of now in gerrit's online repository. Clearly it is the Alpha not the Omega; the beginning, and not even the end of the beginning - which is a great time to get involved
Conclusion
LibreOffice On-Line is just beginning, there is a lot that remains to be done, and we appreciate help with that as we execute over the next year for IceWarp. A few words about IceWarp - having spent a rather significant amount of time pitching this work to people, and having listened to many requests for it - it is fantastic to be working with a company that can marry that great strategic sense with the resources and execution to actually start something potentially market-changing here; go IceWarp !
