LILD Project description
Goal: Stream large medical images over the server (jpeg2000 and tif) and display them in webpage/ipython notebook.
Desired outcome: demo
- Running inside a notebook is a nice-to-have, not critical.
- We need the ability to annotate and change annotations. Zoomify provides such ability. The annotation has a front-end which captures the marks of the user and a back-end which stores them and serves them back on demand. I would like to believe that it is possible for a single back-end to handle both the images and the annotations.
- One nice thing about annotations is that their memory footprint is small, so the communication to the backend when the annotation is altered should be minimal.
- An existing tool that does something similar (but no line annotations) is this: http://brainmaps.org/index.php
- Use openseadragon. A mature open source JS library designed to view large images. Support image tiles/IIIF protocol (to stream from server directly). Tried IIPMooViewer (by IIPImage), but Openseadragon has much better UI/UX. Doesn't support annotations.
- Mirador. An image viewer built on top of Openseadragon. Supports annotations.
- Notebook support: No image viewer supports this out of the box, so they need to be developed. The amount of work could be significant as it requires understanding and adding features to the original image viewer.
- A good project to use as reference is Leaflet, which basically does what we are trying to do, but for maps. It supports annotations and has a popular notebook widget: ipyleaflet
Backend - 2 possible approaches:
- Use a jpeg2000 server to stream jp2 to client directly (open source options: freelib-djatoka, loris)
- Generate image tiles offline, using libraries like VIPS (haven't managed to get it to work yet) or maybe even kakadu.
Approach 1 is preferred because there is no need to pre-generate image tiles beforehand.
Open-source Backends surveyed so far The following are open-source jpeg2000 image servers surveyed so far. Theoretically they all support dynamic streaming of large images (jpeg2000 or image tiles) stored on the server (usually through the IIIF protocol), but I have yet to get any one of them to serve our jp2 images successfully. None of them have built-in support for image annotations.
|Server||Smooth streaming of tiles||JP2 support||Annotation||Setup/coding difficulty||Comments|
|aDORe-Djatoka||Yes||Yes||No||Didn't try (Java)||Project not maintained - no official website. Not recommended.|
|FreeLib-Djatoka||Yes||Yes||No||Low (Java)||I deployed this locally. Worked well on tiff image, but our jp2 didn't work. Author said it is an old project and we should consider alternatives.|
|IIPImage||Yes||Need Kakadu SDK||No||Medium (embed in fcgi-enabled servers)||I didn't have Apache/nginx on my machine so took me a while to set up. Turned out that jp2 doesn't work unless you build IIPImage with Kakadu SDK. So unless we can get that, this is not a viable option.|
|loris||Yes||Yes||No||Didn't try (Python)||I only tried deploying their docker image so far. But from what I have seen from the demos and discussions, Loris should support the image browsing features we need.|
|Image tiles (no server)||Yes||Yes||No||High - Need to use VIPS or kakadu to generate tiles||This is a flexible option, but we need to generate standard jpg image tiles offline before they can served. Some coding is needed to build the system to ingest our jp2 images and generate the tiles automatically.|
(Possible) Development work:
- Develop ipython notebook widget for openseadragon so that we can browse large images in ipython notebook.