art with code

2008-12-30

Filezoo current status ramble

I had a problem with determining the width of the thumbnail rectangle (it's a mess), but I can just use the thumbnail width for the rectangle instead of the current madness. I think. Maybe.

I'm now using the Gnome thumbnailer's large thumbnails (256x256) as "full size" thumbnails for files that are not jpegs or pngs (which are loaded as pixbufs, rendered onto a Cairo surface and scaled down to 800x800.) It makes the blurry thumbs a little less blurry, which is a little more like it. In a "it stabs your eyes a little bit less"-sense. Cairo's bilinear downscaling makes the 800x800 thumbs look horrible too, a better solution would be to recursively create a mipmap pyramid (scale down to nearest 2^n x 2^n, keep halving until n = 7) and use the nearest larger image to draw the thumbnail. Will cause extra 30% memory use for the large images though.

Large image memory use is.. I'm keeping 20 images cached. Maximum image size is 800x800 = 640000 pixels, 4 bytes per pixel = 2.56 MB. So 51.2 megs for a full large image cache.

Small thumb cache worst case is 260 megs, but it does prune it to 130 megs. Those numbers aren't very happy, I should do thumb cache pruning every N frames and throw all unused thumbs away. So that the thumb cache size approaches the visible set as a function of time. Cheap enough to reload more frequently.

Caching the FSEntries is a bit funny when the kernel pages your cache to the disk. It takes longer to unswap the cache than to recreate it for the visible set.

In the optimal situation I would have no cache and instead rely on the kernel's page cache and the programming language to have the filesystem data accessible fast. Walking a directory tree with 25000 files took 130 ms (in OCaml), so it's not free by any means. Perhaps fast enough to bother trying though? Would simplify my life.

The way how things now work is like this:
  • FSCache is a thread-safe cache that stores filesystem data [and the drawing model in a parallel tree.]
  • FSDraw.PreDraw walks the FSCache and tells it which parts of the drawing model to create. PreDraw also requests loading thumbnails and large thumbnails.
  • FSCache.ThumbnailCache stores normal thumbnails in an LRU cache with max size of 4000 thumbs and starts eliminating old thumbnails when it's larger than 2000 thumbs in size.
  • FSCache.FullSizeThumbnailCache stores the large thumbs and has max size of 20 and eliminates old thumbs when it's larger than 10 in size.
  • FSDraw.Draw walks the FSCache drawing model and draws it out.


PreDraw and thumbnail loaders each use their own threads, Draw is in the GTK event loop thread. The thread data flow is: (PreDraw | ThumbnailCache | FullSizeThumbnailCache | FileSystemWatcher) <-> FSCache -> Draw. The thumbnailer threads and the PreDraw thread fill out the FSCache in parallel. Draw and Click and FindCoverage and other GTK event loop functions read from the FSCache, with a possibly locking read when changing directories (locks if the FSCache doesn't contain the directory already.)

Writes to FSCache lock the whole cache. There is [usually] non-locking read access to the FSCache, and walking the drawing model is lock-free (apart from thumbnails.) The drawing model is always in a drawable state, but the sort and sizes of the files may be inconsistent with the UI setting (they are eventually consistent though.) Thumbnail access uses fine-grained locking: Draw locks the FSEntry with the thumbnail when drawing it, while DestroyThumbnail locks the FSEntry to destroy its thumbnails (drawing a destroyed ImageSurface causes a crash and ImageSurfaces need to be explicitly destroyed, as they don't play nicely with the GC (or I'm doing something wrong.))

The drawing walk takes around 7-10 ms for my home directory, likely more for nasty places like /usr/lib, and the time should be largely independent from activity in the other threads (given infinite hardware resources...) The uncached directory read lock can cause a pause, and should be timed.

Waiting on a thumbnail lock is uncommon on the drawing end, as the amount of code in the lock on the destroy thread is something like two assignments, two Dictionary deletions and two frees. Could time the worst case of 2000 thumbs waiting. Quick napkin math: guess 5000 cycles waiting per destroy, 10 Mcycles for 2000 waits, 20 milliseconds at 2GHz. Which'd give a worst-case scenario of "nasty framerate lapse."

Memory use is bugging me. I think I'll make the thumbnail caches have a "delete old" limit of 0, and see about minimizing the FSCache. While a Nautilus/PCManFM/whatever-like 35 meg memory use may be a distant dream with Mono and the libs eating 20+ megs already, <75 megs should be achievable. Or then I get to rewrite everything in Ada SPARK (yeah, right.)

No comments:

Blog Archive