art with code

2008-11-30

Filezoo, day 18


Today in Filezoo land, wrote a little priority queue for loading big thumbnails first and changed the file hash algo used to generate thumbnail names to MD5(file mtime + file size + first 16k of the file). It runs at a constant 0.2 ms if the file is in page cache. Compare to the previous MD5(file) and its 20 ms runtime for a 1.5 MB file.

Also slapped licenses to the source files. GPL for the app-specific parts, MIT for general use libs.

Then tried to find the source of random startup deadlocks, with no success. I think my, ah, homespun method for disposing dialog windows caused popup menu drawing to break at one point too, but I haven't been able to reproduce it after I cargoculted the code a little.

Oh well. Better go think it through.

2008-11-29

Filezoo, 17.3: image thumbnails


Brought to you by the ministry of silly hacks. The implementation's not very pretty, but it does thumbnail jpegs, pngs, gifs and bmps. Using convert. Stores the thumbnails in ~/.filezoo/Thumbs

Filezoo, 17.2: context menu, a bit nicer monitoring


Made the app poll subdir mtimes every second to do invalidation (FileSystemWatcher only watches one dir, turning on the subdir thing makes it hang on big dirs, maybe I should create several.) Also made a context menu with a couple commands in it. Thinking out loud, what might be a good set of commands?

For both files and dirs:
  • Delete (does mv file ~/.Trash)
  • Rename
  • Move to
  • Cut
  • Copy
  • Paste
  • Create Archive

For files:
  • Open File
  • Extract Here (for archives)
  • Run command on file [popup dialog]
  • Edit File (get list of mimetype-specific editors from .desktop files or something?)

For dirs:
  • Go to directory
  • Open Terminal here
  • Gitk [if there's a .git]
  • View as slideshow [if images in directory]
  • Open index.html [if such in directory]
  • Add to playlist / Play [like Winamp Explorer integration]

What else? Email file/dir to? Post to [insert website]?

Should read a list of filters and actions from a config file.
suffix=gif "View in GQView" "gqview $PATH"
entry.name=.git "Gitk" "gitk $PATH"
entry.mime=image/* "Slideshow" "slideshow $PATH"
entry.name=configure && entry.name=Makefile "Build" "cd $PATH && ./configure && make"

2008-11-28

Filezoo, 17.1: rewrite done-ish

Got the rewrite just about done and it works pretty well! The UI response times are generally below 100 ms, with some peaks in the 200 ms range. The big problem now is how recursive traversal became very slow and memory hungry. So I guess I get to move the recursive traversal to a more optimized system today. Then write a small session manager and config reader, and make a right-click menu for the files.

Let's see then... a new recursive traversal system that should work like du: have a dictionary of paths, put each directory's traversal results there, and the traversal results should be something like type result = { path : string, done : bool, subdirs : result list, size : long, count : long }. Space usage, maybe something like: 50 bytes path, 4 bytes done, 20 bytes for subdirs, 8 bytes size, 8 bytes count, total 90 bytes? Add in the pointers needed for the dict and it's closer to 110 bytes per directory. The tree I have has around 200 kdirs, so the memory use for the dict would be around 22 megs. Sounds okayish.

The other memory hole is the FSCache getting huge if you browse around enough. If you go through a million files, the memory use will be in the gigabyte range. LRU-pruning large subtrees would help. Stake out some total count limit and start pruning after breaking that.

Probably won't go all that far with the optimization work today, I'd like to do some useful stuff for a change :9

Filezoo, 16.1: rewrite, dev env screenie


Per request, here's what my development environment looks like. I'm using Kate as the text editor and compiling the code from the shell. I also have a browser window open for looking up docs and wasting precious time.

Kate is nice in that it has a list of open files, a filesystem browser, and the usual bunch of editor features (autocomplete names, indent by tab, unindent by shift-tab, comment out selection with ctrl-D.) I've no idea about Real Programmer's Text Editors. Or IDEs for that matter.

On the Filezoo front, got the rewrite just about done. It's working in a draw-loop blocking fashion at the moment, making it poll with a timer next. Then... tests? And a config manager? And new features?

2008-11-27

Filezoo, 15.2: DirStats static rewrite

Just finished the first pass of rewriting public class DirStats as public static class FSDraw. FSDraw does no filesystem access, no sorting and no layout creation. It runs through the tree of FSEntries given to it, and should have predictable performance.

What I will do next is disable parent dir drawing in DirStats to make its layout equal to the current FSDraw stuff, then write a simple FSCache that creates FSEntries for a directory. Then replace the CurrentDir.Draw in Filezoo with FSDraw.Draw. Small seamless changes...

2008-11-26

Filezoo, day 15: refactoring plan

The current codebase is a wee bit messy, things are confused. The rendering loop does filesystem operations and layout creation, both of which introduce drawing latency. And the renderer [DirStats] also does some UI logic, which makes things more confusing than they could be.

What I'd like to do is move all the blocking operations away from the rendering loop and do them in the background, making it possible to give a sort of limit for worst case performance. E.g. "I'm going to draw a maximum of 5000 rectangles per frame for this screen size and that takes 50 milliseconds. So if there's a free core available (set CPU affinity for bg threads different from rendering thread), the framerate should always be above 20 frames per second."

The road to a better design, something like this maybe?

FSCache -- filesystem model
- store files and directories
- store measurement info and sorted layouts
- do async traversal
- do async extra info fetches (thumbnails)
- fast lock-free read access
- manage watchers and invalidation

Implement by moving all FS-touching stuff, sorting and measuring from
DirStats to DirCache.

FSDraw -- renderer
- fetch data from FSCache and draw it to screen
- provide wanted draw depth information to UI
- provide click target information to UI

Implement by turning DirStats into a static class and splitting the drawing
process into a gathering pass and a drawing pass. The gathering pass is run
at UI action time and tells what the renderer would like to draw. The
drawing pass is independent and draws whatever info there is in the FSCache.

So there is a rendering traversal thread that does an explicit traversal of
the rendered part of the filesystem, and a drawing thread that draws the
current results of the traversal thread.

Filezoo -- user interface, controller
- all click action logic here
- guide traversal based on user actions
- call renderer when needed (expose, rendering traversal in progress)
- set up renderer drawing area transform

Main -- init, finalize
- load/save config and session data, parse command line options
- directory emblem patterns
- directory view settings

Yeah, it's going to be difficult

I got a directory info extraction thing going, but this stuff might be pretty hard. The reason I'd like to use Haskell is that it makes the drawing code easy, but I don't really have an idea on how to do the multithreaded traversal cache. In C#, it's somewhat the reverse. Maybe I should try rewriting the C# drawing code now to fulfill my daily quota of head-wall interaction.

2008-11-25

Filezoo, day 14


(hello.hs)


Took 4 days off, did some probing into the possibilities of porting away from C# into a language that doesn't have the trap known as OO modeling. I'm still not sure if I will, but here's what I found. Correct me if I'm wrong or you disagree.

OCaml doesn't seem to have Pango text drawing support, so it's either not an option, or I'd have to use something sane instead (how the heck do I get selectable text anyhow?) It's also a pain to compile anything on OCaml, though the compilation finishes fast and the type errors aren't indecipherable. OCaml's also familiar to me.

Haskell has Pango text drawing, though I don't know how I can cache the layouts for reuse. The compilation process is nice, very effortless. But GHC takes a long time to compile anything. Gtk2Hs starts as fast as C GTK2, which is great. I'm divided on the language. On the other hand it's functional and fast, on the other hand I don't know it.

C is fast. C has full library support. C works on everything. C compiles fast. C is familiar. C is so verbose that my hands hurt just from thinking about it. Not the actual code, perhaps, but the error handling. And you get to write your own data structures from scratch. And debug them.

So, why not C#? For starters, Mono takes 200 ms to start up. 200 ms takes an application from "starts instantly" to "starts pretty quickly." And it's away from your application code's init time, so you have to do more performance hacks. The slow startup isn't really that big a gripe though, since I'm planning to have the app always running in the panel.

The other complaint I have is the class library. It's doing a lot of work to do the wrong thing. The System.IO.Path functions throw an exception if they don't like the characters in your path. And everything dealing with files uses the System.IO.Path functions. So if you have, say, non-UTF-8 characters in your path, FileSystemWatcher throws an exception if you try to monitor that (perfectly valid and existing) path. And the type system doesn't notify you of exception-throwing functions, so you only discover the errors at runtime, and only if you test thoroughly at that.

As a language C# is like a Java. It's relatively verbose, but the type system does provide some safety. I don't much care for the way it gives me enough rope to hang myself with crap side-effect-riddled OO modeling. The numeric library is lacking arbitrary precision numbers and the libraries for basic data structures are inconsistent (Hashtable has Contains but Dictionary has only ContainsKey. Array has Reverse, String has not (I think in C# 3.0 there is an extension method for that though.))

[edit]
The good things about C# are the runtime and that it's very straightforward. The GC is good as far as I know and the threads are Real Operating System Threads. The standard library is extensive and doesn't usually try to be cute (i.e. doubles are doubles, ints are ints, uints are uints, and so on. Strings are a bit special.) And it's very much an Industry Language, with its Industry Features (like the localization stuff in strings and the localized number formatters.)
[/edit]

Conclusion? I think I'll do a quick straight port to Haskell, and, on noticing that it's not that easy, go back to C# and use the frustration to do push-ups. Sounds like a plan!

X application startup times

To find out the fastest way to create a window, I tested a couple different ways with small applications that create a window and exit. Here's a list of the results from fastest to slowest:
  1. Plain Xlib in C: 13 ms

  2. GTK in Haskell: 40 ms

  3. GTK in C: 40 ms

  4. GTK in OCaml: 60 ms

  5. GLUT in OCaml: 220 ms

  6. GtkGL in OCaml: 350 ms

I didn't time Mono and Gtk# this time, but extrapolating from the timings I did before, it'd take around 250 ms (200 ms Mono startup, 30 ms Gtk.Init, 16 ms frame delay to expose event.)

For a baseline comparison, process creation by forking takes 0.7 ms. And as these are graphical applications, they can only display on the next frame at the earliest, which happens after around 16 ms at 60 fps. So if an application starts in less than 16 ms, it's doing as well as can be realistically expected.

2008-11-24

More fishes


You can make shiny fishes with glitter pens. You can make shiny anything with glitter pens.

2008-11-23

Fishes

Filezoo startup profile

Timed the startup of Filezoo. First, 200 ms of mono overhead (version 1.2.6), then 100 ms of pre-draw init (30 ms Gtk init, 60 ms Filezoo instance creation, 16 ms from Application.Run() to expose handler firing), 200 ms drawing the toolbar for the first time (i.e. loading fonts), and 100 ms for drawing the first frame, for a total of 600 ms. Slow. The ls program takes less than 50 ms.

I'm most shocked by mono's startup cost being so huge. A Hello World takes 100 ms to run in mono, whereas in a compiled language (tested with OCaml) it takes 1 ms.

2008-11-22

Filezoo, start of day 13: some timings, thumbnail proto


Made a small prototype that draws thumbnails for PNGs. It's, hmm, I don't know. It works. But it's worse than grid layout. Compromise between tiny aspect-preserving list thumbs and harder-to-read grid :P

Mostly it just looks sort of cool, and that's all that matters in the end, eh?

Compared directory latency between a Konqueror, Nautilus and Filezoo by opening /usr/lib (3300 entries) from a running instance of each program. Opening the directory took around 1.5 seconds in Konqueror, 4 seconds in Nautilus, 0.2 seconds in Filezoo and 0.05 seconds with ls. Startup times for opening /usr/lib, from executing the application commandline to first frame drawn (hot cache): ~2s for Konqueror, ~5s for Nautilus, 0.8s for Filezoo and 0.05s for ls.

Some things to note: Konqueror and Nautilus draw icons and display about 30 files on the screen. Filezoo doesn't draw icons and displays 3300 files on the screen. ls doesn't draw icons, displays 3300 files on the stdout and has its effective performance limited by the terminal emulator.

Today I'll be going through the code and doing cleanup. Then, time permitting, I'll add selections and a right-click menu for acting on them. Tomorrow is a rest day.

Filezoo, day 12: zoom navigation


Put zoom navigation in, worked on making it a smooth experience, yet there is still relayout flicker at parts. Also added some more unicode icons and made clicking on current dir title reset the current dir view.

Wrote some todo plan, the main points being: Fetch rendered info asynchronously, with default placeholder info (e.g. "Loading...".) Make a worker queue for the async jobs. Make it possible to cancel async jobs. Cancel jobs when the requesting object is not visible. Cache the layouts in DirCache and invalidate with LastModified checks.

I'm still not happy with the navigation experience; it should have no flicker whatsoever, the animation should be smooth, and the zoom and pan should snap at sensible places. The context-sensitive click-to-zoom-out thing is also missing. As are pretty colored lines, etc.

Maybe tomorrow is a good day to do codebase maintenance.

2008-11-21

Filezoo, day 11: silly icons, less traversal, parent dir



Today, in chronological order:
  1. Drew a little concept of how the file manager should integrate with the desktop panel.

  2. Made name comparisons case-insensitive and changed the GUI font from Verdana to Sans (which is.. Bitstream Vera Sans on my machine, I think.)

  3. Worked around System.IO.Path.IsPathRooted blowing up on ISO-8859-1 filenames.

  4. Made a last modified measurer, it shows recently modified files large and old files small. Perfect for a download directory.

  5. Wrote a toplevel usage documentation for DirCache and added modification date and file permissions in the subtitle information.

  6. Added a parent dir reference to each directory. Yes, it's an actual window to the parent dir and lets you zoom there and loop through the filesystem until you run out of double accuracy.

  7. And made a silly unicode icon system, prefixing some predefined paths with unicode glyphs.


The rendering part is ok-fast again, but there are a few niggles: When a file change is detected, the current view completely rebuilds itself, which causes nasty-looking flicker and bolts the view to the top position, which is very annoying. Fixing the rebuild flicker and the view bolting are a priority. I'll also bring back changing view dir by zooming, already started writing the infra methods for it. Documentation project continues with filezoo.cs.

2008-11-20

Filezoo, end of day 10 / start of day 11


6am after no sleep, what better time to blog? Image from an old graphics hack I found lying on the hard disk.

Added importance annotations to the methods. The importance annotations are comments like /** BLOCKING */ and state the method's effect on the user interface.

A BLOCKING method has a non-trivial runtime (or at least lacks speed measurements) and is executed during the UI code (draw event handler or the interaction event handlers.) BLOCKING methods need to be profiled and latency-optimized to maintain a responsive user interface. The fewer BLOCKING methods there are, the better. Currently the draw loop and the recursive draw tree generation are BLOCKING, along with traversal cancellation and cache invalidation.

An ASYNC method runs its work in a separate thread from the UI code. The traversal threads and cache tree updaters are ASYNC. ASYNC code should be correct and not cause too much resource consumption that causes UI latency.

A FAST method is trivially fast or uses other FAST methods in a trivially fast manner (i.e. know that it's not going to infinite loop or anything.) A BLOCKING method measured to be fast can be changed to FAST. If a FAST method is called enough that it forms a bottleneck, its state should be changed to BLOCKING.

I also have DESTRUCTIVE and UNIMPORTANT importance levels, DESTRUCTIVE being reserved to filesystem-changing code, of which I have none yet. UNIMPORTANT is used for the profilers and other non-UI code.

Regarding string sorting, String.CompareOrdinal is faster than String.Compare, but not too correct.

Next up, hmm, more performance work in the shape of splitting the BLOCKING parts that touch the disk over to the ASYNC parts. Documentation for dircache.cs. And maybe at some point I can start adding new UI features? Like pretty colorful lines and magic ? Yeees?

2008-11-19

Filezoo, day 10 and 2/3: Manual rectangle clipping, cache refactoring

Cairo didn't like large coordinates for rectangles, so I'm manually transforming them to screen space and drawing. Which is fun? But necessary.

Rewrote the cache again. This time I found The Right Way to do it and it was remarkably painless. Now I have a single cache server that serializes all cache access and does all cache updates, and a bunch of traversal threads that send information to the cache server. Previously I had OO-modeled it badly with the cache entries passing on information updates to their parents and whatnot. Which was horrendously complex to make threadsafe, very difficult to understand and even more difficult to make work properly with traversal cancellations and cache invalidations. With the new way, I treat everything inside the cache server as single-threaded code and that makes things easy.

I also managed to make the cache invalidation more fine-grained from the previous "What? A file has changed?! Sound the klaxons, HÄLÄRM HÄLÄRM!!! DUMP THE CACHE, EMERGENCY DIVE!! LET'S REDO EVERYTHING!" Which was not too much fun with 1.2 megafile directory hierarchies. Now the cache tries to modify only those entries that were affected by the change, though that means that I get to deal with fun bugs :|

Last third of the day I humbly intend to spend on *drumroll* documentation! And cheers were heard all around!

Here's the rectangle helper for the heck of it. Been too long since I pasted random bits of code here anyhow.

public static void DrawRectangle
(Context cr, double x, double y, double w, double h, Rectangle target)
{
double x_a = cr.Matrix.X0+x*cr.Matrix.Xx;
double y_a = cr.Matrix.Y0+y*cr.Matrix.Yy;
double w_a = cr.Matrix.Xx*w;
double h_a = cr.Matrix.Yy*h;
double y2_a = y_a + h_a;
double x2_a = x_a + w_a;
x_a = Math.Max(-1, Math.Min(target.X+target.Width+1, x_a));
x2_a = Math.Max(-1, Math.Min(target.X+target.Width+1, x2_a));
y_a = Math.Max(-1, Math.Min(target.Y+target.Height+1, y_a));
y2_a = Math.Max(-1, Math.Min(target.Y+target.Height+1, y2_a));
w_a = x2_a - x_a;
h_a = y2_a - y_a;
cr.Save ();
cr.IdentityMatrix ();
cr.Rectangle (x_a, y_a, w_a, h_a);
cr.Restore ();
}

2008-11-18

Filezoo, end of day 9: requirements document, interactivity work


Filezooooooooo!

Ok, got the requirements document written in a very short form. It's slightly draconian, but that's good, right? No? At least the current codebase thoroughly fails at most of the points.

As good intentions go, I was going to work on solidifying traversal. But I guess doing pretty much nothing but traversal for two days took its toll and I ended up hacking on drawing instead. Specifically, I added a frame profiler that the drawing routine uses to figure whether it can do more in-frame work (directory layout and drawing traversal) or if it should try and get an incomplete image out before the user goes mad from waiting. It's pretty nifty but it's also a solution to a problem that shouldn't exist. The drawing model shouldn't be built in the drawing thread, but as an async response to actions in the event loop.

I also refactored a bit, moved the source files to src/ and wrote a build script (that does an AOT optimization pass too! Go Mono!) I added the above screenshots to the repository as well, for the sake of marketing. Maybe that's not such a good idea.

Tomorrow work on fulfilling some documentation requirements and tag parts with their relative importances. And a coordinate system overhaul. Yeah.

P.S. There's no BigDecimal for C#, so I get to stab myself in the knee and write code to jump seamlessly between directory-local coordinate systems. Such fun!

P.P.S. What should I use for unit testing?

Filezoo, start of day 9: more traversal banging, fixes

Traversal is a thorn in my side. Spent yesterday and today morning trying to make it not break when cancelling traversal, and I think it might work now (but how to tell?) So, now that I have that fiendishly difficult easy part out of the way, I can move onto monitoring changes in the filesystem and invalidating the cache based on them. For which I need a fool-proof cache invalidation algorithm. Heh heh, crying already.

Day 9 then. Today's plan is to write the requirements document and get the filesystem model really solid (which might as well take a week or more at this pace.) I would like to do performance work on complex trees (I just happen to have a directory that triggers a most fantastic 200 kfile layout + sort, which takes 5+ seconds), will have to see if I manage to steal some time off the other things.

Now for the 2 pm morning jog I go.

2008-11-17

Filezoo, day 8: Merge of the recursive branch


Branched the master into a recursive branch on day 6, worked on that branch and rewrote most of the drawing and traversal code. Yes, you may facepalm now. But I got out of it and the code works again, so I can't complain too much. Was banging my head against the wall with recursive drawing and traversal for two days though.

Now Filezoo has recursive drawing of the directory hierarchy with titles and all. Relayouting the tree is still clunky when using Count / Total size filters, and there are some nasty performance edge cases when browsing to huge monotonous directory structures (especially while their traversal is still in progress.)

I'm currently writing a requirements document for staking some goalposts for the application, and also to provide development focus. So day 9 will be typing the first version of requirements out. As that won't take all that much time, I'll also be working on performance and making the navigation more natural (it should work by zooming all the way so that there's no need to click on directories / breadcrumb entries.)

Full-scale zoom requires some hacks though, as Cairo can't handle large coordinates and you start running into accuracy problems with doubles at high zooms anyhow. If I find a nice way to use bignums, I'll use those. If not, I'll emulate them. Anyhow, need to do the zooming matrix math with bignums and project and clip to screen space doubles for drawing.

Not all that happy with the new layout, so will have to work on that too...

2008-11-16

Filezoo, day 6

Today was one of those days. Lots of work but very little in the way of results. Rewrote the drawing code some and banged my head against the wall with filesystem traversal. Day off tomorrow.

2008-11-14

Filezoo, end of day 5: Context-sensitive zoom, recursive drawing prototype



Day five of the Filezoo project turns towards the night. Wrote the traversal server, but it was a lot slower than spawning a bunch of ThreadPool work units. So I scrapped that idea. Made clicking on an unreadably small item zoom it to a readable size. Also wrote a prototype of the recursive directory drawing. It's awesome, but crappy. So I'm going to spend a while making it just awesome.

The user interface still lacks a couple of features that would increase the usability of the thing. For starters, there is no overview indicator to show the current scroll position. And zooming out is a bit of a bother (I did make it faster though), clicking outside to zoom out would be handy. The breadcrumb bar also manages to hide the most pertinent parts of the path, which, coupled with the lack of a "Go to parent dir"-button makes navigation impossible once deep enough. Zooming to change directories would be great.

Then on the file management features side - of which there currently are none - a good start would be to add selection and then build a small operation library on top of that. For executing operations on the selection, a context menu would be the tool of choice. Yeah.

Filezoo, start of day 5

The plan for day five... I was thinking of doing some user interface work, making the zooming more context-sensitive (i.e. zoom to the part I'm interested in.) Clicking on an unreadable entry should zoom in to the point where you can comfortably read it. Also, thinking of making the scroll wheel, well, scroll and relegate zoom to where the pan currently is.

In addition to that, recursive drawing of directories. Draw recursively until all the directory entries are smaller than the cut-off point. As the total amount of entries drawn per screen can't really exceed 5000 or so (the vertical resolution, she is limited!), the framerate should remain in the 20fps range. And yeah, only 20fps for drawing 10000 rectangles (2 per entry) is pretty pathetic, but one takes what one can get.

I'll try and write a preliminary traversal server as well, it shouldn't take more than an hour...

[edit]
Timed sorting a bit. Sorting an array of 4000 strings in C# took 130 ms for the first time, 60ms for the sorting the already sorted array. Sorting a list of 4000 strings in OCaml took 4 ms, 2 ms for already sorted. Interestingly, sorting an array of strings took 5ms in OCaml, which is more than what it took for the list sort.

Sorting an array of 4000 integers in OCaml took 2 ms, but only 1.08 ms in C#. So, my guess is that the C# string comparison operator is slow and that the integer sorting special case is highly optimized. So if I can do a partial string sort in terms of an integer array, I might get speeds closer to OCaml's 4ms.

2008-11-13

Filezoo, end of day 4: Speed!


Today on the Filezoo front: Got relayout down to 1 ms from 100 ms by eliminating unnecessary sorting (man, something's wrong when sorting a 3800-item array of strings takes 100 ms.) Drawing a frame of a 4000-entry dir is down to 50 ms from the previous 500-1000 ms. It still has the initial traversal latency, but I'm now masking it by drawing the first frame without the size-part of the titles. Which is good enough UI-wise, but doing a couple dozen directory traversals in parallel makes the disk cry.

The latency from clicking on a directory to completing the first frame is 60ms for my homedir (272 entries) and 180 ms for /usr/lib (3800 entries). Of the 180 ms, 50 ms goes to listing the directory contents, 80 ms is spent sorting the directory entries and 50 ms drawing. If I want to meet the <100 ms initial latency goal, I need to peel off 80 ms somehow...

But anyhow, speed! Tomorrow some UI work and drawing directory contents [finally!] And probably a second round of code cleanup.

In other news, ran 10 km for the first time in years. Been doing 4-8 km runs lately due to the existence of a nice 8 km loop. Had one 3 x 8 km week a month ago, so 10 km isn't really that much of a stretch. Next up: 15 km!

Filezoo: start of day 4

Moved to using Pango for text rendering, now my UTF-8 filenames render correctly. But it's slow. So the theme of the day is performance. From directory opening latency (creating a WaitCallback for a dir and adding it to the ThreadPool sometimes causes a 20ms pause, so when you have lots of dirs, you get to wait a second or more), to crappy overview drawing performance (/usr/lib has only 3380 files in it, and it's already a pain to navigate), to crappy panning performance (need to skip clipped entries altogether.)

For the ThreadPool pauses, I should make a filesystem traversal server that has a hashtable, a request queue and a worker thread. When a dir needs its total size and count, it asks the traversal server, which returns the current result immediately, and puts the dir to the queue if it's not already completely traversed. The worker thread then takes a random entry from the queue and adds the file sizes of the entry to the corresponding dir in the hashtable along with the dir's ancestors. If the dir has no subdirs, the worker thread marks it as completed and walks up its ancestors, marking each completed if all its subdirs are completed. If the dir taken from the queue has subdirs, the worker thread adds the subdirs to the queue.

Ok, sounds like enough work for the day.

Tried to make a Gtk user interface using the GUI designer but failed, and the GUI designer was a huge bother of point-and-click-UI. And since I want a certain visual look, it's well nigh impossible with the normal widgets. Oh, if making GUIs worked like HTML or SVG...

2008-11-12

A small shell script for Googling

google_raw.rb works like this:

$ google_raw -n1 euribor
Euribor Homepage
http://www.euribor.org/
Euribor (Euro Interbank Offered Rate) is the rate at which euro interban
k term deposits within the euro zone are offered by one prime bank to ano
ther prime ...Historical DataAbout EuriborAbout EoniaPress ReleasesWhat's
newFAQsPanel BanksTechnical FeaturesMore results from euribor.org??

$ google_raw -n1 ピース
ピース - Wikipedia
http://ja.wikipedia.org/wiki/%E3%83%94%E3%83%BC%E3%82%B9
出典: フリー百科事典『ウィキペディア(Wikipedia)』. 移動: ナビゲーション, 検索. ピース. Peace(ピース)は、英語で「平和
」を意味する。 Piece(ピース)は、同じく英語で「欠片」を意味する。 ピース ( タバコ) - 日本の煙草の銘柄のひとつ。 ...

And a google shell script to page the output:

google_raw $@ | less

[edit] Thanks to Anonymous for the $@ tip.

Coupled with URxvt's URL matcher, I can google stuff from the shell and click on the links to open them in Firefox. It's not all that useful though, since one tends to have a browser window open at all times anyhow.

Filezoo, end of day 3: open terminal, threads



GitHub page

Implemented "Open Terminal" and managed to get more responsive panning behaviour in huge dirs like /usr/lib (was doing a relayout on every UI event instead of on every redraw, which spammed several relayouts on every mouse drag), and proceeded to piss away the performance gains with increased dir opening latency from launching a bunch of worker threads to do the recursive directory traversal.

Opening a terminal window with the filename already in it appeared to be bothersome, so didn't work on it apart from reading the bash man page. Could use a Gtk text input widget instead.

Split the measurers, comparers and zoomers from filezoo.cs into their own files and rearranged the source a bit. The custom widgets are getting nasty to manage, need to look into replacing them with Gtk widgets or XAML or something wacky like that. The text should be drawn with [something that uses] Pango at least, now there's no missing glyph fallback, and I get to enjoy filenames like "[][][][][][][][].jpg". I'm not familiar with this stuff, suggestions?

The performance still sucks for large dirs, even when zoomed in (need to profile to see if it's from too much draw or from too much layout), so that's a good project for day 4, along with fixing the initial dir opening latency. If it takes longer than 0.1s to draw out the first frame of an opened dir, it's too slow.

In other news, tried out Eagle Mode and it's awesome. Reading text, PostScript and PDF documents inline is really nifty. However, I don't like the theme and the massive waste of screen space for trivial information, especially the file info next to the contents box. 10% of directory box space taken to tell me that the directory is a directory. Oh come on, really?

Eagle Mode also has a raytraced chess game, which takes it beyond awesome into the rarefied sphere of retarded giggling at the sublime beauty of the universe and all that is in it. And it's responsive too!

And that's it for day 3, more tomorrow.

P.S. Is there something like NaNoWriMo for writing applications? "One month, 6000 lines of code!"

Filezoo, start of day 3: zooming and panning



Got [uniform] zooming and panning done. Non-linear zoom would be an interesting exercise, but there are more pressing things to do. So, the todo for the rest of the day:
  1. A button "Open terminal here".

  2. For files, a button "In terminal" -> opens terminal window with "prompt$ [] file_name_here".

  3. Separating the drawing, UI and directory traversal into their own threads.

  4. "I'm doing something that's going to take forever"-spinner for operations that so do.

  5. Partial layouts for showing the status of abovementioned operations.

  6. End-of-day refactoring.



Furthermore, I have this crazy idea that the application and its constituents should be simple single-function programs, which call each other to do their magic. And to take that to the UI level, the UI should consist of small reusable widgets that can be combined to do more complex things.

For example, the file browser view for a single directory should be like let browser dir = ls dir | zoomable -window Filezoo -action:Directory browser -action:File open. And if you wanted to use that to read your mails: let mailer dir = ls ~/mail/$dir | zoomable -window Mail -action:Directory mailer -action:File read_mail. Simple programs, defining a data source and behaviour.

To detour into a parallel, the HTML A- and IMG-tags define behaviour. The src-attribute of the IMG-tag defines a data source. (And if TABLE worked like IMG, the world would be a better place; <TABLE src="stuff.csv"/> and <TABLE src="stuff.xls" width="500" height="400"/>. Why not <IMG src="stuff.csv" width="500" height="400" style="table-style: bar-graph;"/> as well.)

2008-11-11

Filezoo: end of day 2



Ok, got the sort and size widgets done, along with small dotfiles and the refactoring. Didn't get to zooming and showing dir contents yet. Guess I have something to do tomorrow as well.

The git repo is at http://github.com/kig/filezoo/tree/master

Filezoo: visual du with aspirations towards file management


Continuing from yesterday, added a click handler for navigating the directory tree and opening files with gnome-open, and fiddled around a bit with the layout. Currently it works as a very bad filesystem browser, which is what I aim to change today. So, onto the plan!

My plan for today consists of five items:
  1. Sort the directory entries by size or name, and have a button to switch between the two.

  2. Make dotfiles appear small when not scaling by size.

  3. Scale directories by the [logarithm of] directory tree size in files.

  4. Zoom with the mouse.

  5. Show directory contents.


I've done prototypes of the first three, so now I need to make them work together nicely. To draw the listing, I need to get the directory entries of the current directory, then compute the relative size for each, sort them, and finally add the parent dir link at the top (unless the current dir is the root dir.) Then, to do zooming with the mouse, I need to further scale each entry at the drawing phase by a factor gotten from the zooming function (some kind of a gaussian hump, I think), which needs to have the definite integral from 0 to 1 amount to 1.

Showing directory contents can be done with limited-depth recursion.

Ok, so, this is probably how I should refactor the code:

let buildDirInfo dir = map fileInfo (ls dir)

let buildLayout files scaler zoomer =
let fileScales = map (tupleWith scaler) files in
let totalSize = sum (map fst fileScales) in
let fileScales = map (fun (s,f) -> (s / totalSize, f) fileScales in
snd (mapAccum (fun acc (s, f) -> (acc+s, (s * zoomer acc, f)) ) 0.0 fileScales)

let stack_do cr f = Cairo.save cr; f cr; Cairo.restore cr

let draw cr scaler zoomer files level =
match level with
| x when x < 0 -> ()
| x -> let layout = buildLayout files scaler zoomer in
stack_do cr
(fun cr -> iter (fun (s, f) ->
drawFile cr scaler zoomer s f level;
translate 0 s) layout)

let drawFile cr scaler zoomer scale file level =
stack_do cr (fun cr ->
drawFileModel f scale;
if (file.type = directory) then
draw cr scaler zoomer (entries file.path) (level - 1))

let fileInfo f = {
name = basename f;
dirname = dirname f;
path = f;
type = fileType f;
size = fileSize f;
recursiveSize = lazy computeRecursiveSize f;
recursiveFileCount = lazy computeRecursiveFileCount f;
permissions = filePermissions f;
}

2008-11-10

Visual disk usage app using Mono


Continuing on my epic quest to learn .NET and GUI dev with Mono, I wrote today a small du-clone that renders the result visually using Cairo. As you might guess from the color scheme, I'm thinking of integrating it into my desktop theme as a file manager (hey, what good is a theme if you have to use apps that don't conform to the look...)

This one has a GitHub repo as well, http://github.com/kig/filezoo/tree/master

The next thing I'm going to do is add a click handler that navigates the directories and calls gnome-open on the files.

2008-11-09

A simple C# analog clock with Mono and Cairo



In the previous post I said that I had to write my own analog clock app to get the look I wanted. Originally, I had written it in Ruby, using the Librend library, which drew the clock on an OpenGL texture using Cairo, and then displayed it. Today I thought to learn some C# and Gtk#, and rewrote the clock as a Mono application that draws in a Gtk window using Cairo.

I set up a GitHub project for the clock, it's at http://github.com/kig/simpleclock/tree/master

The source code is a documented 200 lines, so it's no big deal to read if you're interested. I'll walk through the perhaps most interesting bit below; the clock transform, which makes it simple to draw the clock parts.

To understand the clock transform, you first need to know how Cairo's coordinate system works. In Cairo's default transform the origin (i.e. 0,0) lies at the top-left corner of the canvas, Y grows down, and rotation increases clockwise from the right. In other words, it's the math 2D cartesian coordinate system with the Y-axis flipped.

What we want for the clock is to have the origin at the center of the clock face with rotation increasing clockwise from the top (i.e. 12:00.) That way, we can simply rotate by e.g. 2π(minute / 60) to draw the minute clock hand. Furthermore, we want to normalize the coordinates to run from -1.0 to 1.0 for the clock area, so that we can trivially scale the clock to different sizes without changing the drawing code.

Here's how I achieved that:

uint boxSize = Math.Min (width, height); // size of the clock box

// First, we center the clock box to the window.
cr.Translate ((width - boxSize) / 2.0, (height - boxSize) / 2.0);

// Then we scale the box so that -1.0 .. 1.0 spans the whole box.
cr.Scale (boxSize / 2.0, boxSize / 2.0);

// And move the origin to the center of the box.
cr.Translate (1.0, 1.0);

// Finally, rotate CCW by 90 degrees to make rotation 0 point up.
// We don't need to flip the rotation direction, because angle grows
// clockwise in the "Y grows down" default transform.
cr.Rotate (-Math.PI / 2.0);


Now drawing the hands is quite trivial, as you can see from the DrawMinuteHand -method:

void DrawMinuteHand (Context cr, uint minute)
{
double rot = (double)minute / 60.0; // => rot is between 0.0 and 1.0
cr.Save ();
cr.Rotate (rot * Math.PI * 2.0); // 2π(minute/60)
cr.Rectangle (0.0, -0.05, 0.8, 0.1); // draw the hand model
cr.Color = new Color (0, 0, 0);
cr.Fill ();
cr.Restore ();
}


To sum up my experiences from writing this little C# project: compiling programs with the Mono compiler is pretty nifty (not Ada-level nifty though, more like OCaml-nifty) and C# is straightforward enough. I was pleasantly surprised that I managed to get the program written in a couple hours :)

2008-11-08

Black on white desktop theme

What with the winter coming and all, I felt like it was the time to do a bit of a refresh on my desktop.

Before, the scheme was light gray and cyan on dark gray.



Now it's black on white.



Using FVWM2 for the window manager and the panel. The drop shadows are courtesy of xcompmgr. Drew the window icons a few years back. Had to make a custom analog clock (using Cairo) after failing to find a good one. Urxvt for the terminals, using its matcher-module for clickable urls. The irssi theme is a customized version of Simplicity.

There are no launchers, app icons or a taskbar. I have the F-keys set to launch apps (MacOS 9 -style) and there's a right-click-summonable desktop menu. I don't really need a taskbar, since I have the pager and Win-[number] hotkeys for desktop switching. To shuffle through windows, a right-click on the window border or titlebar moves it back / brings it to front. I'm using the mouse to switch between windows (with focus follows mouse) and keyboard shortcuts to do the maximize/winshade/close/sticky -tango. No minimize because no taskbar.

All my windowing-related hotkeys are behind the Win-key, leaving the other modifier keys free for apps. Now if apps could just stop using the F-keys :P

Also made Firefox work more like Chrome by installing the following extensions: Hide Menubar, autoHideStatusbar, CyberSearch and Edit Middle. Then hid the bookmarks toolbar and the search bar, replaced the Home-button with New Tab, and added the Bookmarks-button to the Navigation toolbar. Now I have only a single toolbar and the tab list on top, no separate search bar and a statusbar that only shows when loading page, hovering a link or hovering the statusbar area. See screenshot below.

Blog Archive

About Me

My photo

Built art installations, web sites, graphics libraries, web browsers, mobile apps, desktop apps, media player themes, many nutty prototypes, much bad code, much bad art.

Have freelanced for Verizon, Google, Mozilla, Warner Bros, Sony Pictures, Yahoo!, Microsoft, Valve Software, TDK Electronics.

Ex-Chrome Developer Relations.