art with code


Filter Bubbles

Filter bubbles are the latest trend in destroying the world. Let's take a look at how one might construct this WMD of our time and enter a MAD detente with other social media aggregators.

First steps first, what is it that we're actually building here? A filter bubble is a way to organize media items into disjoint sets. A sort of sports team approach to reality: our filter bubble picks these items, the others' filter bubble picks those items. If a media item appears in multiple filter bubbles, it's called MSM (or, mainstream media.)

How would you construct a filter bubble? Take YouTube videos for an example. You might have a recommendation system that chooses which videos to suggest for watching next. Because more binge-watching equals to more ad views on your network, increasing the amount of ad money coming your way, which lets you buy out competition and become the sole survivor. At the same time, time spent on YouTube equals to less time spent on other ad networks, which makes YouTube ads more valuable and further increases the amount of ad money coming your way. So, recommendation system for binge viewing it is!

Suppose you watch a video about the color purple. The recommendation system would flag you as someone who's interested in this kind of stuff (hey, you watched one.) So it'd go and check other people who also watched that video and try to find some commonalities. Maybe 60% of them also checked out a video about the color red. The red video would score high on "watch next". Suppose that 10% of the purple-watchers also gave a thumbs down to a video about the color blue. The recommendation system would avoid showing you the blue video because you might not like it.

So you go and watch the red video. Now, red video viewers might have very strong opinions about blue, and 20% of them vote down blue videos. Some of them also voted down purple videos. The recommendation system would now know not to show you blue videos under any circumstance, and steer away from purple videos as well. On the other hand, the red video viewers quite liked some extremist red videos that dove deep into the esoteric minutiae of the color red. Might be something that pops up on your watch next list, then.

You go and watch an extreme red video. Suppose the extreme red viewers have started disliking more mainstream red videos, not to mention their great dislike for blue and purple videos. Now the recommendation system avoids showing you blue, purple and mainstream red, and populates your watch next list with the purest shade of extreme red.

Welcome to the filter bubble.


The extreme red videos here are an example of an attractor. If you think of the recommendation system as a vector field that guides the viewer in the direction of the recommendation vectors, the extreme red topic would form a sort of a black hole. Topics around it have recommendation vectors that point towards extreme red, but extreme red doesn't have recommendation vectors that point out of it. Once you enter the topics that surround extreme red, there's a high likelihood that you get sucked into it. If you don't get sucked into extreme red, the company would regard that as a failing of their recommendation system and devote time and effort to improve its capability to suck you towards extreme red.

Attractors are special topics. Special in that they make people inside the attractor pull more people into the attractor and prevent their escape. Otherwise they'd be more like regular popular topics: you get drawn into a popular topic, but there's always an escape route towards another popular topic. To make an attractor, the content in the attractor needs to promote behavior that blocks escape from the attractor. For example, an extreme red video that says that mainstream red, purple and blue are all paid shills plotting to destroy the world would call its viewers to vote down other views.

Breaking filter bubbles

If there's an attractor in your recommendation vector field, scramble it. Mark that topic as something where all the watch next links go to far away places or even to places preferentially disliked (i.e. stuff that the group dislikes more than whole population) by the people in the attractor. Reduce the ad payout to content near attractors. Decrease the recommendation weighing of attractor neighborhoods so that escape is more likely.

Create legislation to warn people of viral attractors. Require explicit user consent to apply binge-inducing user engagement systems. Ban binge-inducing products from public spaces and require binge-inducing sites to post warning signs with pictures and cautionary tales of addicts who had their lives ruined by Skinner boxes.


Exchange rates

If you're looking at nominal GDP figures for the last few years, you'll notice a weird thing. Most of the world seems to be in a slump. Canada's GDP has decreased by 20%, UK's GDP has decreased by 15%, Germany's GDP is down by 10%, Japan, same story. Even fast-growing economies like South Korea, Armenia, Vietnam and India have stalled at their 2014 levels.

There are a few exceptions though. The US is still growing as normal. China and Hong Kong as well, though China's had a slight dip in its growth rate. Not to be outdone, Grenada's growing at a good clip, ditto for other East Caribbean states.

What's going on? Here'a graph that explains things:

The USD has appreciated like crazy against other currencies over the last couple of years. If your currency is the USD or fixed to the USD, everything seems to be as usual: economic growth is steady, things are normal, all is well. Imports from other currency areas are cheaper, but your exports are getting more expensive.

If your currency is tracking the US dollar, but isn't fully fixed to it, you'll see something like China. A slowdown in growth, exports get more expensive, imports are a bit cheaper. Enough to get some downcast news articles going on.

If your currency is floating free against the dollar, the sky is falling. Your GDP has just crashed by 20%, your economy is shrinking like a dried grape, the good times are behind you and all that's left is a grim meathook future where you hunt cockroaches and scavenge carrion left behind by the radioactive mutant wolves. On the plus-side your exports are cheap and popular.

But, well, this has happened before. Here's the chart from 2000-2002:

And here's what happened over the next two years.

What goes up, must come down. Another example of this was around the 2008 crash. First the USD depreciated 25% against the euro over two years. Then the sub-prime crisis hit, and the USD appreciated 25% over two years.

Exchange rates go up and down. There are usually some reasons for why, but in the long run reversion to mean takes over. If the USD gets too expensive due to policy differences, either the US changes its policy to make the USD cheaper, or other economies change their policies to make their currencies more expensive.

Even if there is some real "the entire US economy figured out how to 1.5x the productivity of a person, therefore the rest of the world is doomed"-thing going on, it's not a lasting effect. The rest of the world is going to do the same thing. If you've got super good 3D printers and robots and AI and automated manufacturing and design to make the 300 million people in the US as productive as 1.4 billion Chinese, what's to prevent the Chinese from using the same tech and becoming as productive as 6 billion non-augmented Chinese. If you force-multiply people, you force-multiply people.


How's the EU economy doing (part 2)

Europe's not doing so great. Eastern Europe is 10x poorer than the US, Bulgaria is even poorer than China by nominal GDP per capita. Even rich countries in Europe are poor compared to the US, with just 70% of the nominal GDP per capita.

Wait, what? Didn't you just write yesterday that everything is fine, don't worry? Welcome to the land of currency exchange rates. The US dollar is valued at levels unseen since the early 2000s, so suddenly everyone else gets thrown into the poorhouse. A few years earlier, the dollar was valued a third lower, so the story was that EU was an economic juggernaut, eclipsing the US economy. Now the USD/EUR exchange rate is nearly even, and the story is that the US is an unstoppable economic powerhouse, and the EU's economic policy is one of total and utter failure.

What's going on then? Exchange rate fluctuations? One day you're rich, the next you're poor, the day after tomorrow you're rich again.

There are a couple historical analogies to this situation. Both helpfully involving the US as one of the economies involved. First time around, there was the Soviet Union. A serious economic competitor to the US. The USSR had a population 20% larger than the US, but it was culturally fragmented and started off significantly poorer, so it never really reached the total economic bulk of the US before it got sideswiped by Japan into the third place in the late 1980s. Japan was clocking massive annual growth rates at that point and overtook the Soviet Union, despite having only half the population. And Japan kept on growing. At the peak of the Japanese bubble, Japan's GDP per capita was higher than the US one and there was all sorts of crazy talk of the Japanese buying up the entire United States and becoming the largest economy in the world by 2005. Then the bubble burst and Japan was pushed to the third place by the new-born European Monetary Union.

So, is this history repeating itself? Is the EU in a valuation bubble, which burst during the euro crisis, and now the Union is going to disintegrate and disappear from the world stage, much like USSR and Japan? While the situation is a bit similar to the one with the Soviet Union, as both the EU and the USSR are culturally and politically fragmented economies that started off poorer than the US, there are some crucial differences. First off, the population of the EU is about 50% larger than the US, making it harder to overtake without veering into bubble territory. Second, a significant chunk of the EU is made up of historically rich countries, who should know how to manage an economy. Third, the EU GDP per capita has never exceeded that of the US, so it's less likely that we're experiencing a wild valuation bubble bursting. The euro crisis did have a valuation bubble involved in it, but it was confined to the newly-rich fast-growing South Europe and Ireland.

Taken together, these points make it less likely that we're witnessing a USSR / Japan -style situation where the other economy crashes, the US recovers from a slump, and returns to being the biggest economy. In fact, it feels a bit like the Japanese situation, but with the US in Japan's position. You've got a tightly integrated economy with a smaller population overtaking a more loosely-bound economy with a significantly larger population.

At the same time, China has passed both the US and EU to become the largest economy by GDP PPP (and a nominal number one in a few years). At 3x the population size of the EU and a faster-growing economy, China's going to be number one with a comfortable margin for a good while.

With that in mind, you could also think of this as a larger-scale re-enactment of the 1990s. In that scenario, the US would pass the EU in GDP to retake the number two position after China. The resulting shock would cause the EU to disintegrate USSR-style while the US surfs an asset bubble. In a few years, some "let's create a counterweight to China" economic union comes online, overtakes the US and the US asset bubble bursts.

So you've got a weird situation. On one hand, the US and EU are nominally the number 1 and 2 economies in the world, waiting to be passed by China. On the other, they're numbers 3 and 2 by PPP. This is playing out in both empires crumbling in various ways, instead of the more usual "number 3 disintegrates as number 1 and 2 turn on it." Brexit pulling the UK out of the EU, Canada-EU trade deal pulling Canada towards Europe, US talk about breaking up NAFTA, SE Asia pivoting away from the US, EU neighbors dropping plans to join the Union, California fringe movements lobbying for Calexit, and a Eurasian pivot towards China.

How's the EU economy doing?

Things are going well in Europe. The difference in GDP per capita (PPP) between rich and poor EU countries is shrinking. The average EU GDP per capita (41k) is somewhere around South Korea (39k) and Japan (39k).

The high-GDP EU countries are grouped around 45-50,000 PPP dollars per capita, with a few outliers. Starting from the top, Luxembourg (102k) and Ireland (69k) are a bit special: both are tax havens, which inflates their figures significantly by routing a lot of money through the economy with very little sticking around. Up next is Netherlands (51k), also a bit tax havenish. Then we get to the rest of the list, composed of Sweden (50k), Austria (48k), Germany (48k) and Denmark (47k). Belgium (45k) rounds up the list.

The medium-GDP EU countries are in the 30-45,000 PPP dollar range, and compose the majority of the EU. First, we've (still) got UK (43k), nearly tied with neighboring France (42k) and far-flung Finland (42k). Malta (38k), Spain (36k) and Italy (36k) form another closely knit group. Rounding up the 30k+ group, we've got Cyprus (34k), Czech Republic (33k), Slovenia (32k), Slovakia (31k), Lithuania (30k) and Estonia (30k).

We can (and probably have to in a few years) extend the middle-income group with some fast-growing countries falling just short of the 30k divider: Portugal (29k), Poland (28k), Hungary (27k), Greece (27k) and Latvia (26k).

The sub-25k category is composed of new member states. Led by newcomer Croatia (22k), fast-growing Romania (22k) and Bulgaria (20k) finish the list.

To understand this number mess a bit better, let's do a historical comparison. Bulgaria is now at a similar GDP per capita level as the UK was in 1994 (or Sweden in 1991). Poland and Greece are like the UK was in 2001. Czech Republic is like the UK was in 2005. Italy and Spain are like the UK was in 2010. Today's Germany is like the UK might be in 2020. The Netherlands is like the UK in 2026. The EU as a whole is at a similar GDP per capita level as the US was in 2004.

Ten years ago in 2006, the difference in GDP per capita between the Netherlands and Bulgaria was 3.2x. Now the difference has come down to 2.55x, as Bulgaria's economy has caught up with the rest of the Union.

Compared to the US, the EU economy is oscillating at the usual 68-70% range per capita. The exceptional performance of the US economy is weird as usual, it's hanging out with petrostates, tax havens and finance-driven city-states, while having a large population. How does that happen? Is it real? How to replicate it? Who knows.


Brush stroke blending

Brush stroke blending is somewhat of a black art. Which is a shame, since Photoshop's been doing it for more than 15 years now.

Lemme try and explain.

Suppose you've got a pressure-sensitive drawing tablet where the pressure controls the opacity of the brush. If you do a low-pressure stroke, you'd like to have a flat surface of color with roughly the same alpha everywhere (say, alpha 0.5 for half pressure). If you use the usual source-over blend, each of the brush stroke segments would add to the alpha since it's src.a + (1-src.a) * dst.a. As the brush stroke intersects with itself, the alpha builds up all the way to 1. Which is not what you want if you're trying to paint an alpha 0.5 surface.

For a brush stroke with fixed alpha, this would be no problem, you could just vary the blend opacity of the stroke layer. But if you want to have varying alpha inside the stroke layer, you need a different blend. What I was using for hard-edged brushes (and what my Drawmore Touch prototype is using) is MAX blend for the alpha: max(src.a, dst.a). If an alpha 0.5 brush is stamped over a previous alpha 0.5 brush stroke pixel, the result is going to be alpha 0.5. This prevents the stroke layer from accumulating opacity above the brush opacity and makes it possible to do smooth surfaces using pressure-controlled opacity.

But it's broken for soft brushes. With soft brushes you'd like to paint a smooth surface with a soft edge. If you do alpha max, the brush intersections have cross-shaped blending artifacts and it's very difficult to do a smooth surface (you can try with the Drawmore thing: drag left on the brush 100% control to make it 0% hardness, then try to paint a smooth surface. No can do.) GIMP & Krita probably suffer from this too, Photoshop does something more magical.

What I've got in ShaderPaint is my latest attempt at solving this. It does brush stamping and alpha max mixed with source-over clamped to current stamp alpha.

void strokeBlend(vec4 src, float srcA, vec4 dst, out vec4 color)
    // Source-over blend for non-premultiplied alpha.
    color.a = src.a + (1.0-src.a)*dst.a;
    color.rgb = src.rgb*src.a + dst.rgb*dst.a*(1.0-src.a);
    color.rgb /= color.a;

    // Saturate color alpha to brush stamp max alpha.
    // For hard brushes this should be roughly the same as max(dst.a, src.a).
    // For soft brushes, the stroke accumulates alpha up to the brush stamp alpha,
    // which results in a flat stroke area with a smooth edge.
    if (color.a > srcA) {
        color.a = max(dst.a, srcA);

The brush stamping shader is pretty simple. You pass it the last stamp position and the current mouse position (& use them to calculate the last stamp position for the next line segment). The shader then steps along the last stamp -> mouse position -vector with the brush spacing offset. For each of the brush stamp points, accumulate brush color with the in-stroke blending function.

strokeDirection = normalize(strokeDirection);
float stampSeparation = brushRadius * 0.5;
float stampCount = floor(strokeLength / stampSeparation + 0.5);

for (float i = 1.0; i < 200.0; i++) { // Max 200 stamps per segment.
    if (i > stampCount) { // Break once we're done with the stamps.

    // Distance from circular brush.
    float d = length(fragCoord - (lastPoint + strokeDirection*i*stampSeparation)) / brushRadius;

    if (d < 1.0) { // The pixel is inside the brush stamp.
        vec4 src = currentColor;
        // Create a soft border for the brush.
        src.a *= smoothstep(1.0, hardness * max(0.1, 1.0 - (2.0 / (brushRadius))), d);
        strokeBlend(src, opacity, fragColor, fragColor); // Blend the brush stamp into the stroke layer.

Dunno if there's a nicer solution, given the messiness of the blending function.


Easy 3D on the web

Problem: can't do 3D graphics on the web. Solution: WebGL. New problem: no, I mean, I want to put this 3D model onto a web page. Solution: Sketchfab / Three.js / Babylon.js / ShaderToy. New new problem: I need to download libraries and code stuff or host the files on a SaaS or develop a third brain to model using modulo groups of signed distance fields moving along Lissajous curves.

Could easy 3D be part of the web platform? And how? With images, the solution was simple. The usual image file is a 2D rectangle of static pixels and all you really needed to figure out was how to layout, size and composite it with regards to the rest of the web page. When you step outside of simple static 2D images, all hell breaks loose.

Animated GIFs have playback timing and that's not so easy to figure out, and so animated GIFs are pretty broken. Videos need playback controls with volume and scrubbing, potential subtitle tracks, and a bunch of codecs both on the video and audio sides, so they were pretty broken as well. (Then YouTube started using HTML5 videos because mobiles don't do Flash. And magically the video issues were fixed~)

SVGs also need to handle input events and have morphed from "umm, put this in an embed and it'll maybe work" to their current (pretty awesome!) state where an SVG can be used as a static image, an animated image, embedded document and an inline document. Something for everyone!

Displaying a 3D model on a web page is a bit like mixing SVG with video elements. You'd like to have controls to turn the model around - it is 3D after all. And you'd like to have the model animate - again, it's 3D and 3D's good for animations. It'd be also nice to have the model be interactive. I mean, we were promised a 3D data matrix by a bunch of fiction writers in the 80s, and that's going to require some serious animated 3D model clicking action (to be fair, they also promised us a thermonuclear war, but let's not go there right now.)

So. 3D model. Web page. <3d src="castle_in_the_sky.3d" onclick="event.target.querySelector('#gate').classList.add('open')"> Right?

What file format do you standardize on? How do you load in textures and geometry? How do you control the camera? How do you do shaders for the materials? How do you handle user interaction? What are the limits of this 3D model format? How do you make popular 3D packages output this file format (plus animations, plus semantic model for interactivity)? How do you compress the thing and progressively stream it?



I wrote a small painting program on ShaderToy, using the new multipass rendering feature. It's called ShaderPaint and it was a lot of fun to write.

The fun part about writing programs in ShaderToy is the programming model it imposes on you. Think of it as small pixel-size chunks of memory wired together in a directed graph. Each chunk of memory runs a small program that updates its contents. The chunks are wired together so that a chunk can read in the contents of chunks that it's interested in. The read gives you the contents on the last time step, so there's no concurrent crosstalk happening.

This is... rather different. Your usual programming model is all about a single big program, executing on a CPU, reading and writing to a massive pool of memory, modifying anything, anywhere, at any time. To go from that to ShaderToy's memory-centric approach is a big shift in perspective. Instead of thinking in terms of "The program is going to do this and then it's going to do that and then it's going to increment that counter over there.", you start to think like "If the program is running on this memory location, do this. If the program is running on that memory location, do that. If the program is running on the counter's memory location, increment the value." You go from having a single megaprogram to an army of small programs, each assigned to a memory location.

In the figure above, I've sketched the data flow of an output pixel. First, the UI inputs coming from the shader uniforms modify the UI state layer, which is read by the stroke layer to draw a brush stroke on it. The stroke layer is then composited with the draw layer when the brush stroke has ended. The final step is to composite the stroke layer and draw layer onto the output canvas, and draw the UI controls on it, based on the values on the UI state layer.

About Me

My photo

Built art installations, web sites, graphics libraries, web browsers, mobile apps, desktop apps, media player themes, many nutty prototypes, much bad code, much bad art.

Have freelanced for Verizon, Google, Mozilla, Warner Bros, Sony Pictures, Yahoo!, Microsoft, Valve Software, TDK Electronics.

Ex-Chrome Developer Relations.