~mispy

I just wrote a scaling resize handler for the Our World In Data maps. It worked wonderfully in Chrome. It didn't work in Firefox or Safari. Why?

• Firefox: getBoundingClientRect(), which I relied upon for getting the map and container dimensions, accounts for SVG transforms in Chrome but not Firefox. I switched it out for a calculation based on getBBox, which always returns the untransformed user space coordinates.

• Safari: I missed the end parenthesis on the matrix transform string. Chrome and Firefox will parse it anyway and won't bother telling you. Safari will silently fail to apply the matrix at all. Goodness!

I've been using this silly OS for so long that sometimes I forget why I have things set up the way they are. Best write some of it down!

In recent years, I mainly run Linux locally for web development and general scripting shenanigans with Python, Ruby and/or JS. Since it's also how my stuff gets deployed, I interact with servers running it via ssh a lot.

• I use VMware Workstation to run Linux under Windows.

For web development purposes, even if you run native Linux it's worth getting a VM environment set up so you can have a fresh system for testing with. VirtualBox (the non-proprietary alternative) is probably fine too, I just happen to have a VMware license and it's what I'm familiar with.

• I run the latest Ubuntu LTS distribution.

I will freely confess this is mostly just because it is very popular, meaning if you hit any issues they're likely to have been encountered and resolved by someone else and thus easy to google.

• vim is my main text editor.

But I wouldn't necessarily recommend learning it if you're not already familiar: it's not obviously superior to other stuff in a way that justifies the steep learning curve. There are more approachable alternatives these days like Sublime Text or Atom. Perhaps one advantage vim has over these for me is that it's optimized to run in a terminal over ssh, for server-side work.

My approach is pretty focused on shells and terminals rather than editors, which isn't necessarily the best way to do it. I tend to have multiple terminals with multiple vims rather than using features like split panes.

The biggest customization I have is the addition of Sublime Text-like fuzzy file search and grep across a git repo, which runs off unite.vim.

Another .vimrc thing that took me way too many years to discover is that you can get it to interface with the terminal to automatically recognize copy/paste without having to type :set paste. You can also just hook the vim clipboard into the system clipboard so operations like d and p work across terminals.

I could probably fill another entire thing with vim notes! The combination of decades of development and extremely poor discoverability means vim is littered with secrets.

• GNOME Shell is my window manager.

Though really any WM would do. It's rare that I need to open anything other than terminals-- any browsers I need to test web stuff run in the host Windows.

• I use zsh instead of bash.

The most useful feature of zsh I've found is that it does case-insensitive substring tab completion rather than looking for the start of the filename. This means you can tab complete e.g. "Gemfile.lock" by typing "lock\t", instead of "Gem\t" (which would also match a normal "Gemfile" in the same dir). Given just how much time I spend typing in filenames, this is pretty important!

There's a community collection of zsh scripts at oh-my-zsh. The script for Python virtualenvwrapper integration was originally my doing.

• Dotfiles are the main form of terminal-world configuration.

I keep them synced across different systems by putting the canonical files in a Dropbox folder and symlinking them.

• History scroll (up arrow) and reverse-search (ctrl+r) are your friends.

They will save you an incredible amount of typing. They're also useful when you need to execute a complex command again but have forgotten the precise arguments you wanted-- sometimes I've sshed into a server just to reverse search because I know the thing I want is there somewhere!

• ssh is another dear friend.

If you use ssh a lot, make sure to take the time to configure it a bit. When I have multiple terminals open, I often end up sshing to the same target multiple times-- this can be optimized to run off the same connection in ~/.ssh/config:

ControlMaster auto  
ControlPath /tmp/ssh_mux_%h_%p_%r  
ControlPersist 4h  

Don't forget to set up key-based ssh authentication! Even if it weren't for the added security, it just saves a lot of password typing. ssh-copy-id makes this very easy to do.

I have some particular machines I ssh into so frequently that they're aliased to a single letter in my ~/.zshrc. e.g. my personal VPS, on which this site runs: alias z="ssh [email protected]"

• rsync is wonderful, and will solve a lot of file transfer problems.

In particular, it is way faster than scp -r for recursive network copying (and it still runs over ssh!). However, it has a bewildering multitude of command line options. I run this alias for general usage:

alias rs="rsync --human-readable --archive --verbose --compress --progress"

Of course, --archive then expands to -rlptgoD, which is --recursive --links --perms --times --group --owner --devices --specials. Got all that? Don't worry, it mostly Just Works. You use it like this:

rs dir [email protected]:/home/mispy/dir

The Unity manual has a fairly comprehensive overview of how MonoBehaviour event handlers are called in relation to each other. What it doesn't really tell you, though, is when they run relative to the code that created the object in the first place. The important points that are missing:

• Awake and OnEnable are called in the same stack as the line that activates the object
• Start is called after whatever code activated the object finishes
• This leaves a good space to do custom init in between OnEnable and Start

Here's a block of code to illustrate:

using UnityEngine;  
using System.Collections;

public static class ThingMaker {  
    public static void MakeThing(GameObject prefab) {
        Debug.Log("Before Instantiate");
        var obj = Object.Instantiate(prefab).GetComponent<Thing>();
        Debug.Log("After Instantiate");
        // If the prefab was active, at this point Awake and OnEnable have been called but *not* Start
        obj.Init(10);
        // Start runs at some point after this
    }
}

public class Thing : MonoBehaviour {  
    Rigidbody rigidbody;
    int boop = 0;

    void Awake() {
        // This is called immediately as soon as the object is first activated

        // At this point, you wouldn't have had a chance to call Init, so your
        // boop is still zero

        rigidbody = GetComponent<Rigidbody>();
        Debug.LogFormat("Awake {0}", boop);
    }

    void OnEnable() {
        // This is called immediately after Awake
        // However, if you disable and re-enable this object, it will get invoked
        // a second time while Awake won't
        Debug.LogFormat("OnEnable {0}", boop);
    }

    public void Init(int boop) {
        this.boop = boop;
    }

    void Start() {
        // This is called once, but it happens *after* whatever code activated the object
        // finishes running, meaning in this case after Init

        // We now have an appropriate level of boop to work with
        Debug.LogFormat("Start {0}", boop);
    }
}

And the output:

Twitter is pretty important to me. I'm a very quiet person, and it provides a pleasantly low-expectation way of interacting with a bunch of interesting humans (and robots?) that I wouldn't really have otherwise. Over the years, I have seen troubled people find acceptance, the formation of strong friendships and romances, and the activist power of many voices in unison. I have had opportunities to help others solve interesting problems and many people have helped me. Also there are a lot of puns. So many puns.

The social architecture of Twitter is flat and centers around individuals. Anyone can follow anyone; the largest corporation account and the shyest of teens are all mechanically more-or-less identical. Twitter has no real concept of a group or a topic, just a whole bunch of people and a few searching functions like hashtags and mutes.

While this flatness can be very empowering, especially in a political context, I have found it also makes certain kinds of social behavior difficult. When you follow someone on Twitter you are committing to see everything they say unless it is specifically directed to another account that you don't follow. For example, if I follow some brilliant programmer with obnoxious political views I must spend an awful lot of time rolling my eyes as they wax lyrical about the wonders of Ayn Rand and the horrors of millenial entitlement in order to read about interesting experiences with type systems. (I don't actually follow anyone like this, fortunately.)

The way I tweet about stuff myself is also affected. I mostly discuss programming, video games and sometimes ethics or demography. People who follow me do so under the expectation that there will be more of such things. Sometimes, however, I find I am really interested in something else! What if I am reading a book on the history of the Byzantine Empire and want to talk lots about the theological controversy of the Council of Chalcedon? Or just squee about cute anime boys? I end up self-censoring a lot, because at least one of those 2000 or so people is going to be bored by one of those things and there isn't really a way for them to opt out aside from opting out of me entirely.

So I find myself wanting a place on the internet aside from just Twitter, one which has something closer to a traditional forum architecture where the focus is more on particular groups or topics than individuals. In a network based on threads or channels nobody has to look at the contents of any particular topic, so they can be as deep or niche as they like without bothering anyone else. Maybe freenode, or even Reddit?

Let's say you have two overlapping circles. Instead of making something Venn diagram-esque, you'd rather they just marked out a continuous area.

In my particular case, I want the power node supply radius indicators on this hollow-asteroid-turned-into-a-spaceship not to obscure everything underneath them by being too intense at the overlap points:

This turns out to be surprisingly difficult. If it were just a matter of blending the two circles, you might accomplish it by using a Min shader BlendOp. However, what you want is to blend the circles one way and then blend the result of that blend in another way onto the screen. Since the rendering engine draws back-to-front, by the time you get to blending one circle onto another the first circle has already been blended with the scene and the necessary information lost.

Using stencil tests

One method to avoid overlap is to tell the circle shader to just not do anything if another circle has already claimed a given pixel. This is where the stencil buffer comes in handy.

 Shader "Custom/PowerCircle" {
     Properties {
         _MainTex ("Base (RGB)", 2D) = "white" {}
     }

     SubShader {
        Tags { "Queue"="Transparent" }

        Pass {
            Stencil {
                Ref 2
                Comp NotEqual
                Pass Replace
            }

             Blend SrcAlpha OneMinusSrcAlpha     

             CGPROGRAM
             #pragma vertex vert
             #pragma fragment frag
             #include "UnityCG.cginc"

             uniform sampler2D _MainTex;

             struct v2f {
                 half4 pos : POSITION;
                 half2 uv : TEXCOORD0;
             };

             v2f vert(appdata_img v) {
                 v2f o;
                 o.pos = mul (UNITY_MATRIX_MVP, v.vertex);
                 half2 uv = MultiplyUV( UNITY_MATRIX_TEXTURE0, v.texcoord );
                 o.uv = uv;
                 return o;
             }

             half4 frag (v2f i) : COLOR {
                 half4 color = tex2D(_MainTex, i.uv);
                 return color;
             }
             ENDCG
        }

    }

    Fallback off
}

The key here is the Stencil bit. Comp NotEqual means it will only run if the stencil buffer is not already equal to the number given by Ref 2, and Pass Replace means it will set it to 2 afterwards. So any number of this shader will only run once for a given pixel. Does it work? Well...

Not exactly, no. The problem is that each of the circles is actually a square sprite, and whoever renders first hogs all the pixels with useless completely transparent bits. We can fix this by telling the shader to discard any pixels with 0 alpha:

half4 frag (v2f i) : COLOR {  
    half4 color = tex2D(_MainTex, i.uv);
    if (color.a == 0.0)
        discard;
    return color;
}

That's a little better! But we still have weird things going on around the edges. That would be because in this particular texture, the edges are soft and gradually fade out with decreasing alpha values. Since they're not zero alpha they don't get discarded, and can prevent higher-alpha pixels from other circles from being rendered.

We can eliminate the soft edges by editing the texture, and by ensuring the filter mode is set to Point rather than Bilinear:

Success! A continuous bubble of power.

Those soft edges were kinda nice to have, though. It's possible the result could be postprocessed somehow to add them back in, or a different strategy could retain them. For the moment, I've opted for a compromise of keeping the soft edges on the image and just discarding a bunch more low alpha pixels:

half4 frag (v2f i) : COLOR {  
    half4 color = tex2D(_MainTex, i.uv);
    if (color.a < 0.3)
        discard;
    return color;
}

I think that looks kinda nice (for placeholder developer art). If anyone has any other ideas please let me know!

Alternatives

From poking around the Unity forums it sounds like there may be a way to do this by making clever use of ZTest and ZWrite that I didn't manage to puzzle out. The definitive solution is probably to use RenderTexture to blend the circles in advance and then drop them all in at once as a big sprite, but that requires adding more moving parts than I'd like.

A few months ago, I was tasked with porting the networking code of Wander to the PS4, and ended up writing some bits of it from scratch. I've been programming for years, but this was my first foray into C++ and serious gamedev, so I had to learn a lot!

The problem

Wander was designed to be an open world, massively multiplayer¹ exploration game. Players occupy the same universe and should be visible to one another as soon as they, uh, wander into range. Crucially, Wander is non-competitive, meaning clients can be trusted: there's not much to be gained from cheating, unless you get a kick out of teleporting yourself into odd places.

The game is based on CRYENGINE² and written in C++ and Lua. The engine has its own conception of networking, but it's based around matchmaking and lobbies and was generally a bit tangential to our requirements. So we needed an alternative that could connect nearby players, push state to the Cryengine objects, and work on both PC and PS4.

When I joined the project, it was already sorta functional on PC. We were using a C# library called Badumna, invoked from C++ via a C++/CLI wrapper. Badumna was a University of Melbourne project turned startup turned bankrupt turned open source. The website no longer exists, so I had to go find the documentation on archive.org. Not... not terribly promising.

What we tried

Loki gave me two alternatives to pursue: port Badumna to C++, or get Mono working on the PS4.

A quick look through the Badumna code made it clear the first option was waaay beyond my abilities. The library has around 72,000 lines of code, making heavy use of custom attributes and other C# features which aren't directly translatable to C++. Lots of fiddly meta stuff like this:

Much of the code is security features we didn't need, but separating that out from the critical network stuff is non-trivial. And while I had decent experience with C# and some prior knowledge of C, my C++ was not going to be good enough to port an entire networking library from scratch.

So I looked at the second option. Mono on PS4 is pretty experimental and requires AOT compilation, but Badumna was designed to run on mobile devices so I did make some headway and eventually got it to compile. Unfortunately, I ran into a NOT IMPLEMENTED runtime error from the stdlibs: there was no System.Security.Cryptography. And I was not going to try and homebrew a cryptography stdlib, so...

Having hit a dead end and feeling a bit frustrated, we went hunting for alternatives. Loki found a C++ library called VAST which looked promising: spatial P2P for virtual worlds! Exactly what we wanted, and it didn't matter very much that it was based on a 2D instead of 3D coordinate space. Even if we do have a floating island.

I set about trying to get the VAST code working on PS4. The library itself is small, but it has ... a dependency. Namely, The A Dynamically Assembled Protocol Transformation, Integration, and eValuation Environment Communication Environment™:

Yes, that is the actual expansion of that acronym. But wait, it gets better! You can do real-time Common Object Request Broker Architecture® Component Model with Component Integrated A Dynamically Assembled Protocol Transformation, Integration, and eValuation Environment Communication Environment™ Object Request Broker™

what is happening

oh my gods

So, a now-wiser Mispy would read all this as "FLEE, MORTAL" and seek greener pastures, but I was intrigued. This monstrosity runs on freakin' OpenVMS so surely I could get it working on PS4. And after lots of tinkering with preprocessor conditionals and tearing my hair out over the threading API, I succeeded! My VAST prototype successfully established a connection.

Of course at this point I discover that VAST doesn't really do any of the things we wanted it to in the first place.

I am still to this day not entirely clear on what exactly VAST is supposed to do. It might be useful if you're trying to load balance a series of servers attached to a spatial responsibility zone? Maybe? At any rate, what it definitely doesn't do is any of the actual networking part of P2P-- there's no NAT punching, STUN or TURN or what have you.

The key mistake I'd been making here was assuming that since we started with a complex system (Badumna), some approximation of that complexity was needed if I wanted to replace it. Given the task "port Badumna", instead of trying to solve the surface-level problem I should've thought more carefully about why it was needed and if there was another way to tackle the underlying issue.

Similarly, I should have analyzed VAST more carefully before getting too invested in the interesting technical process of getting it to work. Tinker mentality can be dangerous when what you need is careful design!

What we actually went with

Badumna does decentralized peer discovery using distributed hash tables to avoid single-point-of-failure issues. This is huge technological overkill for a small indie game, and it eventually dawned on me that all the peer discovery could just be done by polling some Python script. That left the core "talk to given peer" networking problem, which falls well within the purview of SteamNetworking (based on libjingle) and the PS4 libraries.

I would like to take a moment to express my great fondness for Steam's P2P networking API, by the way! It's super no-nonsense: you don't even need to worry about the current connection state, you just give it a packet and a destination id and off it goes, idempotently setting up whatever it needs to along the way. If the libjingle NAT-punching stuff isn't enough, it'll even go through Steam's own relay servers for you.

So the networking flow now looks like this:

• every few seconds, the game sends a POST request to a cherrypy server with the player's id (either Steam or PSN) and current world coordinates
• the server stores this information in Redis and does a zrangebyscore query to return a json array of other player ids within a certain radius
• the player ids are fed into a platform-agnostic networking layer wrapped around the Steam and PS4 APIs
• connections are established and players start pushing state to each other directly (using interpolation to avoid jerkiness)

Player logouts are handled by expiring ids from the Redis cache if the server doesn't hear from them for a while, which means it can be out of sync but that's fine since there's little harm in trying and failing to connect to the occasional offline player.

Loki worked out the Redis stuff and tied Cryengine into Steam, while I wrote a wrapper for the (rather lower-level) PS4 API to act as a drop-in replacement for SteamNetworking. This was reasonably non-trivial and the kind of thing I would love to open source, but grumble grumble Sony mutter NDAs grumble

Anyway, it worked! The first moment I saw another little Azertash³ swimming around on a TV was very satisfying indeed.

Improvements

There is one feature of Badumna it would be useful to replicate: relay messaging between peers. Currently, a player only sees another player if the two are directly connected to each other. This means the network topology is fully connected and the number of connections grows quadratically with the number of players:

So if you have 10 players in the same local area each of them is connected to 9 other people and there are 45 connections in total. For a small game like Wander this is fine, but if you wanted to handle high population density, you could instead organize players into star topologies where a single player relays information between two or more others. Much less overhead!